I think it’s mostly that people assume men are bad caretakers and can’t handle a child. It’s partly that they are impressed by the fact that you’re not totally inept, but also that they need to do their part to make sure that dad doesn’t screw up with the child.
The “men are inept in the home” narrative is pervasive throughout all media, even things as inane as product commercials on TV.