Nope. That’s the basics of PKI and scalable, secure, low-trust environments.
You are the one who issued the cert. You can add it to your list of trusted certificates. You just have to check that this is the right certificate.
You can indeed do these things. But, are you and your users going to verify every cert for every request and response? That’s a lot of unnecessary cognitive load and tedium, both of which are known to compromise judgement. Are you going to automate it? Ok then how are you going to verify the authenticity of a given cert?
Your man in the middle scare comes from users who ignore cert warnings and continue without checking anything.
Humans are not rational actors. Does everyone read the entire EULA? Not even close.
The problem with your statement, and why it is fallacious, is that you are not accounting for humans besides yourself. I’d even argue that you should also take your human nature into account because we all make mistakes.
Robust security postures do not require everyone to act perfectly but accept and plan for the fact that we’re fallible. That is why chains and webs of trust were created, so that humans and automated services can take an approach of deference towards a less mutable “expert” on whether a claim of authenticity is trustworthy - giving them the capability and responsibility of deciding this for themselves introduces unnecessary targets for exploits.