That’s true, but now you have to remember which server is legit. One benefit of a centralized service is that you have centralized verification, which at one time was a point in Twitter’s favor.
I’m not very well versed in cryptography, but if I understand the certification system for websites, different sites apply to a certificate provider, of which there are multiple. Maybe something like this is possible for the Fediverse? Where a user or community or instance can be “verified” by one or more trusted verification “agencies” or whatever.