Yeah, as someone who modded for several years, there were two insults people loved to throw at us: Either we were power tripping or we were janitors who didn’t matter.
Either of these were used whenever we enforced the rules of our community and kicked out people who didn’t want to play nice with the rest of it. Of course, they will never have a positive opinion of people who enforce a community’s rules.
And that’s the thing: The community. You do not spend several years modding a subreddit without getting to know the people and having some sort of relationship with them. The community is not an abstract, it’s people you get to know - often over several years - and that’s not something you want to leave behind.