You might not, and neither do I have a problem with it. I grew up low income and rode plenty of “sketchy” bus lines where fights would break out. I know when to remove yourself from danger.
Plenty of people I know have, or at least have that perception. My S/Os parents visited from out of the country, and they were harassed by someone screaming racist epithets at them and got scared, and ended up taking Uber the rest of the time they were here. They are old, retired, and we didn’t want to risk them getting uncomfortable. Obviously we have the privilege of doing that, but not everyone does.
Again, it’s not about being poor, it’s about antisocial behavior that destroys the public spaces. That goes for rich assholes too.