So, I get where you're going, but first: it takes much longer, but I do it at home while I'm asleep, so that doesn't really count. It's more the opposite, I really enjoy not having to stop at gas stations anymore. You just never wake up to an empty car anymore.
Then, for longer trips, it obviously takes more time to recharge than to refuel. But as a family of five, we had breaks before we had an ev. Last time we made a longer trip, we picked up my inlaws and wanted to visit some other family members that were about one and a half charge stops away. We took three breaks because someone had to pee, someone was hungry or someone wanted whatever. If you're a flying sales person that that wants to drive 2000 km in a day it'll obviously be annoying. For anybody else that takes some reasonable stops along the way, I doubt it changes much. Just stop the car at a charger and grab a coffee at the next supermarket. Once you had that you can usually drive again.