I think access to holodecks is a huge factor for that vs. doing a pastime in the real world. Because of what you said about being accessible to all vs. only to certain elites. Also because I wonder about the availability of holodecks for those not on starships or in Starfleet. I don’t remember if that’s mentioned in the various series and movies since they are typically centered around being on a starship. Transport tech is shown as being publicly used on Earth. I’d imagine that there would be holodecks available on Earth (maybe other planets), or at least holosuites like in Quark’s. Access to pastimes in the real world, without using holosuites, might be commonplace. Maybe trying something new in a holodeck/suite where serious injury is less likely before trying it in the real world.