Yeah it is important to set boundaries. It’s critical, actually. But it need not be judgmental or cruel. Work is a place where I go to give my time and labor for money. It’s nothing else. I have friends, family, and a therapist. OP’s coworker may not, and that is sad. But their behavior is hurting OP, which means it’s time to set a boundary.
It really depends on a lot of things. I have worked places where there are firm remote work policies and ones where it’s entirely up to your manager. It also depends on the employee’s position and whether they require something only available in the office, like a VPN or dev terminal or physical materials, and whether they are expected in meetings.