Presumably your house is not just a giant glass box though. If that office was not built and not heated it would offset your house Heating as well as everyone else’s.
Obviously not all offices are skyscrapers, but the ones that are are insanely wasteful. Fun fact Heating and Cooling to Greenhouse is expensive and that’s what skyscrapers are giant greenhouses they are wildly inefficient. And there are definitely tons of Industry that we just shove and to skyscrapers for literally no reason that they could be done from home without any change in workflow other than the lack of a commute