I think it’s not so much about middle management. They implement the policies of the actual decision makers.
I think it’s because the people who actually make these decisions perform their work mostly via face-to-face meetings, handshakes, projecting personal charisma, reading body language, and personal networking. This leads to an overestimation of how much of other jobs depend on time spent in the same room with others.
The executive imagines the meetings they missed, leading to lost opportunities. So they see a loss of productivity.
They don’t appreciate how much easier it was to edit that manual or analyze that data without Joe the human tuba trying to breathe around his phlegm in the cube next door, or without the folks three rows over arguing about which director’s vision of Superman was best.