A good rule of thumb is that the energy needs to go somewhere. So if the adapter was drawing a significant amount of power, it would get warm to the touch.
That’s true - And with a halfway decent thermal camera, you can see most of these unused chargers as “hot” spots. They’re so low power that they’re only slightly above ambient, but still something the cameras can see.
That’s how I found out that my desktop speakers consume power even with the physical button being off and status light dark. The power brick stays warm indefinitely, a good 20W feels like! I have to unplug that thing now when not in use. Any normal power brick will be <1W of course.