With PoE you’re doing 2 conversions which could waste more power, AC to 48V at the switch, and then 48V down to whatever the device needs with it’s internal buck converter. You also have slightly more losses on the longer run of low voltage 48V DC through ethernet, vs AC.
On the other side of things:
With PoE you only have 1 AC-DC conversion happening, every wall wart power adapter has an idle power draw even without a load attached to it. With PoE you just have the single switch power supply wasting power.
Overall I doubt the difference will be large enough to matter, and some PoE switches are quite power hungry even with nothing plugged in for some reason, so could end up costing more.
Dc-Dc is pretty efficient, I wouldn’t worry about conversion after the initial 48v, but I would potentially worry about losses in poor quality home wiring on longer runs in bigger homes