The correct way of doing this is to never interact with an iot device directly. Put all of them on the same network with Home Assistant and then control all of them only via Home Assistant. Then you make one exception for home assistant to be accessible to the other networks.
This also allows you to disable Internet access for every single iot device Except home assistant.