I’m actually not looking to do anything very complicated at all. Just connect the internet box directly to the router (via the patch panel). Router itself connected to a switch, and then all my other devices connected to the switch (with every connection to the switch going via the patch panel). I don’t have any plans to do anything with different networks, just the one local network.
The reason for having the patch panel is so I have a simple plug-and-play way of connecting everything up. I didn’t think it would be any different to having a standard walk jack, but with 24 ports instead of 1–4.