Isolate them from your main network. If possible have then on a different public IP either using a VLAN or better yet with an entire physical network just for that - avoids VLAN hopping attacks and DDoS attacks to the game server that will also take your internet down;
If you’re using VLANs then configure your switch properly. Decent switches allows you to restrict the WebUI to a certain VLAN / physical port - this will make sure if your server / IoT devices are hacked they won’t be able to access the Switch’s UI and reconfigure their own port to access the entire network. Note that cheap TP-Link switches usually don’t have a way to specify this;
Only expose the game server service to the Internet. Everything else such as SSH, configuration interfaces and whatnot can be moved to another private network and/or a WireGuard VPN you can connect to when you want to manage the server;
Use custom ports with 5 digits for everything - something like 23901 (up to 65535) to make your service(s) harder to find;
Disable IPv6?
Use nftables / iptables / another firewall and set it to drop everything but those ports you need for the game sever and the management VPN access to work;
Use your firewall to restrict what countries are allowed to access your server. If you’re just doing it for a few friends only allow incoming connection from your country (wiki.nftables.org/wiki-nftables/…/GeoIP_matching)
Realistically speaking if you’re doing this just for a few friends why not require them to access the server through WireGuard VPN? This will reduce the risk a LOT and won’t probably impact the performance. This is a decent setup guide digitalocean.com/…/how-to-set-up-wireguard-on-deb… and you might use this GUI to add/remove clients easily github.com/ngoduykhanh/wireguard-ui.