I would go with an intel cpu for the integrated gpu that can easily handle all the transcodes. 12th gen or greater i7 should be more than enough. GPU passthrough for gaming will be trickier. I’m guessing modern games at 1080p which will probably require something with >12gb for some light future-proofing. 6800/4070 class card would be my bet.
Since you’ve had it running already with a 960, I don’t think you’ll have any issues. And if you’re running the igpu for the host processes/transcodes with the dedicated gpu for gaming, that eliminates any possible issues from having to use a single gpu for everything. Also, I wonder if using the dedicated card for the vm avoids the issues that could pop up from running nvidia in linux.
One thing I forgot to mention, you’re going to need a new psu especially to power the higher class card. 500W might be enough for the lower tier and a much more power efficient processor but you’ll be trading some flexibility which isn’t worth it imho.
Home-Server Upgrade Recommendations
Hi, could you help me and recommend a good upgrade for my (dated) home server?...