I would go with an intel cpu for the integrated gpu that can easily handle all the transcodes. 12th gen or greater i7 should be more than enough. GPU passthrough for gaming will be trickier. I’m guessing modern games at 1080p which will probably require something with >12gb for some light future-proofing. 6800/4070 class card would be my bet.
Do you think the gaming part will be difficult because of the VM and passthrough, or was that simply a comment on the hardware choice? I don’t really care about anti cheat or anything and thought GPU passthrough and some assigned cores won’t be a problem. But I also don’t have any experience with this kind of setup.
Since you’ve had it running already with a 960, I don’t think you’ll have any issues. And if you’re running the igpu for the host processes/transcodes with the dedicated gpu for gaming, that eliminates any possible issues from having to use a single gpu for everything. Also, I wonder if using the dedicated card for the vm avoids the issues that could pop up from running nvidia in linux.
One thing I forgot to mention, you’re going to need a new psu especially to power the higher class card. 500W might be enough for the lower tier and a much more power efficient processor but you’ll be trading some flexibility which isn’t worth it imho.