here is a super user post about pcie virtualization, and it involves writing custom drivers.
Off the top of my head, a similar setup with transcoding comes to mind. In this case I used a shared volume mount between the media server and the transcoding server, and ssh to run ffmpeg on the remote server.
I think an easier setup would be to proxy app calls that use the gpu through ssh to your gpu container, then write the output to a volume that the non gpu host can read from.