Device passthrough is based on IOMMU by re-mapping device addresses and providing to the guest. Hardware can be mapped by just one machine, physical or virtual, so if your guests needs direct I/O to some hardware, you should have it 1:1. IOMMU must be supported by CPU (search for Intel VT-d or AMD-Vi), chipset, BIOS and OS + VM solution.
Originally IOMMU was primarly for passing NICs, HDD controllers, etc. With VGA pass-through you can meet some problems, it's little more complicated, but possible. Do some research, you can start here. You can only solve this with more than one VGA, because that one is already initialized.