PCI passthrough via OVMF/Examples

From ArchWiki
Jump to navigation Jump to search

As PCI passthrough is quite tricky to get right (both on the hardware and software configuration sides), this page presents working, complete VFIO setups. Feel free to look up users' scripts, BIOS/UEFI configuration, configuration files and specific hardware. If you have a problem, it might have been stumbled upon by other VFIO users and fixed in the examples below.

Note: If you have got VFIO working properly, please post your own setup according to the template on the bottom.

Users' setups

mstrthealias: Intel 7800X / X299, GTX 1070

Hardware:

  • CPU: Intel(R) Core(TM) i7-7800X CPU
  • Motherboard: ASRock X299 Taichi (Revision: A, BIOS/UEFI Version: 1.60A)
  • GPU: Asus STRIX GTX 1070
  • RAM: 32GB DDR4

Configuration:

  • Kernel: Kernel version 4.14.8-1-skx (patched crystal_khz=24000).
    • Custom patches:
      • skylakex-crystal_khz-24000.patch (see below)
    • Patches used from linux-ck:
      • enable_additional_cpu_optimizations_for_gcc_v4.9+_kernel_v4.13+.patch
      • 0001-add-sysctl-to-disallow-unprivileged-CLONE_NEWUSER-by.patch
      • 0001-e1000e-Fix-e1000_check_for_copper_link_ich8lan-retur.patch
      • 0002-dccp-CVE-2017-8824-use-after-free-in-DCCP-code.patch
    • Config:
      • PREEMPT, NO_HZ_IDLE, 300HZ, MSKYLAKE
  • GitHub: Link TBD
  • Benchmarks: https://imgur.com/a/hIfQD
  • Using libvirt/QEMU: libvirt 3.10.0 / QEMU 2.11.0
  • Issues you have encountered, special steps taken to make something work a bit better, etc.
    • Skylake-X default clock incorrect in 4.14.8 (https://bugzilla.kernel.org/show_bug.cgi?id=197299)
      • Was unable to resolve timing issue using adjtimex
      • Patching kernel source to crystal_khz = 24000 resolved timing/performance issues
    • Enable 'Intel SpeedShift' in BIOS, installed cpupower', set governor='performance'
      • Verify: dmesg|grep HWP
        • intel_pstate: HWP enabled
    • Enable HT in BIOS
    • Enable 'deadline' IO sceduler:
      • echo 'ACTION=="add|change", KERNEL=="sd*[!0-9]|sr*", ATTR{queue/scheduler}="deadline"' >> /etc/udev/rules.d/60-schedulers.rules
    • Bypass x2apic opt-out:
      • GRUB_CMDLINE_LINUX="... intremap=no_x2apic_optout ..."
    • Isolate cores for Windows VM:
      • GRUB_CMDLINE_LINUX="... isolcpus=2-5,8-11 nohz_full=2-5,8-11 rcu_nocbs=2-5,8-11 ..."
    • Use hugepages (2MB) for all VM memory allocation
    • memoryBacking: <hugepages/><nosharepages/><locked/><access mode='private'/><allocation mode='immediate'/>
    • Extracted rom from GPU; used for <rom file=../> config
    • Using MSI for GPU and GPU Audio (configured in Windows registry; FPS seems same as using line-based interrupts)
  • Hardware setup
    • PCIE1: NVIDIA GeForce GT 710B (for host)
    • Onboard: ASRock XHCI 3.1 USB (for host)
    • Onboard: Intel I219 NIC (bridged)
    • PCIE3: Asus Xonar STX (passthrough to Win10)
    • PCIE5: NVIDIA GeForce GTX 1070 (passthrough to Win10)
    • M2_1: Samsung 960 EVO 500GB (passthrough to Win10)
    • Onboard: Intel XHCI USB 3.0 (passthrough to Win10)
    • Onboard: Intel HDA (passthrough to Win10)
    • Onboard: Intel I211 NIC (passthrough to Win10)
    • Onboard: ASRock AHCI SATA A1/A2 (passthrough to Linux)

DragoonAethis: 6700K, GA-Z170X-UD3, GTX 1070

Hardware:

  • CPU: Intel Core i7-6700K (using iGPU as the host GPU)
  • Motherboard: Gigabyte GA-Z170X-UD3 (Revision 1.0, BIOS/UEFI Version: F23d)
  • GPU: MSI GeForce 1070 Gaming X (10Gbps)
  • RAM: 16GB DDR4 2400MHz

Configuration:

  • Kernel: "Vanilla" Linux (no ACS patch needed).
  • Using libvirt: XML domain, helper scripts, IOMMU groups, etc available in my VFIO repository.
  • Guest OS: Windows 8.1 Pro.
  • The entire HDD is passed to the VM as a raw device (formatted as a single NTFS partition).
  • USB keyboard and mouse are passed to the guest VM and shared with the host with Synergy.
  • Virtualized audio: PulseAudio -> local Unix socket. Previously, I've had a bit more complex setup in which PA on the host was configured to accept TCP connections, and the envvars required for QEMU to use PA were pointed at the PA server running on 127.0.0.1. This way it was not required to change the QEMU user (exact details in the repo), but introduced other minor issues I've resolved later.
  • Bridged networking (with NetworkManager's and this tutorial's help) is used. bridge0 is created, eth0 interface is bound to it. STP disabled, VirtIO NIC is configured in the VM and that VM is seen in the network just as any other computer (and is being assigned an IP address from the router itself, can communicate freely with other computers).
  • For some reason, enabling intel_iommu=on on the kernel cmdline without CSM support enabled in UEFI causes a black screen on boot. Enable it (Windows 8/10 features need to be enabled to show "CSM Support", selecting "Other OS" hides that).

Manbearpig3130's Virtual Gaming Machine

Hardware:

  • CPU: Intel Core i7-6850K 3.6GHz
  • Motherboard: Gigabyte x99-Ultra Gaming (Revision 1.0, BIOS/UEFI Version: F4)
  • Host GPU: AMD Radeon HD6950 1GB
  • Guest GPU: AMD R9 390 8GB
  • RAM: 32GB G-Skill Ripjaws DDR4 runing at 3200MHz

Configuration:

  • Host Kernel: Kernel version Linux 4.7.2-1.
  • Using libvirt QEMU/KVM with OVMF: link to domain XMLs/scripts/notes: https://github.com/manbearpig3130/MBP-VT-d-gaming-machine
  • Host OS: Arch Linux
  • Guest OS: Windows 10 Pro
  • 2x 480GB SSDs set up in LVM striped mode (with mdadm) formatted to ext4 are mounted in linux which contains the guest's qcow2 virtual VirtIO disk file.
  • USB Host controller is passed through, giving most USB ports to the VM, leaving my USB 3.1 controller with attached USB hub for the host.
  • Motherboard has two NICs, one is passed into VM (Works perfectly after installing Killer NIC Driver).
  • VM gets dedicated 16GB RAM via static hugepages.
  • CPU pinning increased performance considerably.
  • Windows boots straight into Steam big picture mode on primary display (43" Sony Bravia). Overall an awesome gaming machine that meets my gaming needs and lust for GNU/Linux at the same time.
  • Quirks:
  • I sometimes have to reinstall the AMD drivers in Windows to get HDMI audio working properly, or roll back to Windows HDMI driver. I normally use a USB headset which works fine anyway.

Bretos' Virtual Gaming Setup

Hardware:

  • CPU: Intel Core i7-7700k
  • Motherboard: Z270 GAMING M3 (MS-7A62)
  • GPU: ASUS GeForce GTX960
  • RAM: Kingston HyperX 3x8GB DDR4 2.4GHz
  • Storage: 2x Corsair MP500 m.2 240G SSDs in mdadm RAID0, 1x WD Black 1TB for storage. 100GB LVM volume as writeback cache for HDD

Configuration:

  • Kernel: vanilla
  • Host OS: Arch Linux
  • Guest OS: Windows 10 Pro
  • Using libvirt/QEMU: GitHub config repository: [1]
  • Issues you have encountered: AUDIO. Had to get USB audio adapter and pass it through.
  • No issues other than audio. Works like a charm.

Skeen's Virtual Gaming Rack Machine

Still work in progress.

Hardware:

  • CPU: AMD FX(tm)-8350
  • Motherboard: MSI 970A SLI Krait Edition (MS-7693) (Revision 5.0, BIOS/UEFI Version: 25.4)
  • Host GPU: ASUS GeForce GTX 480 1536MB
  • Guest GPU: ASUS GeForce GTX 480 1536MB
  • RAM: 2x8GB Kingston HyperX Fury White DDR3 1866MHz
  • Storage: 2x250GB Samsung EVO (MZ-75E250) set up in LVM striped mode (with mdadm), 2x1TB WD Blue (WDC_WD10SPCX) for storage. 250GB LVM volume as writeback cache for HDD.

Configuration:

  • Host Kernel: Linux 4.9.0-3 (No ACS)
  • Host OS: Debian Stretch
  • Guest OS: Windows 10 Home (10_1703_N, International Edition)
  • Using libvirt QEMU/KVM with OVMF: See Github

Issues you have encountered:

  • Identifical GPUs; solved using this section on the wiki, but with the script from the corresponding discussion page. Several adaptations for Debian were required too, but are not applicable for this forum.
  • "Error 43: Driver failed to load";
    • Spoofing vendor_id caused Windows to crash during boot-up.
    • Linux VMs complain unable to find GPU from Grub2, and booted into blind-mode, but would still pick up the graphics card during the boot process, and would remain functional until VM reboot.
    • Vendor_id spoofing turned out to work after solving the real problem (Missing UEFI compatability in VBIOS).
  • Missing UEFI (OVMF) compatability in VBIOS;
    • Requested a GOP/UEFI compatible VBIOS upgrade from ASUS, but ASUS could neither understand the request, or provide the upgrade (The only thing supplied was standard support answers).
    • No compatible VBIOS was found at TechPowerUp.
    • Finally solved by manually hacking GOP/UEFI support into the ROM, using GOPupd. Current rom was dumped within a Windows 10 VM using GPU-Z, then modified using GOPupd, pulled to Linux, and provided using the rom file parameter in the VM XML file.
  • VM only uses one core (even with mode=host-passthrough): solved using this section on the wiki.

Quirks:

  • The GPU that is being passed through, does not support resetting, and thus doing a hard-reboot / shutdown of the VM locks the GPU.
    • The VM cannot be started again unless the Host machine is rebooted.
      • When doing a clean reboot / shutdown, allows the VM to start up as expected without reboot..
    • Removing and rescanning the PCI device, does not change anything.
    • No further attempts at powercycling the GPU from the host has been done (Yet).
  • Passing VM audio to host via PulseAudio results in heavy crackling.

droserasprout poor man's setup

Hardware:

  • CPU: Intel Core i3-6100
  • Motherboard: ASRock H110M2 D3 (BIOS version 0603)
  • Host GPU: Intel HD 530
  • Guest GPU: Sapphire Radeon R7 360
  • RAM: Apacer 8Gb 75.C93DE.G040C, Kingston 4Gb 99U5401-011.A00LF

Configuration:

  • Kernel: linux-lts 4.9.67-1 (vanilla)
  • Host OS: Arch Linux
  • Guest OS: Windows 10 Pro 1709 (build 16299.98)
  • Using libvirt/QEMU: See my configs and IOMMU groups on Github
  • HDD partition is passed to the VM as a raw virtio device.
  • HD Audio is passed too. Works fine with both playing and recording, no latency issues or glitches. After VM is powered off host audio works fine too.
  • Guest's latency is slightly better when CPU cores are isolated for VM.
  • i2c-dev module added to bypass 'EDID signature' error when switching HDMI. Without it I had to switch video output before starting VM for some reason.
  • intremap=no_x2apic_optout kernel option added to bypass motherboard firmware falsely reporting x2APIC method is not supported. Seems to have a strong influence on the guest's latency.
  • Overall performance is pretty close to the native OS setup.

prauat: 2xIntel(R) Xeon(R) CPU E5-2609 v4, 2xGigabyte GeForce GTX 1060 6GB G1 Gaming, Intel S2600CWTR

Hardware:

  • CPU:2xIntel(R) Xeon(R) CPU E5-2609 v4
  • Motherboard: Intel S2600CWTR(Revision ???, BIOS/UEFI Version: SE5C610.86B.01.01.0022.062820171903)
  • GPU: 2xGigabyte GeForce GTX 1060 6GB G1 Gaming [GeForce GTX 1060 6GB] (rev a1)
  • RAM: Samsung M393A2G40EB1-CPB 2133 MHz 64GB (4x16GB)

Configuration:

  • Kernel: Linux 4.14.15-1-ARCH #1 SMP PREEMPT
  • Using libvirt/QEMU: https://github.com/prauat/passvm/blob/master/generic.xml
  • Most important:
  • When using nvidia driver hide virtualization to guest <kvm><hidden state='on'/></kvm>
  • Configuration works with Arch Linux guest os, still work in progress.

Dinkonin's virtual gaming/work setup

Hardware:

  • CPU: Intel(R) Core(TM) i7-7700K CPU @ 4.60GHz
  • Motherboard: MSI Z270 GAMING PRO CARBON (MS-7A63) BIOS Version: 1.80
  • GPU: 1x Gigabyte GeForce GTX 1050 2GB (host), 1x MSI GeForce 1080 AERO 8GB(guest)
  • RAM: 32GB DDR4

Configuration:

  • Kernel: Kernel version linux 4.15.2-2-ARCH.
  • Using libvirt/QEMU (patched from AUR) with OVMF
  • Installed qemu-patched from AUR because of crackling/delayed sound with pulseaduio (still hear ocasional pops/clicks while gaming.
  • Patched video bios with https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher, because of error:
vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff

pauledd's unexeptional setup

Hardware:

  • CPU: Intel Core i7 6700K
  • Motherboard: Gigabyte GA-Z170N-WIFI Retail (Revision 1.0 , BIOS/UEFI Version: F20)
  • GPU: 8GB Palit GeForce GTX 1070 Dual Aktiv PCIe 3.0 x16 (Retail)
  • RAM: 16GB G.Skill RipJaws V DDR4-3200 DIMM CL16 Dual Kit

Configuration:

  • Kernel: 4.15.2-gentoo
  • Using libvirt/QEMU: libvirt-4.0.0, qemu-2.11.1, https://github.com/pauledd/GPU-Passthrough/blob/master/win10-2.xml , using vfio kernel module
  • Had to dump VBIOS in at the host while GPU was normally attached (and drivers loaded) (see https://stackoverflow.com/a/42441234), had to set CPU settings manually according to my cpu (host-passthrough, sockets 1, cores: 4, threads: 2 ) or some games will regularly crash, see my xml how to insert vbios, still have audio clicking/lag with pulseaudio but thats ok for me, no further patching etc.. works out of the box without any issues.
  • 3DMark Results Time Spy Graphic Score: Native Windows 10: 5564 , GPU-Passthrough: 5541

hkk's Windows gaming machine (6700K, 1070, 16GB)

Hardware:

  • CPU: Intel Core i7-6700K 4.5GHz
  • Motherboard: AsRock Fatality Gaming K6 Z170 (rev. 1.05)
  • Host GPU: Intel GPU HD530 with 1GB shared memory
  • Guest GPU: Gigabyte GeForce GTX1070 G1 Gaming 8GB
  • RAM: 16GB G.Skill RipjawsV @ 3333 MHz CL14-15-15-31-2T [DDR4]

Configuration:

  • Host Kernel: Kernel version Linux 4.15.7-1-vfio (with ACS patch included).
  • Using libvirt QEMU/KVM with OVMF
  • Host OS: Arch Linux
  • Guest OS: Windows 10 Pro
  • 128GB Intel 600p SSD splited into 3 partitions: 512MB for EFI, 30GB for / in Btrfs and other gigs for Windows 10 installed straight on SSD.
  • Two more HDDs for Windows. 1TB and 650GB
  • Passed specific devices like X360 and some of single USB ports.
  • One NIC behind NAT on VM machine.
  • VM gets dedicated 8GB RAM via static hugepages.
  • CPU pinning increased performance considerably and machine gets 4/4 cores of my 4/8 CPU
  • Windows boots on second screen with simple script which shutting down display with xrandr.
  • Using Synergy to share mouse and keyboard between systems.
  • Quirks:
  • Synergy is not perfect and will not entirely work in some games.
  • No boot screen. Display is turning on only when Windows is up and ready to go.

sitilge's treachery

Full info: https://git.sitilge.id.lv/sitilge/dotfiles

Hardware:

  • CPU: Intel Core i5 6600K
  • Motherboard: Asus Z170i
  • GPU: Gigabyte Radeon RX460 OC 2GB
  • Storage: Samsung 850 EVO 500GB
  • RAM: Corsair 16GB DDR4
  • Mouse, Keyboard: Logitech M90, Vortex Pok3r

Host Configuration:

  • Kernel: linux-vfio
  • Packages: qemu-git, virtio-win, ovmf

Guest Configuration:

  • OS: Windows 10 Pro
  • CPU: host
  • Motherboard: host
  • GPU: passthrough
  • Storage: 64GB
  • RAM: 8GB
  • Mouse, Keyboard: passthrough

Notes:

  • You can easy simlink the config files using stow -t / boot mkinitcpio and then mkinitcpio -p linux-vfio.
  • -smp cores=4 - guest might utilize only one core otherwise.
  • -soundhw ac97 - I'm passing mobo audio thus ac97. Download, unzip and install the Realtek AC97 drivers within a guest.
  • Use virtio drivers for both block devices and network. For example, the ping went down from 250 to 50.
  • Mouse and keyboard passthrough solved the terrible lag problem which was present in emulation mode.
  • Make sure virtualization is supported and enabled in your firmware (UEFI). The option was hidden in a submenu in my case.
  • As trivial as it sounds, check your cables.
  • Be patient - it took more than 10 minutes for the guest to recognize the GPU.

chestm007's hackery

Hardware:

  • CPU: Ryzen 7 1800x
  • Motherboard: Asus ROG Crosshair VI (Revision 1, BIOS/UEFI Version: 3502)
  • GPU: Asus ROG RX480oc 8GB
  • RAM: 32gb Ripjaws 2400mhz

Configuration:

  • Kernel: 4.16.12-1-ARCH.
  • Using libvirt/QEMU: libvirtd (libvirt) 4.3.0, QEMU emulator version 2.12.0,

Notes:

  • using ic6 audio - works fine for me.
  • have a working looking-glass setup, however cant get spice to pass through keyboard and mouse, currently using a mixture of synergy and a dedicated screen as a workaround

Eduxstad's Infidelity

Hardware:

  • CPU: Ryzen 2600X @ 3.7 GHZ
  • Motherboard: ASUS PRIME B350-PLUS(BIOS/UEFI Version: 4011)
  • GPU1 (Guest): MSI 390 8GB @ Stock
  • GPU2 (Host): XFX 550 4GB @ Stock
  • RAM: 2 x 8GB (16GB) @ 3000 HZ
  • Guest OS: Windows 8.1 Embedded Pro

Configuration:

  • Kernel: 4.17.3-1-ARCH (vanilla).
  • Using libvirt/QEMU: libvirt/virt-manager (https://github.com/eduxstad/vfio-config).
  • Look in the repository for complete documentation of extra steps taken
  • Overview: VM managed using virt-manager, using looking glass for primary io and built in spice display server as backup. Passing vm audio back to pulseaudio. Using hugepages for RAM. SCSI Drivers installed for hardware drive support.

Pi's vr-vm

Hardware:

  • CPU: i7-8700k @ 4.8 GHz
  • Motherboard: MSI Gaming Pro Carbon (BIOS/UEFI Version: A.40/5.12)
  • GPU: Palit RTX 2080 Ti
  • RAM: 4x8GB G.Skill DDR4 @ 3000 MHz

Configuration:

  • Kernel: latest mainline (rc if available)
    • custom built with ZFS, WireGuard
    • CONFIG_PREEMPT_VOLUNTARY=y to work around QEMU bug with long guest boot times
  • Startup scripts/additional info: https://github.com/PiMaker/Win10-VFIO
  • Issues encountered:
    • PUBG would not launch at all
      • Solution: Enable the HyperV clock with <timer name='hypervclock' present='yes'/> and disable hpet with <timer name='hpet' present='no'/>
    • VR would start to stutter badly after about 20-30 minutes of playtime (this one took me about 2 weeks to finally figure out :-)
      • Solution:
        • Enable invariant tsc passthrough with <feature policy='require' name='invtsc'/> (required even if using host-passthrough!)
        • Enable MSI for the GPU (using tool from here)
        • Enable vAPIC and synic in the HyperV configuration
        • Manually move all IRQs to host cores using qemu_fifo.sh script from my GitHub repo above
  • Overview: SteamVR-capable gaming and workstation rig, passing through NVIDIA GPU and onboard USB-controller (leaving an additional ASMedia USB port to the host). 22 GB hugepages memory, 10 of 12 cores (with SMT) passed through. Audio working via Scream (https://github.com/duncanthrax/scream) - with IVSHMEM, surprisingly low latency and no stutters.

coghex's gaming box

Hardware:

  • CPU: i7-8086k @ 5.0 GHZ (8086k is just a binned 8700k)
  • Motherboard: GIGABYTE Z370 AORUS Gaming 7 rev1.0 (BIOS/UEFI Version: F15a)
  • GPU: GIGABYTE GV-N108TAORUSX WB-11GD AORUS GeForce GTX 1080 Ti Waterforce WB Xtreme Edition 11G @ ~2Ghz
  • RAM: 4 x 8GB (32GB) Corsair Dominator Platinum @ 3600 HZ (XMP)

Configuration:

  • Kernel: linux-zen-5.5.8.zen1-1
  • Modules: raid0 raid1 md_mod ext4 vfat ahci vfio_pci vfio vfio_iommu_type1 vfio_virqfd usbhid it87 (aur version is unmaintained and the support for the ITE8686E chip on this board is limited, replace it87 source with that which is found here for more comprehensive support)
  • Virsh: virsh-5.10.0
  • Qemu: qemu-system-x86_64-4.2.0 machine='pc-i440fx-4.2'
  • Performance Services: irqbalance-1.6.0, ananicy-git-2.1.0.r22, cpupower5.5-1
  • EDIT(2020): much has changed since this setup was posted years ago and a custom kernel is no longer needed on this hardware, everything works perfectly...
  • scripts, libvirt XML, and personal configs can be found here: https://github.com/coghex/hoest
  • host boot options: intel_iommu=on iommu=pt rd.driver.pre=vfio-pci acpi_enforce_resources=lax
  • systemd modprobe.d options: kvm ignore_msrs=1 (avoids critical bugs), kvm report_ignored_msrs=N (cleans up journal logs)
  • libvirt features: acpi, apic, kvm hidden state='on', vmport state='off'
  • guest hyper-v options: hv-relaxed, hv-vapic, hv-spinlocks (retries='8191'), hv-vpindex, hv-runtime, hv-synic, hv-stimer, hv-stimer-direct, hv-reset, hv-vendor_id (value='1234567890ab'), hv-frequencies, hv-reenlightenment, hv-tlbflush, hv-ipi, (hv-evmcs and hv-no-nonarch-coresharing seemingly do not work yet in virsh)
  • make sure to use the multifunction field for the GPU's hdmi audio controller and set them to the same slot, otherwise the audio interrupts will hang. Someone should probably add that to the guide...
  • im running the clock at 100Hz, the people running it at 1000 with the zen or ck kernel should know that the MuQSS scheduler works the same regardless of this speed and 1000 will just add more useless interrupts.
  • cpu pinning works best for single VM performance, default host-passthrough works best for multiple running VMs.
  • on windows, the MSI_util_v2 gets used every update to reset MSI interrupts on the GPU.

Hardware Specific:

  • Fully-Functional Passthrough Devices: this motherboard has many PCI slots, all of these devices have been working flawlessly with little setup for years now:
    • Inatek USB Card: KT5001 [2]
    • Creative Sound Card: 70SB155000001 [3]
    • EDUP WiFi Card: AC9636GS (must use virtio usb passthrough for bluetooth functionality) [4]
    • Intel Optane SSD: SSDPED1D480GASX [5]
    • Zotac GeForce GT 710: ZT-71304-20L (this one does not seem to be available on amazon anymore, a shame since its one of the few high performance PCIEx1 cards...) [6]
  • none of the proprietary gigabyte software works, in fact, it blue screens windows and installs itself as a startup program, forever locking you out.
  • if anyone else uses this exact motherboard, there are two internal USB IOMMU groups, even with the ACS patch. one will include the usb labeled "USB 3.1", and the other will include all the other USBs. this means if you want more than just a keyboard and mouse, you will need either a usb hub to plug into the 3.1 slot and passthrough, or a PCIE USB bus.
  • the two ethernet ports are in different IOMMU groups, making this a perfect motherboard for vfio.
  • the ACS patch is needed on this motherboard if you want to use two graphics cards at once in seperate IOMMU groups. this sets the main GPU to PCIEx8 instead of x16.

Roobre's VFIO setup

Hardware:

  • CPU: Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz (OC'ed to 4.50)
  • Motherboard: ASUS ROG MAXIMUS VIII GENE, v3801
  • GPU: EVGA GTX 1080Ti
  • RAM: 32GB DDR4 2400 (2x Ballistix)

Configuration:

Issues:

  • Pulseaudio never worked good (too much crackling), so I ended up passing-through an USB 3.1 PCI controller and connecting an USB audio card to it. That card is then connected to one of my MoBo's inputs, and echoed using pulseaudio's `loopback` module.
  • Synergy works really great. On some games (ones who take control of the mouse pointer, e.g. first-person), you need to lock the mouse cursor to the VM window to avoid issues (camera moving too fast).

laenco's VFIO setup

Hardware:

  • CPU: Ryzen 9 3950X @ 4.15Ghz all-cores via PBO
  • Motherboard: Asus ROG STRIX X470-F GAMING (BIOS/UEFI Version: 5406)
  • GPU1 (Guest): Palit GeForce GTX 1080 8GB @ Stock
  • GPU2 (Host): MSI RX 570 8GB @ Stock
  • RAM: 4 x 16GB (64GB) @ 3333 MHz

Configuration:

  • Guest OS: Windows 10 Pro
  • Kernel: 5.4.13-arch1-1-gc (-ck is also good). No ACS patch.
  • Using vanilla QEMU 4.2.0
  • AMD Ryzen currently (2020.01.20) got bugged with smp threads option - VM stuck on start.
  • Got classic Nvidia error 43 - classically fixed. But also added some cpu flags which are set automatically with kvm=on found here https://github.com/qemu/qemu/blob/master/target/i386/cpu.c#L4008
  • As pure qemu have no option to pin cpu cores and self threads - using python script "cpu_affinity" - credits to https://github.com/zegelin/qemu-affinity/ and also a copy in my repo. Requires debug-threads=on
  • Using dynamically allocated hugepages 2Mb
  • Hardly using VirtIO
  • Using hardware usb switch like Aten US224-AT and hdmi switch "many-to-one", which allow me to have one monitor, mouse, keyboard and some usb devices, and switch them by button between host and guest.
  • Repo with current major system config and script for VM could be found here https://github.com/laenco/vfio-config

Poncho's VFIO setup

Hardware:

  • CPU: Ryzen 7 2700x @ stock (PBO)
  • Motherboard: MSI B450-A PRO MAX (BIOS/UEFI version: 7B86vM5)
  • GPU1 (Guest): MSI GeForce GTX 1660 Ti Gaming X 6GB @ Stock
  • GPU2 (Host): AsRock RX 570 8GB @ Stock
  • RAM: 2 x 16GB @ 3200MHz

Configuration:

  • Guest OS: Windows 10 Home
  • Kernel: 5.4.17-1-MANJARO vanilla, no ACS patch
  • libvirt 5.10.0/QEMU 4.2.0: win10.xml gist

Issues I have encountered:

The main issue that plagued me for a while was crashes. The crashes were occurring more often in more demanding games, and less often when the host was as idle as possible. Mostly just the game crashed, but sometimes it would take the VM with it. I finally solved this by changing my RAM speed from 3466MHz to 2666MHz. I'll try slowly bumping the RAM speed back up step by step to find the point of instability and I'll edit this once I've found it. Edit: It's 3200MHz.

Describing setup loosely:

  • On the hardware side, my 620 Watt PSU is perfectly adequate, despite some early concerns.
  • 16 PCI lanes for the Guest card, 4 for the Host card. 8+8 is also a solution but I haven't had the need to try this.
  • Regarding the VM setup, I pinned and isolated 12 logical processors, leaving 4 to the host. The isolation was achieved using these scripts. I needed the git version of cpuset for it to work. The pinning alone didn't change performance at all.
  • Audio passthrough is done through the usual pulseaudio solution, I have no demonic interference, works almost perfectly. I have to plug my headset directly into the VM when I want my mic to not sound garbage. ICH9.
  • I did try enabling MSI on the GPU in an attempt to fix the crashes described above, but all I got was a small but significant reduction in performance.
  • Regarding input, I got a bit lucky. My motherboard has two USB 3 ports all alone in a single IOMMU group. I got a 4 port USB switch and the only complaint I have with it is that while switching from Host to Guest is near instantaneous, the other way around takes about 3 seconds.
  • No trouble at all getting the NVIDIA gpu to run in a VM, used the general solution in the wiki, including <kvm><hidden state='on'/></kvm>
  • As for storage, I just gave the VM a whole raw SATA SSD. Benchmarking shows about a 50% performance drop, but I haven't really noticed significantly longer loading times in games. In the future I might try reinstalling windows on a virtual image for cloning purposes and use the SSD as a game drive.
  • All in all, there is about a 10% performance loss in CPU intensive games, compared to bare metal. after four months of use, I've found that some games do not run as well as others. EFT can drop over 25 fps compared to native. GTA V can drop more. Insurgency Sandstorm starts stuttering, something frame time related I think, when there's anything graphical going on in the host machine. Less demanding games run great. Hoping Zen 3 saves me with more CPU resources, L3 cache, bandwidth etc. Ideally, I imagine separate NUMA nodes would be great for this, but way above my price range.

zane's not working box

Hardware:

  • MacBook Pro 11,x (2014 Model)
  • CPU: Intel Core i7-4770HQ
  • Motherboard: Apple
  • GPU: Iris Pro 5200 for host, GTX 1660 eGPU over Thunderbolt 2 for guest
  • RAM: 16GB

Configuration:

  • Kernel: linux-vfio from aur 5.5.8
  • qemu: 4.2.0
  • libvirt: 5.10.0
  • ovmf: 1:r26976.bd85bf54c2
  • libvirt/QEMU: libvirt setup; qemu setup

Description:

  • The qemu script include lines for setting up device mapped file for raw disk access. 3D Performace is about 40% to 80% native depending on the application, with periodic lag spike/stutter.

Issues:

  • Use apple_set_os.efi or spoof_osx_version with refind to avoid black screen on start. This prevents Apple firmware from shutting down host iGPU when booting Linux/Windows.
  • CPU pinning for guest is mandatory as it removes majority of stutters. After that isolate host CPU cores and pin emulator/IO threads as well. Pi's script for pinning IRQ handlers also helps. Hugepages for memory helps.
  • Kernel parameters: intel_iommu=on iommu=pt pcie_acs_override=downstream pci=realloc vfio-pci.ids=10de:2184,10de:1aeb,10de:1aec,10de:1aed,8086:0d01,8086:156d,8086:156c isolcpus=0-5 nohz_full=0-5 rcu_nocbs=0-5 default_hugepagesz=1G hugepagesz=1G hugepages=12 mitigations=off pcie_aspm=off module_blacklist=nvidia audit=0 loglevel=3 quiet. Everything starting with mitigations=off are optional. pci=realloc is mandatory or you will get NVRM: This PCI I/O region assigned to your NVIDIA device is invalid: NVRM: BAR1 is 0M @ 0x0 (PCI:0000:0a:00.0) error in dmesg and Error 43 for the Nvidia driver in guest.
  • Add vfio_pci vfio vfio_iommu_type1 vfio_virqfd to your mkinitcpio.conf as normal. Add options kvm ignore_msrs=1 and options kvm report_ignored_msrs=N to your /etc/modprobe.d/kvm.conf as well.
  • For me ACSO patch is mandatory, available from linux-vfio aur.
  • Enabling MSI for guest GPU seemingly helps. Using ioh3420 device and passthrough GPU on top of that DOES NOT seem to help, while making PulseAudio output cracks badly. Setting mixing-engine=off for PulseAudio also makes it cracks badly so consider USB soundcard if needed. (I personally use the sound out on my monitor from guest). While I'm not sure what this option does, setting in.buffer-length on PulseAudio audiodev reduces cracks.

Muata's VFIO setup

Hardware:

  • CPU: i7 4790
  • Motherboard: MSI B85M-G43 BIOS/UEFI Version: V3.9 (03/30/2015)
  • GPU: NVIDIA GeForce GTX 1060 6GB (MSI Gaming+)
  • RAM: 16GB

Configuration:

  • Kernel: linux-zen 5.5.11-1
  • Using libvirt/QEMU: VFIO setup;
  • qemu: 4.2.0
  • libvirt: 5.10.0
  • No issues at moment of writing this.

> I had some issues with the network, for example, I couldn't connect to Activision games servers (CoD: MW, Overwatch) but I've changed firewall settings from public to private and everything is good for now.

> For the first time, I had windows on .raw image and disk was throttling a lot, I've set up raid0 on my 2 HDD's, then I created 3 partitions with LVM - 120GB for windows, 700GB for data(games), 700gb for Linux data and passthrough two of partition as Virtio-BLK. RAID&LVM

> Audio passthrough is done through the usual PulseAudio solution, works nicely.

> For some people who, maybe looking how to passthrough GPU - because it's not obvious when you doing it for the first time and it's not on the wiki though, so when you pass a correct group of vfio-pci.ids then you need to add in (easiest way) a Virtual Machine Manager - Add hardware - PCI Host Device - You graphic card (for me it was 0000:01:00:0 NVIDIA Corporation GP106 [GeForce GTX 1060 6GB]).

peterge's VFIO setup

Hardware:

  • CPU: AMD Ryzen 5 1600X
  • Motherboard: MSI x370 Gaming Pro Carbon
  • GPU: SAPPHIRE PULSE Radeon RX 5700 XT (Host), Asus ROG Strix GeForce GTX 1080 (VM)
  • RAM: 16GB

Configuration:

  • Kernel: 5.4.24-1-MANJARO
  • Using libvirt/QEMU: Xml File
  • No issues right now.

> Mouse and keyboard passthrough with evdev.

> Monitor connected to both GPUs to display native 144hz.

> Needed this option in /etc/modprobe.d/kvm.conf, related to my Ryzen CPU: options kvm ignore_msrs

> Had problems passing sound to host with a ICH6 device, sound was crackeling and a bit delayed. Then switched to scream-ivshmem and sound is working perfectly now. Its great in combination with looking-glass.

> Tried to build a solution where i can use my main GPU (GTX 1080) on Host and VM, by rebinding drivers. Rebinding drivers worked on the GTX 1080, but it was not able to use it to offload games from my host gpu back then Reddit Post (Got it working with an R9 380 for the host, but needed to restart xorg every time). Then bought the RX 5700 XT, where GPU offloading worked like a charm, but due to the early stage of driver support driver rebinding is not supported Reddit Post. So i currently have 2 GPUs in my rig.

Adding your own setup

Add a new section with your nickname, CPU, motherboard and GPU models, then copy and paste this template to your section:

Hardware:

* '''CPU''': 
* '''Motherboard''': (Revision , BIOS/UEFI Version: )
* '''GPU''': 
* '''RAM''': 

Configuration:

* '''Kernel''': Kernel version (vanilla/CK/Zen/ACS-patched or not).
* Using '''libvirt/QEMU''': link to domain XMLs/scripts/notes (Git repo preferred).
* Issues you have encountered, special steps taken to make something work a bit better, etc.
* Describe your setup loosely here, so that when other wiki users are looking for something, they can easily skim through available setups.

Replace proper sections with your own data. Make sure to provide the exact motherboard model, revision (if possible - should be on both the motherboard itself and the box it came in) and BIOS/UEFI version you are using. Describe your exact software setup and add a link to your configuration files. (GitHub, GitLab, BitBucket, etc can host a public repository which you may update once in a while, but uploading them to pastebins is fine, too. Do not post the entire config file contents here.)