User:Bai-Chiang/VFIO notes

From ArchWiki

A simple VFIO setup guide. Requirements: a spare set of monitor keyboard and mouse. The extra monitor will connect to the GPU to be passthroughed, and keyboard and mouse will passthrough as usb device.

Isolate the GPU

Following the wiki section:

Prepare virtualization platform

Following the wiki section:

Install guest

I have some tested scripts in this repo.

Linux guest

Download Linux ISO to /var/lib/libvirt/images/linux.iso. The location is chosen such that libvirt could access the directory.

Run following command to start installation. The BIOS menu should appear on the extra monitor connect to passthroughed GPU. 

# virt-install \
    --name vm_name \
    --memory 8192 \
    --sysinfo host \
    --cpu host-passthrough,cache.mode=passthrough,topology.sockets=1,topology.cores=4,topology.threads=2 \
    --os-variant name='ubuntu22.04' \
    --cdrom "/var/lib/libvirt/images/linux.iso" \
    --network network=default,model.type=virtio \
    --graphics none \
    --noautoconsole \
    --boot uefi,bootmenu.enable='yes' \
    --features smm.state=on,kvm.hidden.state=on \
    --disk path="/var/lib/libvirt/images/vm_name.qcow2",size=128,bus=virtio \
    --hostdev 01:00.0,type=pci \
    --hostdev 01:00.1,type=pci \
    --hostdev 0x046d:0xc542,type=usb \
    --hostdev 0x046d:0x2010,type=usb \
    --xml xpath.create=./features/hyperv/vendor_id \
    --xml xpath.set=./features/hyperv/vendor_id/@state=on \
    --xml xpath.set=./features/hyperv/vendor_id/@value=whateverid \
    --xml xpath.create=./os/firmware/feature \
    --xml xpath.set=./os/firmware/feature/@enabled=no \
    --xml xpath.set=./os/firmware/feature/@name=secure-boot \
    --iothreads 1 \
   --cputune vcpupin0.vcpu=0,vcpupin0.cpuset=2,vcpupin1.vcpu=1,vcpupin1.cpuset=8,vcpupin2.vcpu=2,vcpupin2.cpuset=3,vcpupin3.vcpu=3,vcpupin3.cpuset=9,vcpupin4.vcpu=4,vcpupin4.cpuset=4,vcpupin5.vcpu=5,vcpupin5.cpuset=10,vcpupin6.vcpu=6,vcpupin6.cpuset=5,vcpupin7.vcpu=7,vcpupin7.cpuset=11 \
   --xml xpath.create=./cputune/emulatorpin \
   --xml xpath.set=./cputune/emulatorpin/@cpuset='0,6' \
   --xml xpath.create=./cputune/iothreadpin \
   --xml xpath.set=./cputune/iothreadpin/@iothread=1 \
   --xml xpath.set=./cputune/iothreadpin/@cpuset='0,6'
Note: If you are passing Nvidia laptop GPU, create /var/lib/libvirt/qemu/SSDT1.dat following PCI passthrough via OVMF#"Error 43: Driver failed to load" with mobile (Optimus/max-q) nvidia GPUs. Then add these args to previous command 
--qemu-commandline="-acpitable" \
--qemu-commandline=file=/var/lib/libvirt/qemu/SSDT1.dat
  • --os-variant use osinfo-query os command list all os variants name
  • --features kvm.hidden.state=on and --xml xpath.set=./features/hyperv/vendor_id/.. will prevent Video card driver virtualisation detection
  • --xml xpath.set=./os/firmware/feature/.. will disable secure boot
  • --sysinfo host will exposes the host's BIOS info to the VM.
  • --network network=default,model.type=virtio will attach the VM to bridge named default using virtio driver.
  • --hostdev 01:00.0,type=pci passthrough PCI device with address 01:00.0. (get the address from lspci)
  • --hostdev 0x046d:0xc542,type=usb passthrough USB device by vendor and product. (get from lsusb)
  • From --cputune line to the end set CPU pinning following the example in the wiki, change according to your CPU topology.

Windows guest

Download Windows ISO to /var/lib/libvirt/images/windows.iso and latest windows virtio driver ISO image to /var/lib/libvirt/images/virtio-win.iso.

Similarly, start VM with command 

# virt-install \
    --name windows \
    --memory 8192 \
    --sysinfo host \
    --cpu host-passthrough,cache.mode=passthrough,topology.sockets=1,topology.cores=4,topology.threads=2 \
    --os-variant name=win11 \
    --cdrom "/var/lib/libvirt/images/windows.iso" \
    --disk path="/var/lib/libvirt/images/virtio-win.iso",device=cdrom,bus=sata \
    --network network=default,model.type=virtio \
    --graphics none \
    --noautoconsole \
    --boot uefi,bootmenu.enable='yes' \
    --features smm.state=on,kvm.hidden.state=on \
    --tpm default \
    --disk path="/var/lib/libvirt/images/win.qcow2",size=256,bus=virtio \
    --hostdev 01:00.0,type=pci \
    --hostdev 01:00.1,type=pci \
    --hostdev 0x046d:0xc542,type=usb \
    --hostdev 0x24ae:0x2010,type=usb \
    --xml xpath.create=./features/hyperv/vendor_id \
    --xml xpath.set=./features/hyperv/vendor_id/@state=on \
    --xml xpath.set=./features/hyperv/vendor_id/@value=whateverid \
    --xml xpath.create=./os/firmware/feature \
    --xml xpath.set=./os/firmware/feature/@enabled=yes \
    --xml xpath.set=./os/firmware/feature/@name=secure-boot 
    --iothreads 1 \
   --cputune vcpupin0.vcpu=0,vcpupin0.cpuset=2,vcpupin1.vcpu=1,vcpupin1.cpuset=8,vcpupin2.vcpu=2,vcpupin2.cpuset=3,vcpupin3.vcpu=3,vcpupin3.cpuset=9,vcpupin4.vcpu=4,vcpupin4.cpuset=4,vcpupin5.vcpu=5,vcpupin5.cpuset=10,vcpupin6.vcpu=6,vcpupin6.cpuset=5,vcpupin7.vcpu=7,vcpupin7.cpuset=11 \
   --xml xpath.create=./cputune/emulatorpin \
   --xml xpath.set=./cputune/emulatorpin/@cpuset='0,6' \
   --xml xpath.create=./cputune/iothreadpin \
   --xml xpath.set=./cputune/iothreadpin/@iothread=1 \
   --xml xpath.set=./cputune/iothreadpin/@cpuset='0,6'
    
    # Following args are for laptop Nvidia GPU, see #Linux guest section
    --qemu-commandline="-acpitable" \
    --qemu-commandline=file=/var/lib/libvirt/qemu/SSDT1.dat
  • TPM requires the swtpm package.
  • Since --sysinfo exposed host BIOS and system information to the guest, if the computer comes with windows OEM activation, use following command to get the OEM key, and use it during installation process.
# strings /sys/firmware/acpi/tables/MSDM | tail -1
  • With virtio storage, --disk bus=virtio, during windows partition click Load driver, and select driver with correct windows version. This will install driver from virtio-win.iso mounted as CD drive.
  • Since the network driver also using virtio, there is no internet during Windows setup, which helps bypassing Microsoft account.
  • After installation, navigate to the CD drive directory, and install virtio driver, which includes ethernet driver.
Retrieved from "https://wiki.archlinux.org/index.php?title=User:Bai-Chiang/VFIO_notes&oldid=795949"