https://wiki.archlinux.org/api.php?action=feedcontributions&user=NeoTheFox&feedformat=atomArchWiki - User contributions [en]2024-03-29T10:59:37ZUser contributionsMediaWiki 1.41.0https://wiki.archlinux.org/index.php?title=Chromium&diff=713557Chromium2022-01-25T10:45:50Z<p>NeoTheFox: Add Vivaldi to related pages (chromium based)</p>
<hr />
<div>[[Category:Web browser]]<br />
[[Category:Google]]<br />
[[de:Chromium]]<br />
[[es:Chromium]]<br />
[[ja:Chromium]]<br />
[[zh-hans:Chromium]]<br />
{{Related articles start}}<br />
{{Related|Browser extensions}}<br />
{{Related|Firefox}}<br />
{{Related|Opera}}<br />
{{Related|Vivaldi}}<br />
{{Related articles end}}<br />
<br />
[[Wikipedia:Chromium (web browser)|Chromium]] is an open-source graphical web browser based on the [[Wikipedia:Blink (web engine)|Blink]] rendering engine. It is the basis for the proprietary Google Chrome browser.<br />
<br />
See [https://chromium.googlesource.com/chromium/src/+/master/docs/chromium_browser_vs_google_chrome.md this page] for an explanation of the differences between Chromium and Google Chrome. Additionally:<br />
<br />
* Sync is unavailable in Chromium 89+ (2021-03-02) [https://archlinux.org/news/chromium-losing-sync-support-in-early-march/]<br />
<br />
{{Note|Sync can be temporarily restored by [https://gist.github.com/foutrelis/14e339596b89813aa9c37fd1b4e5d9d5 using Chrome's OAuth2 credentials] or [https://www.chromium.org/developers/how-tos/api-keys getting your own], but pay attention to the disclaimers and do not consider this to be a long-term solution.<br />
Consider switching to [https://www.xbrowsersync.org xbrowsersync] for bookmarks syncing as long term solution.<br />
}}<br />
<br />
See [[List of applications/Internet#Blink-based]] for other browsers based on Chromium.<br />
<br />
== Installation ==<br />
<br />
There are several packages available to [[install]] Chromium with:<br />
<br />
* {{Pkg|chromium}} — stable release;<br />
* {{AUR|chromium-dev}} — development release;<br />
* {{AUR|chromium-snapshot-bin}} — nightly build.<br />
<br />
Google Chrome packages:<br />
<br />
* {{AUR|google-chrome}} — stable release;<br />
* {{AUR|google-chrome-beta}} — beta release;<br />
* {{AUR|google-chrome-dev}} — development release.<br />
<br />
{{Note|From the [https://www.chromium.org/Home/chromium-privacy Chromium privacy page]: "Features that communicate with Google made available through the compilation of code in Chromium are subject to the [https://www.google.com/policies/privacy/ Google Privacy Policy]." For those who want to avoid all integration with Google services, there are some [[List of applications/Internet#Privacy-focused chromium spin-offs|privacy-focused spin-offs]].}}<br />
<br />
== Configuration ==<br />
<br />
=== Default applications ===<br />
<br />
To set Chromium as the default browser and to change which applications Chromium launches when opening downloaded files, see [[default applications]].<br />
<br />
=== Certificates ===<br />
<br />
Chromium uses [[Network Security Services]] for certificate management. Certificates can be managed in {{ic|chrome://settings/certificates}}.<br />
<br />
=== Making flags persistent ===<br />
<br />
{{Note|The {{ic|chromium-flags.conf}} file and the accompanying custom launcher script are specific to the Arch Linux {{Pkg|chromium}} package. For {{AUR|google-chrome}} and {{AUR|google-chrome-dev}}, use {{ic|chrome-flags.conf}} and {{ic|chrome-dev-flags.conf}} instead.}}<br />
<br />
You can put your flags in a {{ic|chromium-flags.conf}} file under {{ic|$HOME/.config/}} (or under {{ic|$XDG_CONFIG_HOME}} if you have configured that environment variable).<br />
<br />
No special syntax is used; flags are defined as if they were written in a terminal.<br />
<br />
* The arguments are split on whitespace and shell quoting rules apply, but no further parsing is performed.<br />
* In case of improper quoting anywhere in the file, a fatal error is raised.<br />
* Flags can be placed in separate lines for readability, but this is not required.<br />
* Lines starting with a hash symbol (#) are skipped. (This is only supported by the {{Pkg|chromium}} launcher script and will not work when using {{ic|chrome-flags.conf}} with the {{AUR|google-chrome}} package.)<br />
<br />
Below is an example {{ic|chromium-flags.conf}} file that defines the flags {{ic|--start-maximized --incognito}}:<br />
<br />
{{hc|~/.config/chromium-flags.conf|<br />
# This line will be ignored.<br />
--start-maximized<br />
--incognito<br />
}}<br />
<br />
=== Force GPU acceleration ===<br />
<br />
{{Warning|Disabling the rendering blacklist may cause unstable behavior, including crashes of the host. See the bug reports in {{ic|chrome://gpu}} for details.}}<br />
<br />
By default Chromium on Linux does not use any GPU acceleration. To force GPU acceleration, [[append]] the following flags to [[/Tips and tricks#Making flags persistent|persistent configuration]]:<br />
<br />
{{hc|~/.config/chromium-flags.conf|<br />
--ignore-gpu-blocklist<br />
--enable-gpu-rasterization<br />
--enable-zero-copy<br />
}}<br />
<br />
Additionally the flag {{ic|--disable-gpu-driver-bug-workarounds}} may need to be passed to prevent GPU workaround from being used. Flags in {{ic|chrome://gpu}} should state "Hardware accelerated" when configured and available.<br />
<br />
{{Out of date|A fix has been merged into mesa as of May 2021. [https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10850]}}<br />
<br />
{{ic|--enable-native-gpu-memory-buffers}} is broken since mesa 20.1.1 [https://gitlab.freedesktop.org/mesa/mesa/-/issues/3119#note_533902]<br />
<br />
=== Hardware video acceleration ===<br />
<br />
{{Note|1=<nowiki/><br />
* There is no official support from Chromium or Arch Linux for this feature [https://chromium.googlesource.com/chromium/src/+/master/docs/gpu/vaapi.md#vaapi-on-linux]. However, {{Pkg|chromium}} from official repositories is compiled with VA-API support and you may ask for help in [https://bbs.archlinux.org/viewtopic.php?id=244031 the dedicated forum thread].<br />
* VA-API does not work with the {{Pkg|chromium}} package when using the native Wayland backend, but it does work in {{AUR|chromium-wayland-vaapi}}.<br />
}}<br />
<br />
To enable VA-API support in Chromium:<br />
<br />
* Install the correct VA-API driver for your video card and verify VA-API has been enabled and working correctly, see [[Hardware video acceleration]]. For proprietary NVIDIA support, installing {{AUR|libva-vdpau-driver-chromium}} or {{AUR|libva-vdpau-driver-vp9-git}} is required.<br />
* Set the option {{ic|1=--enable-features=VaapiVideoDecoder}}. This is enough when using ANGLE GL renderer and {{Pkg|libva-intel-driver}}.<br />
* When using ANGLE, Chromium forces the older i965 driver and fails when {{Pkg|intel-media-driver}} is used. As a workaround, [[Hardware video acceleration#Configuring VA-API|configure VA-API manually]]. See [https://github.com/intel/media-driver/issues/818] for details.<br />
* To use the system GL renderer on Xorg, use either {{ic|1=--use-gl=egl}} or {{ic|1=--use-gl=desktop}}. On XWayland, use the {{ic|1=--use-gl=egl}} flag (Currently exhibits choppiness {{Bug|67035}} on some systems).<br />
<br />
==== Tips and tricks ====<br />
<br />
To check if it's working play a video which is using a codec supported by your VA-API driver (''vainfo'' tells you which codecs are supported, but Chromium will only support VP9 and h264):<br />
<br />
* Open the DevTools by pressing {{ic|Ctrl+Shift+I}} or on the ''Inspect'' button of the context (right-click) menu<br />
* Add the Media inspection tab: ''Hamburger menu > More tools > Media''<br />
* In the newly opened Media tab, look at the hardware decoder state of the video decoder<br />
<br />
Test on a large enough video. Starting with version 86, Chromium on desktop [https://bugs.chromium.org/p/chromium/issues/detail?id=684792 will only accelerate videos larger than 720p].<br />
<br />
To reduce CPU usage while watching YouTube where VP8/VP9 hardware decoding is not available use the [https://chrome.google.com/webstore/detail/h264ify/aleakchihdccplidncghkekgioiakgal h264ify] or [https://chrome.google.com/webstore/detail/enhanced-h264ify/omkfmpieigblcllmkgbflkikinpkodlk enhanced-h264ify] extension.<br />
<br />
On some systems (especially on XWayland) you might need to [[#Force GPU acceleration]]. Only {{ic|--ignore-gpu-blocklist}} is enough for our purposes.<br />
<br />
{{Expansion|Provide a link to some bug report.}}<br />
<br />
You might need to disable the Skia renderer, as it is currently not compatible with video decode acceleration: {{ic|1=--disable-features=UseSkiaRenderer}}<br />
<br />
=== PDF viewer plugin ===<br />
<br />
Chromium and Google Chrome are bundled with the ''Chromium PDF Viewer'' plugin. If you do not want to use this plugin, check ''Open PDFs using a different application'' in {{ic|chrome://settings/content/pdfDocuments}}.<br />
<br />
=== Flash Player plugin ===<br />
<br />
Support for Flash Player was removed in Chromium 88.[https://www.chromium.org/flash-roadmap#TOC-Flash-Support-Removed-from-Chromium-Target:-Chrome-88---Jan-2021-]<br />
<br />
=== Running on XWayland ===<br />
<br />
If you are using NVIDIA's proprietary driver, running Chromium on XWayland may cause the GPU process to occasionally crash. To prevent the GPU process from crashing, add the following flags:<br />
<br />
--use-angle=vulkan --use-cmd-decoder=passthrough<br />
<br />
{{Note|This does not prevent all XWayland-related crashes.}}<br />
<br />
=== Native Wayland support ===<br />
<br />
Since version 97, native [[Wayland]] support in Chromium can be enabled with the following flags [https://chromium.googlesource.com/chromium/src/+/43cfb2f92a5cdc1a787d7326e74676884abf5052]:<br />
<br />
--ozone-platform-hint=auto<br />
<br />
See [[#Making flags persistent]] for a permanent configuration. The flag is also available via [[#chrome://_URLs|browser flags menu]].<br />
<br />
This will select wayland Ozone backend when in wayland session, so you can use a single desktop entry if you switch between X11 and Wayland often.<br />
<br />
== Tips and tricks ==<br />
<br />
The following tips and tricks should work for both Chromium and Chrome unless explicitly stated.<br />
<br />
=== Browsing experience ===<br />
<br />
==== chrome:// URLs ====<br />
<br />
A number of tweaks can be accessed via Chrome URLs. See '''chrome://chrome-urls''' for a complete list.<br />
<br />
* '''chrome://flags''' - access experimental features such as WebGL and rendering webpages with GPU, etc.<br />
* '''chrome://extensions''' - view, enable and disable the currently used Chromium extensions.<br />
* '''chrome://gpu''' - status of different GPU options.<br />
* '''chrome://sandbox''' - indicate sandbox status.<br />
* '''chrome://version''' - display version and switches used to invoke the active {{ic|/usr/bin/chromium}}.<br />
<br />
An automatically updated, complete listing of Chromium switches (command line parameters) is available [https://peter.sh/experiments/chromium-command-line-switches/ here].<br />
<br />
==== Chromium task manager ====<br />
<br />
Shift+ESC can be used to bring up the browser task manager wherein memory, CPU, and network usage can be viewed.<br />
<br />
==== Chromium overrides/overwrites Preferences file ====<br />
<br />
If you enabled syncing with a Google Account, then Chromium will override any direct edits to the Preferences file found under {{ic|~/.config/chromium/Default/Preferences}}. To work around this, start Chromium with the {{ic|--disable-sync-preferences}} switch:<br />
$ chromium --disable-sync-preferences<br />
<br />
If Chromium is started in the background when you login in to your desktop environment, make sure the command your desktop environment uses is:<br />
$ chromium --disable-sync-preferences --no-startup-window<br />
<br />
==== Search engines ====<br />
<br />
Make sites like [https://wiki.archlinux.org wiki.archlinux.org] and [https://en.wikipedia.org wikipedia.org] easily searchable by first executing a search on those pages, then going to ''Settings > Search'' and click the ''Manage search engines..'' button. From there, "Edit" the Wikipedia entry and change its keyword to '''w''' (or some other shortcut you prefer). Now searching Wikipedia for "Arch Linux" from the address bar is done simply by entering "'''w arch linux'''".<br />
<br />
{{Note| Google search is used automatically when typing something into the URL bar. A hard-coded keyword trigger is also available using the '''?''' prefix.}}<br />
<br />
==== Tmpfs ====<br />
<br />
===== Cache in tmpfs =====<br />
<br />
{{Note|Chromium stores its cache separate from its browser profile directory.}}<br />
<br />
To limit Chromium from writing its cache to a physical disk, one can define an alternative location via the {{ic|--disk-cache-dir}} flag:<br />
$ chromium --disk-cache-dir="$XDG_RUNTIME_DIR/chromium-cache"<br />
<br />
Cache should be considered temporary and will '''not''' be saved after a reboot or hard lock. Another option is to setup the space in {{ic|/etc/fstab}}:<br />
<br />
{{hc|/etc/fstab|2=<br />
tmpfs /home/''username''/.cache tmpfs noatime,nodev,nosuid,size=400M 0 0<br />
}}<br />
<br />
===== Profile in tmpfs =====<br />
<br />
Relocate the browser profile to a [[Wikipedia:Tmpfs|tmpfs]] filesystem, including {{ic|/tmp}}, or {{ic|/dev/shm}} for improvements in application response as the entire profile is now stored in RAM.<br />
<br />
Use an active profile management tool such as {{Pkg|profile-sync-daemon}} for maximal reliability and ease of use. It symlinks or bind mounts and syncs the browser profile directories to RAM. For more, see [[Profile-sync-daemon]].<br />
<br />
==== Launch a new browser instance ====<br />
<br />
When you launch the browser, it first checks if another instance using the same data directory is already running. If there is one, the new window is associated with the old instance. If you want to launch an independent instance of the browser, you must specify separate directory using the {{ic|--user-data-dir}} parameter:<br />
<br />
$ chromium --user-data-dir=''/path/to/some/directory''<br />
<br />
{{Note|The default location of the user data is {{ic|~/.config/chromium/}}.}}<br />
<br />
==== Directly open *.torrent files and magnet links with a torrent client ====<br />
<br />
By default, Chromium downloads {{ic|*.torrent}} files directly and you need to click the notification from the bottom-left corner of the screen in order for the file to be opened with your default torrent client. This can be avoided with the following method:<br />
<br />
* Download a {{ic|*.torrent}} file.<br />
* Right-click the notification displayed at the bottom-left corner of the screen.<br />
* Check the "''Always Open Files of This Type''" checkbox.<br />
<br />
See [[xdg-open]] to change the default assocation.<br />
<br />
==== Touch Scrolling on touchscreen devices ====<br />
<br />
You may need to specify which touch device to use. Find your touchscreen device with {{ic| xinput list}} then launch Chromium with the {{ic|1=--touch-devices='''x'''}} parameter, where "'''x'''" is the id of your device. {{Note|If the device is designated as a slave pointer, using this may not work, use the master pointer's ID instead.}}<br />
<br />
==== Reduce memory usage ====<br />
<br />
By default, Chromium uses a separate OS process for each ''instance'' of a visited web site. [https://www.chromium.org/developers/design-documents/process-models#Supported_Models] However, you can specify command-line switches when starting Chromium to modify this behaviour.<br />
<br />
For example, to share one process for all instances of a website:<br />
<br />
$ chromium --process-per-site<br />
<br />
To use a single process model:<br />
<br />
$ chromium --single-process<br />
<br />
{{Warning|The single-process model is discouraged because it is unsafe and may contain bugs not present in other models.[https://www.chromium.org/developers/design-documents/process-models#TOC-Single-process]}}<br />
<br />
In addition, you can suspend or store inactive Tabs with extensions such as [https://chrome.google.com/webstore/detail/tab-suspender/fiabciakcmgepblmdkmemdbbkilneeeh?hl=en Tab Suspender] and [https://chrome.google.com/webstore/detail/onetab/chphlpgkkbolifaimnlloiipkdnihall?hl=en OneTab].<br />
<br />
==== User Agent ====<br />
<br />
The User Agent can be arbitrarily modified at the start of Chromium's base instance via its {{Ic|<nowiki>--user-agent="[string]"</nowiki>}} parameter.<br />
<br />
==== DOM Distiller ====<br />
<br />
Chromium has a similar reader mode to Firefox. In this case it's called DOM Distiller, which is an [https://github.com/chromium/dom-distiller open source project].<br />
It is disabled by default, but can be enabled using the {{Ic|chrome://flags/#enable-reader-mode}} flag, which you can also make [[#Making flags persistent|persistent]].<br />
Not only does DOM Distiller provide a better reading experience by distilling the content of the page, it also simplifies pages for print. Even though the latter checkbox option has been removed from the print dialog, you can still print the distilled page, which basically has the same effect.<br />
<br />
After enabling the flag, you will find a new "Toggle reader mode" menu item and corresponding icon in the address bar when Chromium thinks the website you are visiting could do with some distilling.<br />
<br />
==== Forcing specific GPU ====<br />
<br />
In multi-GPU systems, Chromium automatically detects which GPU should be used for rendering (discrete or integrated). This works 99% of the time, except when it does not - if a unavailable GPU is picked (for example, discrete graphics on VFIO GPU passthrough-enabled systems), {{ic|chrome://gpu}} will complain about not being able to initialize the GPU process. On the same page below '''Driver Information''' there will be multiple GPUs shown (GPU0, GPU1, ...). There is no way to switch between them in a user-friendly way, but you can read the device/vendor IDs present there and configure Chromium to use a specific GPU with flags:<br />
<br />
$ chromium --gpu-testing-vendor-id=0x8086 --gpu-testing-device-id=0x1912<br />
<br />
...where {{ic|0x8086}} and {{ic|0x1912}} is replaced by the IDs of the GPU you want to use (as shown on the {{ic|chrome://gpu}} page).<br />
<br />
==== Import bookmarks from Firefox ====<br />
<br />
To ease the transition, you can import bookmarks from [[Firefox]] into Chromium.<br />
<br />
Navigate Chromium to {{ic|chrome://settings/importData}}<br />
<br />
If Firefox is already installed on your computer, you can directly import bookmarks as well as many other things from Firefox.<br />
<br />
Make sure '''Mozilla Firefox''' is selected. Optionally, you can uncheck some unwanted items here. Click the '''Import''' and then '''Done'''. You are done with it.<br />
<br />
{{note|If you have not created any bookmarks in Chromium yet, the bookmarks will show up in your bookmarks bar. If you already have bookmarks, the bookmarks will be in a new folder labeled "Imported From Firefox"}}<br />
<br />
If you import bookmarks from another PC, you have to export bookmarks from Firefox first.<br />
<br />
{{ic|''Ctrl + Shift + O > Import and Backup > Export Bookmarks To HTML}} in Firefox''<br />
<br />
The procedure is pretty much the same. You need to go to {{ic|chrome://settings/importData}}. However, this time, in the '''From''' drop-down menu, select '''Bookmarks HTML File''' and click the '''Choose File''' button and upload the desired bookmark file.<br />
<br />
==== Enabling native notifications ====<br />
<br />
Go to {{ic|chrome://flags#enable-system-notifications}} and select ''Enabled''.<br />
<br />
==== U2F authentication ====<br />
<br />
Install {{Pkg|libfido2}} library. This provides the udev rules required to enable access to the [[U2F]] key as a user.<br />
U2F keys are by default only accessible by root, and without these rules Chromium will give an error.<br />
<br />
==== Dark mode ====<br />
<br />
To enable dark mode (used in ''prefers-color-scheme'' in CSS, JavaScript, Settings and Dev-Tools) and enable the dark theme (normally used for incognito mode) [[append]] the following flag to [[#Making flags persistent|persistent configuration]]:<br />
<br />
{{hc|1=~/.config/chromium-flags.conf|2=<br />
--force-dark-mode<br />
--enable-features=WebUIDarkMode<br />
}}<br />
<br />
===== Dark mode by system preference =====<br />
<br />
[https://bugs.chromium.org/p/chromium/issues/detail?id=998903 This Chromium issue] aims to bring dark mode based on GTK theme selection into Chromium.<br />
<br />
In the future, all that will be required to properly use system preference, is setting ''Designs'' to GTK in {{ic|chrome://settings/appearance}}.<br />
<br />
==== Enable Side Panel ====<br />
<br />
The Side Panel can be enabled through {{ic|chrome://flags}}. You can enable or disable '''Side panel''', and change options such as '''Side panel border''' and '''Side panel drag and drop'''.<br />
<br />
=== Profile maintenance ===<br />
<br />
Chromium uses [[SQLite]] databases to manage history and the like. Sqlite databases become fragmented over time and empty spaces appear all around. But, since there are no managing processes checking and optimizing the database, these factors eventually result in a performance hit. A good way to improve startup and some other bookmarks- and history-related tasks is to defragment and trim unused space from these databases.<br />
<br />
{{Pkg|profile-cleaner}} and {{AUR|browser-vacuum}} in the [[AUR]] do just this.<br />
<br />
=== Security ===<br />
<br />
==== Disable JIT ====<br />
<br />
At the cost of reduced performance, you can disable just-in-time compilation of JavaScript to native code, which is responsible for [https://microsoftedge.github.io/edgevr/posts/Super-Duper-Secure-Mode/ roughtly half of the security vulnerabilities in the JS engine], using the flag {{ic|1=--js-flags=--jitless}}.<br />
<br />
==== WebRTC ====<br />
<br />
WebRTC is a communication protocol that relies on JavaScript that can leak one's actual IP address and hardware hash from behind a VPN. While some software may prevent the leaking scripts from running, it's probably a good idea to block this protocol directly as well, just to be safe. As of October 2016, there is no way to disable WebRTC on Chromium on desktop, there are extensions available to disable local IP address leak, one is this [https://chrome.google.com/webstore/detail/webrtc-network-limiter/npeicpdbkakmehahjeeohfdhnlpdklia extension].<br />
<br />
One can test WebRTC via https://browserleaks.com/webrtc.<br />
<br />
{{Warning|Even though IP leak can be prevented, Chromium still sends your unique hash, and there is no way to prevent this. Read more on https://www.browserleaks.com/webrtc#webrtc-disable}}<br />
<br />
==== SSL certificates ====<br />
<br />
Chromium does not have an SSL certificate manager. It relies on the NSS Shared DB {{ic|~/.pki/nssdb}}. In order to add SSL certificates to the database, users will have to use the shell.<br />
<br />
===== Adding CAcert certificates for self-signed certificates =====<br />
<br />
Grab the CAcerts and create an {{ic|nssdb}}, if one does not already exist. To do this, first install the {{Pkg|nss}} package, then complete these steps:<br />
<br />
$ mkdir -p $HOME/.pki/nssdb<br />
$ cd $HOME/.pki/nssdb<br />
$ certutil -N -d sql:.<br />
<br />
$ curl -k -o "cacert-root.crt" "http://www.cacert.org/certs/root.crt"<br />
$ curl -k -o "cacert-class3.crt" "http://www.cacert.org/certs/class3.crt"<br />
$ certutil -d sql:$HOME/.pki/nssdb -A -t TC -n "CAcert.org" -i cacert-root.crt <br />
$ certutil -d sql:$HOME/.pki/nssdb -A -t TC -n "CAcert.org Class 3" -i cacert-class3.crt<br />
<br />
{{Note|Users will need to create a password for the database, if it does not exist.}}<br />
<br />
Now users may manually import a self-signed certificate.<br />
<br />
===== Example 1: Using a shell script to isolate the certificate from TomatoUSB =====<br />
<br />
Below is a simple script that will extract and add a certificate to the user's {{ic|nssdb}}:<br />
<br />
#!/bin/sh<br />
#<br />
# usage: import-cert.sh remote.host.name [port]<br />
#<br />
REMHOST=$1<br />
REMPORT=${2:-443}<br />
exec 6>&1<br />
exec > $REMHOST<br />
echo | openssl s_client -connect ${REMHOST}:${REMPORT} 2>&1 |sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'<br />
certutil -d sql:$HOME/.pki/nssdb -A -t "P,," -n "$REMHOST" -i $REMHOST <br />
exec 1>&6 6>&-<br />
<br />
Syntax is advertised in the commented lines.<br />
<br />
References:<br />
*https://web.archive.org/web/20180718193807/https://blog.avirtualhome.com/adding-ssl-certificates-to-google-chrome-linux-ubuntu<br />
*https://chromium.googlesource.com/chromium/src/+/master/docs/linux/cert_management.md<br />
<br />
===== Example 2: Using Firefox to isolate the certificate from TomatoUSB =====<br />
<br />
The {{Pkg|firefox}} browser can be used to save the certificate to a file for manual import into the database.<br />
<br />
Using firefox:<br />
#Browse to the target URL.<br />
#Upon seeing the "This Connection is Untrusted" warning screen, click: ''I understand the Risks > Add Exception...''<br />
#Click: ''View > Details > Export'' and save the certificate to a temporary location ({{ic|/tmp/easy.pem}} in this example).<br />
<br />
Now import the certificate for use in Chromium:<br />
$ certutil -d sql:$HOME/.pki/nssdb -A -t TC -n "easy" -i /tmp/easy.pem<br />
<br />
{{Note|Adjust the name to match that of the certificate. In the example above, "easy" is the name of the certificate.}}<br />
<br />
Reference:<br />
*https://sahissam.blogspot.com/2012/06/new-ssl-certificates-for-tomatousb-and.html<br />
<br />
==== Canvas Fingerprinting ====<br />
<br />
Canvas fingerprinting is a technique that allows websites to identify users by detecting differences when rendering to an HTML5 canvas. This information can be made inaccessible by using the {{ic|--disable-reading-from-canvas}} flag.<br />
<br />
To confirm this is working run [https://panopticlick.eff.org this test] and make sure "hash of canvas fingerprint" is reported as undetermined in the full results.<br />
<br />
{{Note|1=<nowiki></nowiki><br />
* Some extensions require reading from canvas and may be broken by setting {{ic|--disable-reading-from-canvas}}.<br />
* YouTube player does not work properly without canvas reading. [https://github.com/qutebrowser/qutebrowser/issues/5345][https://bbs.archlinux.org/viewtopic.php?pid=1907406]<br />
}}<br />
<br />
==== Privacy extensions ====<br />
<br />
See [[Browser extensions#Privacy]].<br />
<br />
{{Tip|Installing too many extensions might take up much space in the toolbar. Those extensions which you would not interact with anyway (e.g. [https://chrome.google.com/webstore/detail/gcbommkclmclpchllfjekcdonpmejbdp HTTPS Everywhere]) can be hidden by right-clicking on the extension and choosing ''Hide in Chromium menu''.}}<br />
<br />
==== Do Not Track ====<br />
<br />
To enable [[wikipedia:Do Not Track|Do Not Track]], visit {{ic|chrome://settings}}, scroll down to ''Advanced'' and under ''Privacy and security'', check ''Send a "Do Not Track" request with your browsing traffic''.<br />
<br />
==== Force a password store ====<br />
<br />
Chromium uses a password store to store your passwords and the ''Chromium Safe Storage'' key, which is used to encrypt cookie values. [https://codereview.chromium.org/24734007]<br />
<br />
By default Chromium auto-detects which password store to use, which can lead to you apparently losing your passwords and cookies when switching to another desktop environment or window manager.<br />
<br />
You can force Chromium to use a specific password store by launching it with the {{ic|--password-store}} flag with one of following the values [https://chromium.googlesource.com/chromium/src/+/master/docs/linux/password_storage.md]:<br />
<br />
* {{ic|gnome}}, uses [[Gnome Keyring]]<br />
* {{ic|kwallet5}}, uses [[KDE Wallet]]<br />
* {{ic|basic}}, saves the passwords and the cookies' encryption key as plain text in the file {{ic|Login Data}}<br />
* {{ic|detect}}, the default auto-detect behavior<br />
<br />
For example, to force Chromium to use Gnome Keyring in another desktop or WM use {{ic|1=--password-store=gnome}}, see [[#Making flags persistent]] for making it permanent.<br />
<br />
When using a password store of another desktop environment you probably also want to unlock it automatically. See [[GNOME/Keyring#Using the keyring]] and [[KDE Wallet#Unlock KDE Wallet automatically on login]].<br />
<br />
== Troubleshooting ==<br />
<br />
=== Fonts ===<br />
<br />
{{Note|Chromium does not fully integrate with fontconfig/GTK/Pango/X/etc. due to its sandbox. For more information, see the [https://dev.chromium.org/developers/linux-technical-faq Linux Technical FAQ].}}<br />
<br />
==== Tab font size is too large ====<br />
<br />
Chromium will use the GTK settings as described in [[GTK#Configuration]]. When configured, Chromium will use the {{ic|gtk-font-name}} setting for tabs (which may mismatch window font size). To override these settings, use {{ic|1=--force-device-scale-factor=1.0}}.<br />
<br />
=== WebGL ===<br />
<br />
There is the possibility that your graphics card has been blacklisted by Chromium. See [[#Force GPU acceleration]].<br />
<br />
If you are using Chromium with [[Bumblebee]], WebGL might crash due to GPU sandboxing. In this case, you can disable GPU sandboxing with {{ic|optirun chromium --disable-gpu-sandbox}}.<br />
<br />
Visit {{ic|chrome://gpu/}} for debugging information about WebGL support.<br />
<br />
Chromium can save incorrect data about your GPU in your user profile (e.g. if you use switch between an Nvidia card using Optimus and Intel, it will show the Nvidia card in {{ic|chrome://gpu}} even when you are not using it or primusrun/optirun). Running using a different user directory, e.g, {{ic|1=chromium --user-data-dir=$(mktemp -d)}} may solve this issue. For a persistent solution you can reset the GPU information by deleting {{ic|~/.config/chromium/Local\ State}}.<br />
<br />
=== Incorrect HiDPI rendering ===<br />
<br />
Chromium will automatically scale for a [[HiDPI]] display, however, this may cause an incorrect rendered GUI.<br />
<br />
The flag {{ic|1=--force-device-scale-factor=1}} may be used to overrule the automatic scaling factor.<br />
<br />
When [[#Native Wayland support|native Wayland support]] is enabled, Chromium will automatically scale based on the configured scale of each monitor.<br />
<br />
=== Password prompt on every start with GNOME Keyring ===<br />
<br />
See [[GNOME/Keyring#Passwords are not remembered]].<br />
<br />
=== Chromecasts in the network are not discovered ===<br />
<br />
You will need to enable the Media Router Component Extension in {{ic|chrome://flags/#load-media-router-component-extension}}.<br />
<br />
=== Everything is syncing except for password ===<br />
<br />
If synchronization is not working for password only (you can check it on {{ic|chrome://sync-internals/}}) delete profile login data:<br />
<br />
$ rm ~/.config/chromium/Default/Login\ Data*<br />
<br />
See [https://support.google.com/chrome/thread/9947763?hl=en&msgid=23687608 Google Chrome Help forum] for details.<br />
<br />
=== Losing cookies and passwords when switching between desktop environments ===<br />
<br />
If you see the message {{ic|Failed to decrypt token for service AccountId-*}} in the terminal when you start Chromium, it might try to use the wrong password storage backend. This might happen when you switch between Desktop Environments.<br />
<br />
See [[#Force a password store]].<br />
<br />
=== Hang on startup when Google Sync enabled ===<br />
<br />
Try launching Chrome with {{ic|1=--password-store=basic}} or another appropriate password store.<br />
<br />
See [[#Force a password store]].<br />
<br />
=== Chromium asks to be set as the default browser every time it starts ===<br />
<br />
If you are using KDE and have once set Firefox as the default browser (by clicking the button inside Firefox), you might find Chromium asks to be set as the default browser every time it starts, even if you click the "set as default" button.<br />
<br />
Chromium checks for this status by running {{ic|xdg-settings check default-web-browser chromium.desktop}}. If the output is "no", it is not considering itself to be the default browser. The script {{ic|xdg-settings}} checks for the following MIME associations and expect all of them to be {{ic|chromium.desktop}}:<br />
<br />
{{bc|<br />
x-scheme-handler/http<br />
x-scheme-handler/https<br />
text/html}}<br />
<br />
To fix it, go to ''System settings > Applications > Default applications > Web browser'' and choose Chromium. Then, set the MIME association for {{ic|text/html}}:<br />
<br />
$ xdg-mime default chromium.desktop text/html<br />
<br />
Finally, [[XDG_MIME_Applications#New_MIME_types|update the MIME database]]:<br />
<br />
$ update-mime-database ~/.local/share/mime<br />
<br />
=== "This browser or app may not be secure" error logging in to Google ===<br />
<br />
As of 2020.04.20 if you run chromium with {{ic|1=--remote-debugging-port=9222}} flag for web development, you cannot log in to your Google account. Temporarily disable this flag to login and then you can enable it back.<br />
<br />
=== Chromium stuck at 60fps when using a 144Hz + 60Hz monitor ===<br />
<br />
There is a suitable workaround for this issue, [[append]] the following flags to [[#Making flags persistent|persistent configuration]]:<br />
<br />
{{hc|1=~/.config/chromium-flags.conf|2=<br />
--use-gl=egl<br />
--ignore-gpu-blocklist<br />
--enable-gpu-rasterization<br />
}}<br />
<br />
This should make Chromium run at 144fps when used on your 144hz display, assuming your compositor is refreshing at 144fps. <br />
Keep in mind it might be a little choppy {{Bug|67035}}, but this is way better than it being stuck at 60fps.<br />
<br />
=== Chromium slow scroll speed ===<br />
<br />
Mouse whell scrolling in chromium and electron based applications may be too slow for daily usage. Here are some solutions.<br />
<br />
[[Libinput#Mouse wheel scrolling speed scaling]] injects {{ic|libinput_event_pointer_get_axis_value}} function in libinput and provides an interface to change scale factor. This is not a application level injection, so an addition script for application specific scale factor tuning is needed. Note that scroll on chromium's small height developer tools may be too fast when scale factor is big enough.<br />
<br />
[[IMWheel]] increases scroll distance by replaying X wheel button event for multiple times. However, chromium assumes the real scroll and the replayed ones as two events. There is a small but noticeable delay between them, so one mouse wheel scroll leads to twice page jumps. Also, touchpad scroll needs additional care.<br />
<br />
[https://chrome.google.com/webstore/detail/linux-scroll-speed-fix/mlboohjioameadaedfjcpemcaangkkbp Linux Scroll Speed Fix] and [https://chrome.google.com/webstore/detail/smoothscroll/nbokbjkabcmbfdlbddjidfmibcpneigj SmoothScroll] are two chromium extensions with suppport for scroll distance modification. Upon wheel scroll in a web page, the closest scrollable ancestor of current focused node will be found, then a scroll method with given pixel distance will be called on it, even if it has been scrolled to bottom. So once you scroll into a text editor or any scrollable element, you can never scroll out of it, except moving mouse. Also, extension based methods can not be used outside chromium.<br />
<br />
== See also ==<br />
<br />
* [https://www.chromium.org/ Chromium homepage]<br />
* [https://chromereleases.googleblog.com/ Google Chrome release notes]<br />
* [https://chrome.google.com/webstore/ Chrome web store]<br />
* [[Wikipedia:Chromium (web browser)#Differences from Google Chrome|Differences between Chromium and Google Chrome]]<br />
* [https://peter.sh/experiments/chromium-command-line-switches/ List of Chromium command-line switches]<br />
* [[Profile-sync-daemon]] - Systemd service that saves Chromium profile in tmpfs and syncs to disk<br />
* [[Tmpfs]] - Tmpfs Filesystem in {{ic|/etc/fstab}}<br />
* [https://www.kernel.org/doc/html/latest/filesystems/tmpfs.html Official tmpfs kernel Documentation]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PipeWire&diff=713418PipeWire2022-01-24T08:01:07Z<p>NeoTheFox: /* Sharing audio devices computers on the network */ minor spelling correction</p>
<hr />
<div>[[Category:Multimedia]]<br />
[[fi:PipeWire]]<br />
[[ja:PipeWire]]<br />
[[pt:PipeWire]]<br />
[[ru:PipeWire]]<br />
{{Related articles start}}<br />
{{Related|PipeWire/Examples}}<br />
{{Related articles end}}<br />
<br />
[https://pipewire.org PipeWire] is a new low-level multimedia framework. It aims to offer capture and playback for both audio and video with minimal latency and support for [[PulseAudio]], [[JACK]], [[ALSA]] and [[GStreamer]]-based applications.<br />
<br />
The daemon based on the framework can be configured to be both an audio server (with PulseAudio and JACK features) and a video capture server.<br />
<br />
PipeWire also supports containers like [[Flatpak]] and does not rely on the {{ic|audio}} and {{ic|video}} [[user group]]s. Instead, it uses a [[Polkit]]-like security model, asking Flatpak or Wayland for permission to record screen or audio.<br />
<br />
== Installation ==<br />
<br />
[[Install]] the {{Pkg|pipewire}} package from the official repositories. There is also {{Pkg|lib32-pipewire}} for [[multilib]] support.<br />
<br />
Pipewire uses [[systemd/User]] for management of the server and automatic socket activation.<br />
<br />
Optionally, install {{Pkg|pipewire-docs}} to review the documentation.<br />
<br />
Pipewire can work as drop-in replacement for others audio servers. See [[#Audio]] for details.<br />
<br />
{{Expansion|Describe what {{Pkg|gst-plugin-pipewire}} does exactly.}}<br />
<br />
=== GUI ===<br />
<br />
* {{App|Helvum|GTK-based patchbay for pipewire, inspired by the JACK tool catia.|https://gitlab.freedesktop.org/ryuukyu/helvum|{{Pkg|helvum}}}}<br />
<br />
=== WirePlumber ===<br />
<br />
{{Accuracy|The [https://wiki.gentoo.org/wiki/Pipewire#Starting_PipeWire_with_user_session gentoo wiki] says "Starting with 0.3.39, PipeWire also requires media-video/wireplumber to run properly". The Arch {{pkg|pipewire}} package has only an optional dependency on ''pipewire-session-manager''.|section=Unclear that we need pipewire-media-session or wireplumber}}<br />
<br />
PipeWire currently uses the simple example session manager {{ic|pipewire-media-session}}, but [https://pipewire.pages.freedesktop.org/wireplumber/index.html WirePlumber] is more powerful and [https://gitlab.freedesktop.org/pipewire/media-session#pipewire-media-session recommended]. To switch to WirePlumber, [[install]] the {{Pkg|wireplumber}} package. It will replace {{Pkg|pipewire-media-session}}.<br />
<br />
Similar to PipeWire's configuration files, the configuration files for WirePlumber can be found under {{ic|/usr/share/wireplumber}}. The best course of action to change the WirePlumber configuration is to copy the entire directory to {{ic|/etc}}. [https://pipewire.pages.freedesktop.org/wireplumber/installing-wireplumber.html]<br />
<br />
== Configuration ==<br />
<br />
The PipeWire package provides an initial set of configuration files in {{ic|/usr/share/pipewire}}. You should not edit these files directly, as package updates will overwrite your changes. To configure PipeWire, you can copy files from {{ic|/usr/share/pipewire}} to the alternate system-wide location {{ic|/etc/pipewire}}, or to the user location {{ic|~/.config/pipewire}}. [https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Config-PipeWire#configuration-file-pipewireconf]<br />
<br />
=== Profiles ===<br />
<br />
Pipewire brings a custom "Pro Audio" profile in addition to the PulseAudio profiles, selectable through pavucontrol. The effect of which is described here: https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/FAQ#what-is-the-pro-audio-profile<br />
<br />
== Usage ==<br />
<br />
=== Audio ===<br />
<br />
{{Expansion|What happens when none of the following is configured/installed and an application sends audio directly to pipewire? The {{Pkg|pipewire}} package contains many audio-related files, e.g. {{ic|/etc/alsa/conf.d/50-pipewire.conf}}.}}<br />
<br />
PipeWire can be used as an audio server, similar to PulseAudio and JACK. It aims to replace both PulseAudio and JACK, by providing a PulseAudio-compatible server implementation and ABI-compatible libraries for JACK clients. See the blog post [https://blogs.gnome.org/uraeus/2020/09/04/pipewire-late-summer-update-2020/ PipeWire Late Summer Update 2020] for more information.<br />
<br />
==== ALSA clients ====<br />
<br />
Install {{Pkg|pipewire-alsa}} (and remove {{Pkg|pulseaudio-alsa}} if it was installed) to route all application using the ALSA API through PipeWire.<br />
<br />
==== PulseAudio clients ====<br />
<br />
Install {{Pkg|pipewire-pulse}}. It will replace {{Pkg|pulseaudio}} and {{Pkg|pulseaudio-bluetooth}}. Reboot, re-login or execute {{ic|systemctl start --user pipewire-pulse.service}} to see the effect. <br />
<br />
Normally, no further action is needed, as the user service {{ic|pipewire-pulse.socket}} should be enabled automatically by the package. To check if the replacement is working, run the following command and see the output:<br />
<br />
{{hc|1=$ pactl info|2=<br />
...<br />
Server Name: PulseAudio (on PipeWire 0.3.32)<br />
...<br />
}}<br />
<br />
==== JACK clients ====<br />
<br />
Install {{Pkg|pipewire-jack}} for [[JACK]] support. There is also {{Pkg|lib32-pipewire-jack}} for [[multilib]] support.<br />
<br />
{{man|1|pw-jack}} may be used to start JACK clients, but it is technically not required, as it only serves as a wrapper around the {{ic|PIPEWIRE_REMOTE}}, {{ic|PIPEWIRE_DEBUG}} and {{ic|PIPEWIRE_LATENCY}} environment variables.<br />
<br />
It is possible to request a custom buffer size by setting a quotient of buffersize/samplerate (which equals the block latency in seconds):<br />
<br />
PIPEWIRE_LATENCY="128/48000" ''application''<br />
<br />
==== Bluetooth devices ====<br />
<br />
PipeWire handles Bluetooth audio devices if the {{Pkg|pipewire-pulse}} package is installed. The package includes the {{ic|/etc/pipewire/media-session.d/with-pulseaudio}} file, whose existence prompts the media session daemon to enable the {{ic|bluez5}} module.<br />
<br />
===== Automatic profile selection =====<br />
<br />
To automatically switch between HSP/HFP and A2DP profiles when an input stream is detected, set the {{ic|bluez5.autoswitch-profile}} property to {{ic|true}}:<br />
<br />
{{hc|/etc/pipewire/media-session.d/bluez-monitor.conf (or ~/.config/pipewire/media-session.d/bluez-monitor.conf|output=<br />
...<br />
rules = [<br />
{<br />
...<br />
actions = {<br />
update-props = {<br />
...<br />
bluez5.autoswitch-profile = true<br />
...<br />
}}<br />
<br />
==== PipeWire native patch sets ====<br />
<br />
We have Helvum for graphical visualization and creation of connections, but the rest is not in yet. The following are bash scripts which save wiresets, load wiresets, and dewire all connections. For saving and loading, use a command-line parameter for the filename.<br />
<br />
{{hc|pw-savewires|<nowiki><br />
#!/bin/bash<br />
<br />
if [[ "$#" -ne 1 ]]; then<br />
echo<br />
echo 'usage: pw-savewires filename'<br />
echo<br />
exit 0<br />
fi<br />
<br />
rm $1 &> /dev/null<br />
while IFS= read -r line; do<br />
link_on=`echo $line | cut -f 4 -d '"'`<br />
link_op=`echo $line | cut -f 6 -d '"'`<br />
link_in=`echo $line | cut -f 8 -d '"'`<br />
link_ip=`echo $line | cut -f 10 -d '"'`<br />
echo "Saving: " "'"$link_on:$link_op"','"$link_in:$link_ip"'"<br />
echo "'"$link_on:$link_op"','"$link_in:$link_ip"'" >> $1<br />
done < <(pw-cli dump short link)<br />
</nowiki>}}<br />
<br />
{{hc|pw-loadwires|<nowiki><br />
#!/bin/python<br />
<br />
import sys<br />
import csv<br />
import os<br />
<br />
if len(sys.argv) < 2:<br />
print('\n usage: pw-loadwires filename\n')<br />
quit()<br />
<br />
with open(sys.argv[1], newline='') as csvfile:<br />
pwwreader = csv.reader(csvfile, delimiter=',', quotechar='"')<br />
for row in pwwreader:<br />
print('Loading: ' + row[0] + ' --> ' + row[1])<br />
process = os.popen('pw-link ' + row[0] + ' ' + row[1])<br />
</nowiki>}}<br />
<br />
{{hc|pw-dewire|<nowiki><br />
#!/bin/bash<br />
while read -r line; do<br />
echo 'Dewiring: ' $line '...'<br />
pw-link -d $line<br />
done < <(pw-cli dump short link {{!}} grep -Eo '^[0-9]+')<br />
</nowiki>}}<br />
<br />
==== Sharing audio devices with computers on the network ====<br />
<br />
While PipeWire itself isn't network transparent, its pulse implementation supports [https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Config-PulseAudio#network-support network streaming]. An easy way to share audio between computers on the network is to use the [[Avahi]] daemon for discovery. <br />
Make sure that the {{ic|avahi-daemon.service}} is running on all computers that will be sharing audio.<br />
<br />
To share the local audio devices load the appropriate modules on the host:<br />
<br />
pactl load-module module-native-protocol-tcp listen=''192.168.1.10'' # Use the local IP address<br />
pactl pactl load-module module-zeroconf-publish<br />
<br />
Then load the discovery module on the clients:<br />
<br />
pactl load-module module-zeroconf-discover<br />
<br />
==== Run PipeWire on top of native JACK ====<br />
<br />
PipeWire can also run as a JACK client on top of the native JACK daemon if desired. See [https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/JACK JACK and PipeWire] for more information.<br />
<br />
=== WebRTC screen sharing ===<br />
<br />
Most applications used to rely on X11 for capturing the desktop (or individual applications), for example when using WebRTC in web browsers (e.g. on Google Hangouts). On Wayland, the sharing mechanism is handled differently for security reasons. PipeWire enables sharing content under Wayland with fine-grained access controls.<br />
<br />
This requires {{Pkg|xdg-desktop-portal}} and one of its backends [https://jgrulich.cz/2018/07/04/how-to-enable-and-use-screen-sharing-on-wayland to be installed]. The available backends are:<br />
<br />
* {{Pkg|xdg-desktop-portal-gnome}} for GNOME.<br />
* {{Pkg|xdg-desktop-portal-kde}} for KDE.<br />
* {{Pkg|xdg-desktop-portal-wlr}} for wlroots-based Wayland compositors (e.g. [[Sway]], [https://github.com/djpohly/dwl dwl])<br />
<br />
{{Note|1={{Pkg|xdg-desktop-portal}} 1.10.0 fixed a mismatch between specification and implementation of its D-Bus interface. [https://github.com/flatpak/xdg-desktop-portal/pull/609] Hence, some clients may not work with xdg-desktop-portal 1.10.0 or newer. Current progress for [https://github.com/obsproject/obs-studio/pull/5294 OBS Studio], [https://bugs.chromium.org/p/chromium/issues/detail?id=1250940 Chromium] and [https://bugzilla.mozilla.org/show_bug.cgi?id=1731495 Firefox].}}<br />
<br />
Firefox (84+) supports this method by default, while on Chromium (73+) one needs to enable [https://bugs.chromium.org/p/chromium/issues/detail?id=682122 WebRTC PipeWire support] by setting the corresponding (experimental) flag at the URL {{ic|chrome://flags/#enable-webrtc-pipewire-capturer}}.<br />
<br />
{{Pkg|obs-studio}} (27+) supports this method by using the new PipeWire capture source.<br />
<br />
{{Accuracy|Since [https://github.com/flatpak/xdg-desktop-portal-gtk/pull/225 this pull request] was merged, the following note about specific app/window sharing may be not correct anymore for {{Pkg|xdg-desktop-portal-gtk}}. Also see the ticket tracking the discussion at [https://github.com/flatpak/xdg-desktop-portal-gtk/issues/204].}}<br />
<br />
Note that the only supported feature is sharing the entire desktop and not a specific app/window [https://github.com/emersion/xdg-desktop-portal-wlr/wiki/FAQ#will-this-let-me-share-individual-windows][https://github.com/KDE/xdg-desktop-portal-kde/blob/master/TODO].<br />
<br />
==== xdg-desktop-portal-wlr ====<br />
<br />
For {{ic|xdg-desktop-portal-wlr}} to work, the {{ic|1=XDG_CURRENT_DESKTOP}} and {{ic|1=WAYLAND_DISPLAY}} environment variables have to be set in the [[Systemd/User#Environment_variables|systemd user session]]. {{ic|1=XDG_CURRENT_DESKTOP}} has to be set to the name of your compositor, e.g. {{ic|1=XDG_CURRENT_DESKTOP=sway}}. {{ic|1=WAYLAND_DISPLAY}} is set automatically by the compositor. The recommended way to bring these environment variables over to the systemd user session is to run {{ic|1=systemctl --user import-environment WAYLAND_DISPLAY XDG_CURRENT_DESKTOP}} after launching the compositor, e.g. with the compositors configuration file. See [https://github.com/emersion/xdg-desktop-portal-wlr#running] and [https://github.com/emersion/xdg-desktop-portal-wlr/wiki] for more details.<br />
<br />
{{Tip|To share an individual monitor with {{ic|xdg-desktop-portal-wlr}} if you have more than one, you can use configuration file options, see {{man|5|xdg-desktop-portal-wlr|SCREENCAST OPTIONS}}:<br />
{{hc|~/.config/xdg-desktop-portal-wlr/config|2=<br />
chooser_type = none<br />
output_name = ''Monitor''<br />
}}<br />
In [[Sway]], you can get the ''Monitor'' value using the {{ic|swaymsg -t get_outputs}} command.}}<br />
<br />
=== Video ===<br />
<br />
{{Expansion|{{Pkg|pipewire-v4l2}}}}<br />
<br />
Although the software is not yet production-ready, it is safe to play around with. Most applications that rely on [[GStreamer]] to handle e.g. video streams should work out-of-the-box using the PipeWire GStreamer plugin, see [[GStreamer#PipeWire]]. Applications like e.g. {{Pkg|cheese}} are therefore already able to share video input using it.<br />
<br />
== Audio post-processing ==<br />
<br />
=== EasyEffects ===<br />
<br />
EasyEffects (former PulseEffects) is a GTK utility which provides a large array of audio effects and filters to individual application output streams and microphone input streams. Notable effects include an input/output equalizer, output loudness equalization and bass enhancement, input de-esser and noise reduction plug-in. See [https://github.com/wwmm/easyeffects the GitHub page] for a full list of effects.<br />
<br />
In order to use EasyEffects, install {{Pkg|easyeffects}}. See [https://github.com/wwmm/easyeffects/wiki/Community-presets Community Presets] for a collection of preset configurations. See [https://github.com/jaakkopasanen/AutoEq AutoEq] for collection of AI generated EQ presets for headphones.<br />
<br />
{{Note|For PulseEffects legacy version, see [[PulseAudio#PulseEffects]].}}<br />
<br />
=== NoiseTorch ===<br />
<br />
NoiseTorch is an alternative way for noise suppression. {{AUR|noisetorch}}. There also exists a binary version, {{AUR|noisetorch-bin}}, as well as a {{AUR|noisetorch-git}}.<br />
<br />
After starting it the module can be loaded for the selected microphone. It is possible to adjust the voice activation threshold, which should be set to the highest level, not filtering out any actual voice.<br />
<br />
=== Noise suppression for voice ===<br />
<br />
Install {{AUR|noise-suppression-for-voice}} and see https://github.com/werman/noise-suppression-for-voice#pipewire. Then, set the noise cancelled source as default in your audio settings. You might need to restart your application prior being able to use it.<br />
<br />
=== JamesDSP ===<br />
<br />
[https://github.com/Audio4Linux/JDSP4Linux#readme JamesDSP for Linux] (available as {{AUR|jamesdsp}}) provides open-source sound effects for PipeWire and PulseAudio. It uses its own effects engine and without depending on LADSPA, Calf, etc. JamesDSP was initially published as an audio effects processor for Android devices.<br />
<br />
=== LADSPA, LV2 and VST plugins ===<br />
<br />
If you want to choose between the full list of available LADSPA, LV2 and VST plugins, you can apply them using a custom Pulseaudio null sink and Carla Jack host. Install {{Pkg|pipewire-pulse}}, {{Pkg|pipewire-jack}} and {{Pkg|carla}}. At the begin, create a new Pulseaudio null sink named {{ic|default_null_sink}}.<br />
<br />
pactl load-module module-null-sink object.linger=1 media.class=Audio/Sink sink_name=default_null_sink channel_map=FL,FR<br />
<br />
Start Carla through Pipewire, {{ic|pw-jack carla-rack}}. In ''Rack'' tab add whichever plugin you want. Make sure they are ''stereo'' type. You can change their order, the one on top of the list will be the first to receive the audio stream, just like in EasyEffects. Afterwards move to ''Patchbay'' tab and connect the {{ic|default_null_sink}} L/R monitors to Carla inputs, then Carla outputs to the playbacks of your desired device (speakers, earphones, HDMI, etc). Save the configuration to a local folder, i.e. {{ic|~/Documents/carla_sink_effects.carxp}}.<br />
<br />
You can test the effects while a multimedia application is reproducing audio, i.e. watching a video on a website through Firefox. There are two methods to do it. The first one, inside Carla ''Patchbay'' tab, disconnecting all Firefox connections and linking its L/R outputs to {{ic|default_null_sink}} playbacks. The second through {{Pkg|pavucontrol}}, locating Firefox audio stream and redirecting it to {{ic|default_null_sink}} (this should remember the connection to automatically redirect the application to the same sink on the next instance).<br />
<br />
To apply these settings at startup, create two systemd user service units:<br />
<br />
{{hc|~/.config/systemd/user/jack-carla-rack.service|output=<br />
[Unit]<br />
Description=Load Carla Rack JACK host<br />
<br />
[Service]<br />
PassEnvironment="PIPEWIRE_LINK_PASSIVE=true"<br />
Type=exec<br />
ExecStart=/usr/bin/pw-jack carla-rack -n<br />
<br />
[Install]<br />
WantedBy=default.target<br />
}}<br />
{{hc|~/.config/systemd/user/pulseaudio-null-sink@.service|output=<br />
[Unit]<br />
Description=Load %i Pulseaudio null sink<br />
Before=jack-carla-rack.service<br />
After=pipewire-pulse.service<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/bin/pactl load-module module-null-sink object.linger=1 media.class=Audio/Sink sink_name=%i channel_map=FL,FR<br />
ExecStop=/usr/bin/pactl unload-module module-null-sink<br />
RemainAfterExit=yes<br />
<br />
[Install]<br />
WantedBy=default.target<br />
}}<br />
<br />
Then override ''jack-carla-rack'' service specifying the full path of your Carla configuration at ''Environment'' directive:<br />
<br />
{{hc|~/.config/systemd/user/jack-carla-rack.service.d/override.conf|output=<br />
Environment="CARLA_CONFIG_FILE=/home/username/Documents/carla_sink_effects.carxp"<br />
ExecStart=<br />
ExecStart=/usr/bin/pw-jack carla-rack -n $CARLA_CONFIG_FILE<br />
}}<br />
<br />
At last, enable these two services specifying {{ic|default_null_sink}} as argument for ''pulseaudio-null-sink'' service:<br />
<br />
systemctl --user enable pulseaudio-null-sink@default_null_sink.service<br />
systemctl --user enable jack-carla-rack.service<br />
<br />
Note that if you set the {{ic|default_null_sink}} as the default device in system settings, all applications will be redirected to it and the volume keys will change its level, not the one on the speakers. If you want to control volume speakers, leave them as the default in system settings and redirect your desired application to {{ic|default_null_sink}} inside pavucontrol (Pipewire compatibility layer will remember the connection on the next instance of the same application).<br />
<br />
== Troubleshooting ==<br />
<br />
=== Audio ===<br />
<br />
==== Microphone is not detected by PipeWire ====<br />
<br />
PipeWire's {{ic|alsa-monitor}} module uses {{Pkg|alsa-card-profiles}} to detect devices by default. If this is not working for you, try to turn off {{ic|api.alsa.use-acp}}, or optionally turn on {{ic|api.alsa.use-ucm}}. <br />
<br />
If using {{Pkg|pipewire-media-session}}:<br />
{{hc|/etc/pipewire/media-session.d/alsa-monitor.conf (or ~/.config/pipewire/media-session.d/alsa-monitor.conf)|output=<br />
...<br />
rules = [<br />
{<br />
...<br />
actions = {<br />
update-props = {<br />
...<br />
api.alsa.use-acp = false<br />
...<br />
}}<br />
<br />
Otherwise, if using {{Pkg|wireplumber}}:<br />
{{hc|/etc/wireplumber/main.lua.d/50-alsa-config.lua (or ~/.config/wireplumber/main.lua.d/50-alsa-config.lua)|output=<br />
...<br />
alsa_monitor.rules = {<br />
{<br />
...<br />
apply_properties = {<br />
-- Use ALSA-Card-Profile devices. They use UCM or the profile<br />
-- configuration to configure the device and mixer settings.<br />
-- ["api.alsa.use-acp"] = true,<br />
<br />
-- Use UCM instead of profile when available. Can be<br />
-- disabled to skip trying to use the UCM profile.<br />
["api.alsa.use-ucm"] = true,<br />
...<br />
}}<br />
<br />
Then, restart pipewire and check available devices:<br />
<br />
{{hc| 1=<br />
$ pw-record --list-targets<br />
|2=<br />
Available targets ("*" denotes default): 62<br />
58: description="Built-in Audio" prio=1872<br />
60: description="Built-in Audio" prio=2000<br />
* 62: description="Built-in Audio (Loopback PCM)" prio=1984<br />
}}<br />
<br />
==== Sound does not automatically switch to Bluetooth headphones ====<br />
<br />
{{Accuracy|The linked upstream issue is specific to the xfce pulseaudio panel plugin.}}<br />
<br />
Run {{ic|pactl load-module module-switch-on-connect}} and configure your desktop environment to automatically run that command on login. See [https://gitlab.freedesktop.org/pipewire/wireplumber/-/issues/89 wireplumber issue #89] for more details.<br />
<br />
==== No sound after connecting to Bluetooth device ====<br />
<br />
As of 2020-12-07, if there is no sound after connecting a Bluetooth device, you might need to switch the default sink and/or move a sink input to the correct sink. Use {{ic|pactl list sinks}} to list the available sinks and {{ic|pactl set-default-sink}} to switch the default sink to the Bluetooth device. This can be automated via [[udev]] using a script similar to [https://gist.github.com/tinywrkb/04e7fd644afa9b92d33a3a99ab07ee9e this one].<br />
<br />
See this [https://www.reddit.com/r/archlinux/comments/jydd02/pipewirepulse_03164_in_testing_now_replaces/gd3m7fu/?context=3 Reddit thread] for a discussion of the issue. According to author of the script, the headset profile (HSP) might still have problems.<br />
<br />
==== Low volume ====<br />
<br />
After replacing PulseAudio with Pipewire, sound may work fine, but after a reboot, the volume becomes intolerably low.<br />
<br />
Open {{ic|alsamixer}}, use {{ic|F6}} to select the proper soundcard, and make sure the ALSA volumes are at 100%. {{ic|alsactl}} should maintain this setting after reboot.<br />
<br />
==== Increasing RLIMIT_MEMLOCK ====<br />
<br />
Dec 13 11:11:11 HOST pipewire-pulse[99999]: Failed to mlock memory 0x7f4f659d8000 32832: This is not a problem but for best performance, consider increasing RLIMIT_MEMLOCK<br />
<br />
Install {{Pkg|realtime-privileges}} and add your own user to the {{ic|realtime}} group.<br />
<br />
Alternatively, increasing memlock from 64kB to 128kB seems enough to fix this. If you are running {{ic|pipewire-pulse}} under [[systemd/User]], add:<br />
<br />
username soft memlock 64<br />
username hard memlock 128<br />
<br />
to {{ic|/etc/security/limits.d/username.conf}}<br />
<br />
==== Changing the sample rate ====<br />
<br />
By default PipeWire sets a global sample rate of 48kHz. If you need to change it (e.g. you own a DAC supporting a higher value), you can set a new default:<br />
<br />
{{hc|/etc/pipewire/pipewire.conf (or ~/.config/pipewire/pipewire.conf)|output=<br />
...<br />
context.properties = {<br />
...<br />
default.clock.rate = ''sample_rate''<br />
...<br />
}}<br />
<br />
PipeWire can also change output sample rates supported by your DAC. To configure, uncomment and set the line {{ic|1=default.clock.allowed-rates = [ 48000 ]}}, for example, {{ic|[ 44100 48000 88200 96000 ]}}. The sample rate follows the sample rate of the audio stream being played when the card is idle.<br />
<br />
To check out which output sample rate and sample format are the data sent to DAC (probably you need to change digits):<br />
cat /proc/asound/card0/pcm0p/sub0/hw_params<br />
To check out which input sample rate is used, change {{ic|pcm0p}} to {{ic|pcm0c}} ({{ic|c}} is short for "capture", {{ic|p}} is for "playback").<br />
<br />
==== Sound quality (resampling quality) ====<br />
<br />
If you used PulseAudio with {{ic|1=resample-method = speex-float-10}} or {{ic|soxr-vhq}}, then you might consider uncommenting and changing {{ic|1=resample.quality = 4}} to {{ic|10}} or the maximum {{ic|15}} in {{ic|stream.properties}} block in both {{ic|/etc/pipewire/client.conf}} and {{ic|/etc/pipewire/pipewire-pulse.conf}} (copy them from {{ic|/usr/share/pipewire/}} if they do not exist). Do not forget to restart PipeWire (without sudo): {{ic|systemctl --user restart pipewire.service pipewire-pulse.socket}} (never forget {{ic|pipewire-pulse.socket}} if you want your config changes to be applied).<br />
<br />
There is a very little quality difference between {{ic|10}} and {{ic|15}}, but the CPU load difference is 2-3x. And the latency difference between {{ic|4}}, {{ic|10}}, {{ic|15}} is yet to be investigated by anybody. {{ic|1= resample.quality = 15}} on 44100→48000 Hz on Ryzen 2600 causes {{ic|pipewire}} or {{ic|pipewire-pulse}} processes to cause 4.0% one CPU core load.<br />
<br />
You can compare resamplers here: https://src.infinitewave.ca/ (do not pay attention to anything above 18 KHz and over 120 dB). speex is listed as "Xiph.org Speex".<br />
<br />
PipeWire uses its own resampling algorithm called Spa. Like with SoX's {{ic|sox}}, Speex's {{ic|speexenc}}, PipeWire includes its standalone version: {{ic|spa-resample}}. Usage:<br />
spa-resample -q 15 -f s24 -r 48000 input16bit44100orAnythingElse.wav output24bit48000hz.wav<br />
<br />
It is probably somehow possible to use other resamplers by creating your own sink. Or just use a plugin in your music player (e.g., Qmmp has SoX plugin).<br />
<br />
==== External sound card not activated after reconnect ====<br />
<br />
Check {{ic|~/.config/pipewire-media-session/default-profile}} if there is any entry with default profile "off" and remove it. If that does not help, remove all files from {{ic|~/.config/pipewire-media-session/}} and restart PipeWire using {{ic|systemctl --user restart pipewire.service}}.<br />
<br />
==== No Sound or pactl info shows Failure: Connection refused ====<br />
<br />
It means applications are unable to connect to the PipeWire-Pulse service, confirm that {{ic|/etc/pipewire/pipewire-pulse.conf}} exists and is not empty and restart PipeWire-Pulse using {{ic|systemctl --user restart pipewire-pulse.service}}.<br />
<br />
If that does not fix it, run {{ic|strace -f -o /tmp/pipe.txt pactl info}} and pastebin {{ic|/tmp/pipe.txt}} while seeking help on IRC ([ircs://irc.oftc.net/pipewire #pipewire] on OFTC) or the mailing-lists.<br />
<br />
==== Low audio quality on Bluetooth ====<br />
<br />
In case Bluetooth playback stutters, check the [[unit status]] of the {{ic|pipewire.service}} user unit for errors similar as below:<br />
<br />
Feb 17 18:23:01 HOST pipewire[249297]: (bluez_input.18:54:CF:04:00:56.a2dp-sink-60) client too slow! rate:512/48000 pos:370688 status:triggered<br />
<br />
If they appear, check the currently selected codec using {{ic|pactl list sinks}} and try changing it by setting {{ic|bluez5.codecs}} to one of {{ic|sbc aac ldac aptx aptx_hd}}:<br />
<br />
{{hc|/etc/pipewire/media-session.d/bluez-monitor.conf (or ~/.config/pipewire/media-session.d/bluez-monitor.conf)|output=<br />
...<br />
properties = {<br />
...<br />
bluez5.codecs = [sbc]<br />
...<br />
}}<br />
<br />
Try enabling mSBC support (fixes mic on Sony 1000XM3, i.e. Headphones WH-1000XM3 and Earbuds WF-1000XM3):<br />
<br />
{{hc|/etc/pipewire/media-session.d/bluez-monitor.conf (or ~/.config/pipewire/media-session.d/bluez-monitor.conf)|output=<br />
...<br />
properties = {<br />
...<br />
bluez5.enable-msbc = true<br />
...<br />
}}<br />
<br />
Restart PipeWire by [[restart]]ing the {{ic|pipewire.service}} user unit for the changes to take effect.<br />
<br />
==== No devices detected after PipeWire update and reboot (git / >=0.3.23) ====<br />
<br />
As of commit [https://gitlab.freedesktop.org/pipewire/pipewire/-/commit/012a68f8ef33705f1a40ec8ac294b8cce7f6aa88 012a68f8], a new [[user unit]] has been added which is disabled by default, meaning there is no ''pipewire-media-session'' running on user login. Run this program on login by [[enabling]] the {{ic|pipewire-media-session.service}} user unit.<br />
<br />
If the user or the package manager have not sorted out the configuration file changes after update, then another instance of ''pipewire-media-session'' might be running in {{ic|pipewire.service}}. Verify whether this is the case by checking the [[unit status]] of the {{ic|pipewire.service}} user unit. If it shows ''pipewire'' and ''pipewire-media-session'' running, update your system and/or user configuration:<br />
<br />
{{hc|/etc/pipewire/pipewire.conf (or ~/.config/pipewire/pipewire.conf)|output=<br />
...<br />
context.exec = {<br />
...<br />
# Line below should be commented out<br />
#"/usr/bin/pipewire-media-session" = { args = "" }<br />
...<br />
}}<br />
<br />
==== Noticeable audio delay when starting playback ====<br />
<br />
This is caused by node suspension when inactive. It can be disabled by editing {{ic|/etc/pipewire/media-session.d/*-monitor.conf}} depending on where the delay occurs and changing property {{ic|session.suspend-timeout-seconds}} to 0 to disable or to experiment with other values and see what works. Alternatively you can comment out the line {{ic|suspend-node}} in {{ic|/etc/pipewire/media-session.d/media-session.conf}}. Restart both the {{ic|pipewire}} and {{ic|pipewire-pulse}} systemd services to apply these changes, or alternatively reboot.<br />
<br />
If you are using {{ic|wireplumber}} instead of {{ic|pipewire-media-session}}, then you can copy the example config file located at {{ic|/usr/share/wireplumber/main.lua.d/50-alsa-config.lua}} to {{ic|/etc/wireplumber/main.lua.d/50-alsa-config.lua}} and add the line {{ic|1=["session.suspend-timeout-seconds"] = 0}} to section {{ic|apply_properties}} at the end of the file.<br />
<br />
==== Audio cutting out when multiple streams start playing ====<br />
<br />
This problem can typically be diagnosed by running {{ic|journalctl --user -b -u pipewire-pulse}} and finding lines similar to:<br />
<br />
pipewire-pulse[21740]: pulse-server 0x56009b9d5de0: [Nightly] UNDERFLOW channel:0 offset:370676 underrun:940<br />
<br />
According to the [https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Troubleshooting#underrununderflow-and-broken-pipe-errors official PipeWire troubleshooting guide], to solve this problem edit {{ic|/etc/pipewire/media-session.d/alsa-monitor.conf}}, uncomment the line saying {{ic|1=api.alsa.headroom = 0}} and change its value to {{ic|1024}}.<br />
<br />
If you are using {{ic|wireplumber}} instead of {{ic|pipewire-media-session}}, then you can copy the example config file located at {{ic|/usr/share/wireplumber/main.lua.d/50-alsa-config.lua}} to {{ic|/etc/wireplumber/main.lua.d/50-alsa-config.lua}} and at the end of the file, under section {{ic|apply_properties}}, change {{ic|1=--["api.alsa.headroom"] = 0}} to {{ic|1=["api.alsa.headroom"] = 1024}}<br />
<br />
==== Audio is distorted ====<br />
<br />
* For microphones, try navigating to the card that is having issues after running {{ic|alsamixer}} and use the arrow keys to reduce any "Mic Boost" or "Internal Mic Boost" options.<br />
* Follow [[#Changing the sample rate]], reducing the sample rate to to {{ic|44100}} (44.1 kHz).<br />
<br />
==== Audio problems after standby ====<br />
<br />
If the sound is missing or otherwise garbled after waking the machine up from sleep, it might help to reinitialize ALSA:<br />
<br />
# alsactl init<br />
<br />
==== High latency with USB DACs (e.g. Schiit DACs) ====<br />
<br />
Changing sample rates or formats might help reduce latency with some DACs such as Schiit Hel 2.[https://www.reddit.com/r/osugame/comments/msifdd/usb_dacamp_and_audio_lag/]<br />
Using matching rules in ''pipewire-media-session'' we can set properties for devices.[https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Config-pipewire-media-session#matching-rules]<br />
<br />
Copy the default configuration of {{ic|alsa-monitor.conf}} for ''pipewire-media-session'' into either {{ic|/etc/pipewire/media-session.d}} or {{ic|~/.config/pipewire/media-session.d}}.<br />
Then append a new rule-block similar to the following one:<br />
<br />
{{hc|/etc/pipewire/media-session.d/alsa-monitor.conf (or ~/.config/pipewire/media-session.d/alsa-monitor.conf)|output=<br />
...<br />
rules = {<br />
...<br />
{<br />
matches = [<br />
{<br />
node.name = "alsa_output.<name of node>"<br />
}<br />
]<br />
actions = {<br />
update-props = {<br />
audio.format = "S24_3LE"<br />
audio.rate = 96000<br />
# Following value should be doubled until audio doesn't cut out or other issues stop occurring<br />
api.alsa.period-size = 128<br />
...<br />
}}<br />
<br />
{{ic|alsa_output.<name of node>}} node can be obtained using {{ic|pw-top}}.<br />
<br />
Your DAC might support a different format or sample rate. You can check what your DAC supports by querying [[ALSA]]:<br />
<br />
First get the card number of your DAC:<br />
<br />
{{hc|$ aplay -l|<br />
...<br />
card 3: S2 [Schiit Hel 2], device 0: USB Audio [USB Audio]<br />
Subdevices: 0/1<br />
Subdevice #0: subdevice #0<br />
...<br />
}}<br />
<br />
So in this example it would be card 3.<br />
Get all supported sample rates and formats:<br />
<br />
{{hc|$ cat /proc/asound/cardX/streamX|<br />
...<br />
Playback:<br />
...<br />
Interface 1<br />
Altset 1<br />
Format: S16_LE<br />
Channels: 2<br />
Endpoint: 0x05 (5 OUT) (ASYNC)<br />
Rates: 44100, 48000, 88200, 96000, 176400, 192000, 352800, 384000<br />
Data packet interval: 125 us<br />
Bits: 16<br />
...<br />
Interface 1<br />
Altset 2<br />
Format: S24_3LE<br />
Channels: 2<br />
Endpoint: 0x05 (5 OUT) (ASYNC)<br />
Rates: 44100, 48000, 88200, 96000, 176400, 192000, 352800, 384000<br />
Data packet interval: 125 us<br />
Bits: 24<br />
...<br />
Interface 1<br />
Altset 3<br />
Format: S32_LE<br />
Channels: 2<br />
Endpoint: 0x05 (5 OUT) (ASYNC)<br />
Rates: 44100, 48000, 88200, 96000, 176400, 192000, 352800, 384000<br />
Data packet interval: 125 us<br />
Bits: 32<br />
...<br />
...<br />
}}<br />
<br />
In this case {{ic|S16_LE, S24_3LE, S32_LE}} are the supported formats and {{ic|44100, 48000, 88200, 96000, 176400, 192000, 352800, 384000}} are the supported sample rates across all formats.<br />
<br />
=== Video ===<br />
<br />
==== OBS (etc.) display nothing, even if they ask for a window/screen ====<br />
<br />
If you are sure that you have {{Pkg|xdg-desktop-portal}} installed as well as either {{Pkg|xdg-desktop-portal-gtk}} or {{Pkg|xdg-desktop-portal-kde}}, check the running state of the daemons.<br />
<br />
In OBS, if everything is working, you should see this in {{ic|stdout}}:<br />
<br />
...<br />
info: [pipewire] desktop selected, setting up screencast<br />
info: [pipewire] created stream 0x5632d7456850<br />
info: [pipewire] playing stream…<br />
<br />
For multi-monitor setups the {{Pkg|slurp}} package will allow to capture of all the screens.<br />
<br />
== See also ==<br />
<br />
* [https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/home Wiki] — PipeWire Wiki on Freedesktop GitLab<br />
* [https://blogs.gnome.org/uraeus/2018/01/26/an-update-on-pipewire-the-multimedia-revolution-an-update/ Pipewire Update Blog Post] — Blog post from January 2018 outlining the state of PipeWire at the time<br />
* [https://blogs.gnome.org/uraeus/2020/09/04/pipewire-late-summer-update-2020/ PipeWire Late Summer Update 2020] — Blog post from September 2020</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PipeWire&diff=713417PipeWire2022-01-24T07:58:50Z<p>NeoTheFox: /* Audio */ Add network audio sharing</p>
<hr />
<div>[[Category:Multimedia]]<br />
[[fi:PipeWire]]<br />
[[ja:PipeWire]]<br />
[[pt:PipeWire]]<br />
[[ru:PipeWire]]<br />
{{Related articles start}}<br />
{{Related|PipeWire/Examples}}<br />
{{Related articles end}}<br />
<br />
[https://pipewire.org PipeWire] is a new low-level multimedia framework. It aims to offer capture and playback for both audio and video with minimal latency and support for [[PulseAudio]], [[JACK]], [[ALSA]] and [[GStreamer]]-based applications.<br />
<br />
The daemon based on the framework can be configured to be both an audio server (with PulseAudio and JACK features) and a video capture server.<br />
<br />
PipeWire also supports containers like [[Flatpak]] and does not rely on the {{ic|audio}} and {{ic|video}} [[user group]]s. Instead, it uses a [[Polkit]]-like security model, asking Flatpak or Wayland for permission to record screen or audio.<br />
<br />
== Installation ==<br />
<br />
[[Install]] the {{Pkg|pipewire}} package from the official repositories. There is also {{Pkg|lib32-pipewire}} for [[multilib]] support.<br />
<br />
Pipewire uses [[systemd/User]] for management of the server and automatic socket activation.<br />
<br />
Optionally, install {{Pkg|pipewire-docs}} to review the documentation.<br />
<br />
Pipewire can work as drop-in replacement for others audio servers. See [[#Audio]] for details.<br />
<br />
{{Expansion|Describe what {{Pkg|gst-plugin-pipewire}} does exactly.}}<br />
<br />
=== GUI ===<br />
<br />
* {{App|Helvum|GTK-based patchbay for pipewire, inspired by the JACK tool catia.|https://gitlab.freedesktop.org/ryuukyu/helvum|{{Pkg|helvum}}}}<br />
<br />
=== WirePlumber ===<br />
<br />
{{Accuracy|The [https://wiki.gentoo.org/wiki/Pipewire#Starting_PipeWire_with_user_session gentoo wiki] says "Starting with 0.3.39, PipeWire also requires media-video/wireplumber to run properly". The Arch {{pkg|pipewire}} package has only an optional dependency on ''pipewire-session-manager''.|section=Unclear that we need pipewire-media-session or wireplumber}}<br />
<br />
PipeWire currently uses the simple example session manager {{ic|pipewire-media-session}}, but [https://pipewire.pages.freedesktop.org/wireplumber/index.html WirePlumber] is more powerful and [https://gitlab.freedesktop.org/pipewire/media-session#pipewire-media-session recommended]. To switch to WirePlumber, [[install]] the {{Pkg|wireplumber}} package. It will replace {{Pkg|pipewire-media-session}}.<br />
<br />
Similar to PipeWire's configuration files, the configuration files for WirePlumber can be found under {{ic|/usr/share/wireplumber}}. The best course of action to change the WirePlumber configuration is to copy the entire directory to {{ic|/etc}}. [https://pipewire.pages.freedesktop.org/wireplumber/installing-wireplumber.html]<br />
<br />
== Configuration ==<br />
<br />
The PipeWire package provides an initial set of configuration files in {{ic|/usr/share/pipewire}}. You should not edit these files directly, as package updates will overwrite your changes. To configure PipeWire, you can copy files from {{ic|/usr/share/pipewire}} to the alternate system-wide location {{ic|/etc/pipewire}}, or to the user location {{ic|~/.config/pipewire}}. [https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Config-PipeWire#configuration-file-pipewireconf]<br />
<br />
=== Profiles ===<br />
<br />
Pipewire brings a custom "Pro Audio" profile in addition to the PulseAudio profiles, selectable through pavucontrol. The effect of which is described here: https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/FAQ#what-is-the-pro-audio-profile<br />
<br />
== Usage ==<br />
<br />
=== Audio ===<br />
<br />
{{Expansion|What happens when none of the following is configured/installed and an application sends audio directly to pipewire? The {{Pkg|pipewire}} package contains many audio-related files, e.g. {{ic|/etc/alsa/conf.d/50-pipewire.conf}}.}}<br />
<br />
PipeWire can be used as an audio server, similar to PulseAudio and JACK. It aims to replace both PulseAudio and JACK, by providing a PulseAudio-compatible server implementation and ABI-compatible libraries for JACK clients. See the blog post [https://blogs.gnome.org/uraeus/2020/09/04/pipewire-late-summer-update-2020/ PipeWire Late Summer Update 2020] for more information.<br />
<br />
==== ALSA clients ====<br />
<br />
Install {{Pkg|pipewire-alsa}} (and remove {{Pkg|pulseaudio-alsa}} if it was installed) to route all application using the ALSA API through PipeWire.<br />
<br />
==== PulseAudio clients ====<br />
<br />
Install {{Pkg|pipewire-pulse}}. It will replace {{Pkg|pulseaudio}} and {{Pkg|pulseaudio-bluetooth}}. Reboot, re-login or execute {{ic|systemctl start --user pipewire-pulse.service}} to see the effect. <br />
<br />
Normally, no further action is needed, as the user service {{ic|pipewire-pulse.socket}} should be enabled automatically by the package. To check if the replacement is working, run the following command and see the output:<br />
<br />
{{hc|1=$ pactl info|2=<br />
...<br />
Server Name: PulseAudio (on PipeWire 0.3.32)<br />
...<br />
}}<br />
<br />
==== JACK clients ====<br />
<br />
Install {{Pkg|pipewire-jack}} for [[JACK]] support. There is also {{Pkg|lib32-pipewire-jack}} for [[multilib]] support.<br />
<br />
{{man|1|pw-jack}} may be used to start JACK clients, but it is technically not required, as it only serves as a wrapper around the {{ic|PIPEWIRE_REMOTE}}, {{ic|PIPEWIRE_DEBUG}} and {{ic|PIPEWIRE_LATENCY}} environment variables.<br />
<br />
It is possible to request a custom buffer size by setting a quotient of buffersize/samplerate (which equals the block latency in seconds):<br />
<br />
PIPEWIRE_LATENCY="128/48000" ''application''<br />
<br />
==== Bluetooth devices ====<br />
<br />
PipeWire handles Bluetooth audio devices if the {{Pkg|pipewire-pulse}} package is installed. The package includes the {{ic|/etc/pipewire/media-session.d/with-pulseaudio}} file, whose existence prompts the media session daemon to enable the {{ic|bluez5}} module.<br />
<br />
===== Automatic profile selection =====<br />
<br />
To automatically switch between HSP/HFP and A2DP profiles when an input stream is detected, set the {{ic|bluez5.autoswitch-profile}} property to {{ic|true}}:<br />
<br />
{{hc|/etc/pipewire/media-session.d/bluez-monitor.conf (or ~/.config/pipewire/media-session.d/bluez-monitor.conf|output=<br />
...<br />
rules = [<br />
{<br />
...<br />
actions = {<br />
update-props = {<br />
...<br />
bluez5.autoswitch-profile = true<br />
...<br />
}}<br />
<br />
==== PipeWire native patch sets ====<br />
<br />
We have Helvum for graphical visualization and creation of connections, but the rest is not in yet. The following are bash scripts which save wiresets, load wiresets, and dewire all connections. For saving and loading, use a command-line parameter for the filename.<br />
<br />
{{hc|pw-savewires|<nowiki><br />
#!/bin/bash<br />
<br />
if [[ "$#" -ne 1 ]]; then<br />
echo<br />
echo 'usage: pw-savewires filename'<br />
echo<br />
exit 0<br />
fi<br />
<br />
rm $1 &> /dev/null<br />
while IFS= read -r line; do<br />
link_on=`echo $line | cut -f 4 -d '"'`<br />
link_op=`echo $line | cut -f 6 -d '"'`<br />
link_in=`echo $line | cut -f 8 -d '"'`<br />
link_ip=`echo $line | cut -f 10 -d '"'`<br />
echo "Saving: " "'"$link_on:$link_op"','"$link_in:$link_ip"'"<br />
echo "'"$link_on:$link_op"','"$link_in:$link_ip"'" >> $1<br />
done < <(pw-cli dump short link)<br />
</nowiki>}}<br />
<br />
{{hc|pw-loadwires|<nowiki><br />
#!/bin/python<br />
<br />
import sys<br />
import csv<br />
import os<br />
<br />
if len(sys.argv) < 2:<br />
print('\n usage: pw-loadwires filename\n')<br />
quit()<br />
<br />
with open(sys.argv[1], newline='') as csvfile:<br />
pwwreader = csv.reader(csvfile, delimiter=',', quotechar='"')<br />
for row in pwwreader:<br />
print('Loading: ' + row[0] + ' --> ' + row[1])<br />
process = os.popen('pw-link ' + row[0] + ' ' + row[1])<br />
</nowiki>}}<br />
<br />
{{hc|pw-dewire|<nowiki><br />
#!/bin/bash<br />
while read -r line; do<br />
echo 'Dewiring: ' $line '...'<br />
pw-link -d $line<br />
done < <(pw-cli dump short link {{!}} grep -Eo '^[0-9]+')<br />
</nowiki>}}<br />
<br />
==== Sharing audio devices computers on the network ====<br />
<br />
While PipeWire itself isn't network transparent, its pulse implementation supports [https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Config-PulseAudio#network-support network streaming]. An easy way to share audio between computers on the network is to use the [[Avahi]] daemon for discovery. <br />
Make sure that the {{ic|avahi-daemon.service}} is running on all computers that would be sharing audio.<br />
<br />
To share the local audio devices load the appropriate modules on the host:<br />
<br />
pactl load-module module-native-protocol-tcp listen=''192.168.1.10'' # Use the local IP address<br />
pactl pactl load-module module-zeroconf-publish<br />
<br />
Then load the discovery module on the clients:<br />
<br />
pactl load-module module-zeroconf-discover<br />
<br />
==== Run PipeWire on top of native JACK ====<br />
<br />
PipeWire can also run as a JACK client on top of the native JACK daemon if desired. See [https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/JACK JACK and PipeWire] for more information.<br />
<br />
=== WebRTC screen sharing ===<br />
<br />
Most applications used to rely on X11 for capturing the desktop (or individual applications), for example when using WebRTC in web browsers (e.g. on Google Hangouts). On Wayland, the sharing mechanism is handled differently for security reasons. PipeWire enables sharing content under Wayland with fine-grained access controls.<br />
<br />
This requires {{Pkg|xdg-desktop-portal}} and one of its backends [https://jgrulich.cz/2018/07/04/how-to-enable-and-use-screen-sharing-on-wayland to be installed]. The available backends are:<br />
<br />
* {{Pkg|xdg-desktop-portal-gnome}} for GNOME.<br />
* {{Pkg|xdg-desktop-portal-kde}} for KDE.<br />
* {{Pkg|xdg-desktop-portal-wlr}} for wlroots-based Wayland compositors (e.g. [[Sway]], [https://github.com/djpohly/dwl dwl])<br />
<br />
{{Note|1={{Pkg|xdg-desktop-portal}} 1.10.0 fixed a mismatch between specification and implementation of its D-Bus interface. [https://github.com/flatpak/xdg-desktop-portal/pull/609] Hence, some clients may not work with xdg-desktop-portal 1.10.0 or newer. Current progress for [https://github.com/obsproject/obs-studio/pull/5294 OBS Studio], [https://bugs.chromium.org/p/chromium/issues/detail?id=1250940 Chromium] and [https://bugzilla.mozilla.org/show_bug.cgi?id=1731495 Firefox].}}<br />
<br />
Firefox (84+) supports this method by default, while on Chromium (73+) one needs to enable [https://bugs.chromium.org/p/chromium/issues/detail?id=682122 WebRTC PipeWire support] by setting the corresponding (experimental) flag at the URL {{ic|chrome://flags/#enable-webrtc-pipewire-capturer}}.<br />
<br />
{{Pkg|obs-studio}} (27+) supports this method by using the new PipeWire capture source.<br />
<br />
{{Accuracy|Since [https://github.com/flatpak/xdg-desktop-portal-gtk/pull/225 this pull request] was merged, the following note about specific app/window sharing may be not correct anymore for {{Pkg|xdg-desktop-portal-gtk}}. Also see the ticket tracking the discussion at [https://github.com/flatpak/xdg-desktop-portal-gtk/issues/204].}}<br />
<br />
Note that the only supported feature is sharing the entire desktop and not a specific app/window [https://github.com/emersion/xdg-desktop-portal-wlr/wiki/FAQ#will-this-let-me-share-individual-windows][https://github.com/KDE/xdg-desktop-portal-kde/blob/master/TODO].<br />
<br />
==== xdg-desktop-portal-wlr ====<br />
<br />
For {{ic|xdg-desktop-portal-wlr}} to work, the {{ic|1=XDG_CURRENT_DESKTOP}} and {{ic|1=WAYLAND_DISPLAY}} environment variables have to be set in the [[Systemd/User#Environment_variables|systemd user session]]. {{ic|1=XDG_CURRENT_DESKTOP}} has to be set to the name of your compositor, e.g. {{ic|1=XDG_CURRENT_DESKTOP=sway}}. {{ic|1=WAYLAND_DISPLAY}} is set automatically by the compositor. The recommended way to bring these environment variables over to the systemd user session is to run {{ic|1=systemctl --user import-environment WAYLAND_DISPLAY XDG_CURRENT_DESKTOP}} after launching the compositor, e.g. with the compositors configuration file. See [https://github.com/emersion/xdg-desktop-portal-wlr#running] and [https://github.com/emersion/xdg-desktop-portal-wlr/wiki] for more details.<br />
<br />
{{Tip|To share an individual monitor with {{ic|xdg-desktop-portal-wlr}} if you have more than one, you can use configuration file options, see {{man|5|xdg-desktop-portal-wlr|SCREENCAST OPTIONS}}:<br />
{{hc|~/.config/xdg-desktop-portal-wlr/config|2=<br />
chooser_type = none<br />
output_name = ''Monitor''<br />
}}<br />
In [[Sway]], you can get the ''Monitor'' value using the {{ic|swaymsg -t get_outputs}} command.}}<br />
<br />
=== Video ===<br />
<br />
{{Expansion|{{Pkg|pipewire-v4l2}}}}<br />
<br />
Although the software is not yet production-ready, it is safe to play around with. Most applications that rely on [[GStreamer]] to handle e.g. video streams should work out-of-the-box using the PipeWire GStreamer plugin, see [[GStreamer#PipeWire]]. Applications like e.g. {{Pkg|cheese}} are therefore already able to share video input using it.<br />
<br />
== Audio post-processing ==<br />
<br />
=== EasyEffects ===<br />
<br />
EasyEffects (former PulseEffects) is a GTK utility which provides a large array of audio effects and filters to individual application output streams and microphone input streams. Notable effects include an input/output equalizer, output loudness equalization and bass enhancement, input de-esser and noise reduction plug-in. See [https://github.com/wwmm/easyeffects the GitHub page] for a full list of effects.<br />
<br />
In order to use EasyEffects, install {{Pkg|easyeffects}}. See [https://github.com/wwmm/easyeffects/wiki/Community-presets Community Presets] for a collection of preset configurations. See [https://github.com/jaakkopasanen/AutoEq AutoEq] for collection of AI generated EQ presets for headphones.<br />
<br />
{{Note|For PulseEffects legacy version, see [[PulseAudio#PulseEffects]].}}<br />
<br />
=== NoiseTorch ===<br />
<br />
NoiseTorch is an alternative way for noise suppression. {{AUR|noisetorch}}. There also exists a binary version, {{AUR|noisetorch-bin}}, as well as a {{AUR|noisetorch-git}}.<br />
<br />
After starting it the module can be loaded for the selected microphone. It is possible to adjust the voice activation threshold, which should be set to the highest level, not filtering out any actual voice.<br />
<br />
=== Noise suppression for voice ===<br />
<br />
Install {{AUR|noise-suppression-for-voice}} and see https://github.com/werman/noise-suppression-for-voice#pipewire. Then, set the noise cancelled source as default in your audio settings. You might need to restart your application prior being able to use it.<br />
<br />
=== JamesDSP ===<br />
<br />
[https://github.com/Audio4Linux/JDSP4Linux#readme JamesDSP for Linux] (available as {{AUR|jamesdsp}}) provides open-source sound effects for PipeWire and PulseAudio. It uses its own effects engine and without depending on LADSPA, Calf, etc. JamesDSP was initially published as an audio effects processor for Android devices.<br />
<br />
=== LADSPA, LV2 and VST plugins ===<br />
<br />
If you want to choose between the full list of available LADSPA, LV2 and VST plugins, you can apply them using a custom Pulseaudio null sink and Carla Jack host. Install {{Pkg|pipewire-pulse}}, {{Pkg|pipewire-jack}} and {{Pkg|carla}}. At the begin, create a new Pulseaudio null sink named {{ic|default_null_sink}}.<br />
<br />
pactl load-module module-null-sink object.linger=1 media.class=Audio/Sink sink_name=default_null_sink channel_map=FL,FR<br />
<br />
Start Carla through Pipewire, {{ic|pw-jack carla-rack}}. In ''Rack'' tab add whichever plugin you want. Make sure they are ''stereo'' type. You can change their order, the one on top of the list will be the first to receive the audio stream, just like in EasyEffects. Afterwards move to ''Patchbay'' tab and connect the {{ic|default_null_sink}} L/R monitors to Carla inputs, then Carla outputs to the playbacks of your desired device (speakers, earphones, HDMI, etc). Save the configuration to a local folder, i.e. {{ic|~/Documents/carla_sink_effects.carxp}}.<br />
<br />
You can test the effects while a multimedia application is reproducing audio, i.e. watching a video on a website through Firefox. There are two methods to do it. The first one, inside Carla ''Patchbay'' tab, disconnecting all Firefox connections and linking its L/R outputs to {{ic|default_null_sink}} playbacks. The second through {{Pkg|pavucontrol}}, locating Firefox audio stream and redirecting it to {{ic|default_null_sink}} (this should remember the connection to automatically redirect the application to the same sink on the next instance).<br />
<br />
To apply these settings at startup, create two systemd user service units:<br />
<br />
{{hc|~/.config/systemd/user/jack-carla-rack.service|output=<br />
[Unit]<br />
Description=Load Carla Rack JACK host<br />
<br />
[Service]<br />
PassEnvironment="PIPEWIRE_LINK_PASSIVE=true"<br />
Type=exec<br />
ExecStart=/usr/bin/pw-jack carla-rack -n<br />
<br />
[Install]<br />
WantedBy=default.target<br />
}}<br />
{{hc|~/.config/systemd/user/pulseaudio-null-sink@.service|output=<br />
[Unit]<br />
Description=Load %i Pulseaudio null sink<br />
Before=jack-carla-rack.service<br />
After=pipewire-pulse.service<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/bin/pactl load-module module-null-sink object.linger=1 media.class=Audio/Sink sink_name=%i channel_map=FL,FR<br />
ExecStop=/usr/bin/pactl unload-module module-null-sink<br />
RemainAfterExit=yes<br />
<br />
[Install]<br />
WantedBy=default.target<br />
}}<br />
<br />
Then override ''jack-carla-rack'' service specifying the full path of your Carla configuration at ''Environment'' directive:<br />
<br />
{{hc|~/.config/systemd/user/jack-carla-rack.service.d/override.conf|output=<br />
Environment="CARLA_CONFIG_FILE=/home/username/Documents/carla_sink_effects.carxp"<br />
ExecStart=<br />
ExecStart=/usr/bin/pw-jack carla-rack -n $CARLA_CONFIG_FILE<br />
}}<br />
<br />
At last, enable these two services specifying {{ic|default_null_sink}} as argument for ''pulseaudio-null-sink'' service:<br />
<br />
systemctl --user enable pulseaudio-null-sink@default_null_sink.service<br />
systemctl --user enable jack-carla-rack.service<br />
<br />
Note that if you set the {{ic|default_null_sink}} as the default device in system settings, all applications will be redirected to it and the volume keys will change its level, not the one on the speakers. If you want to control volume speakers, leave them as the default in system settings and redirect your desired application to {{ic|default_null_sink}} inside pavucontrol (Pipewire compatibility layer will remember the connection on the next instance of the same application).<br />
<br />
== Troubleshooting ==<br />
<br />
=== Audio ===<br />
<br />
==== Microphone is not detected by PipeWire ====<br />
<br />
PipeWire's {{ic|alsa-monitor}} module uses {{Pkg|alsa-card-profiles}} to detect devices by default. If this is not working for you, try to turn off {{ic|api.alsa.use-acp}}, or optionally turn on {{ic|api.alsa.use-ucm}}. <br />
<br />
If using {{Pkg|pipewire-media-session}}:<br />
{{hc|/etc/pipewire/media-session.d/alsa-monitor.conf (or ~/.config/pipewire/media-session.d/alsa-monitor.conf)|output=<br />
...<br />
rules = [<br />
{<br />
...<br />
actions = {<br />
update-props = {<br />
...<br />
api.alsa.use-acp = false<br />
...<br />
}}<br />
<br />
Otherwise, if using {{Pkg|wireplumber}}:<br />
{{hc|/etc/wireplumber/main.lua.d/50-alsa-config.lua (or ~/.config/wireplumber/main.lua.d/50-alsa-config.lua)|output=<br />
...<br />
alsa_monitor.rules = {<br />
{<br />
...<br />
apply_properties = {<br />
-- Use ALSA-Card-Profile devices. They use UCM or the profile<br />
-- configuration to configure the device and mixer settings.<br />
-- ["api.alsa.use-acp"] = true,<br />
<br />
-- Use UCM instead of profile when available. Can be<br />
-- disabled to skip trying to use the UCM profile.<br />
["api.alsa.use-ucm"] = true,<br />
...<br />
}}<br />
<br />
Then, restart pipewire and check available devices:<br />
<br />
{{hc| 1=<br />
$ pw-record --list-targets<br />
|2=<br />
Available targets ("*" denotes default): 62<br />
58: description="Built-in Audio" prio=1872<br />
60: description="Built-in Audio" prio=2000<br />
* 62: description="Built-in Audio (Loopback PCM)" prio=1984<br />
}}<br />
<br />
==== Sound does not automatically switch to Bluetooth headphones ====<br />
<br />
{{Accuracy|The linked upstream issue is specific to the xfce pulseaudio panel plugin.}}<br />
<br />
Run {{ic|pactl load-module module-switch-on-connect}} and configure your desktop environment to automatically run that command on login. See [https://gitlab.freedesktop.org/pipewire/wireplumber/-/issues/89 wireplumber issue #89] for more details.<br />
<br />
==== No sound after connecting to Bluetooth device ====<br />
<br />
As of 2020-12-07, if there is no sound after connecting a Bluetooth device, you might need to switch the default sink and/or move a sink input to the correct sink. Use {{ic|pactl list sinks}} to list the available sinks and {{ic|pactl set-default-sink}} to switch the default sink to the Bluetooth device. This can be automated via [[udev]] using a script similar to [https://gist.github.com/tinywrkb/04e7fd644afa9b92d33a3a99ab07ee9e this one].<br />
<br />
See this [https://www.reddit.com/r/archlinux/comments/jydd02/pipewirepulse_03164_in_testing_now_replaces/gd3m7fu/?context=3 Reddit thread] for a discussion of the issue. According to author of the script, the headset profile (HSP) might still have problems.<br />
<br />
==== Low volume ====<br />
<br />
After replacing PulseAudio with Pipewire, sound may work fine, but after a reboot, the volume becomes intolerably low.<br />
<br />
Open {{ic|alsamixer}}, use {{ic|F6}} to select the proper soundcard, and make sure the ALSA volumes are at 100%. {{ic|alsactl}} should maintain this setting after reboot.<br />
<br />
==== Increasing RLIMIT_MEMLOCK ====<br />
<br />
Dec 13 11:11:11 HOST pipewire-pulse[99999]: Failed to mlock memory 0x7f4f659d8000 32832: This is not a problem but for best performance, consider increasing RLIMIT_MEMLOCK<br />
<br />
Install {{Pkg|realtime-privileges}} and add your own user to the {{ic|realtime}} group.<br />
<br />
Alternatively, increasing memlock from 64kB to 128kB seems enough to fix this. If you are running {{ic|pipewire-pulse}} under [[systemd/User]], add:<br />
<br />
username soft memlock 64<br />
username hard memlock 128<br />
<br />
to {{ic|/etc/security/limits.d/username.conf}}<br />
<br />
==== Changing the sample rate ====<br />
<br />
By default PipeWire sets a global sample rate of 48kHz. If you need to change it (e.g. you own a DAC supporting a higher value), you can set a new default:<br />
<br />
{{hc|/etc/pipewire/pipewire.conf (or ~/.config/pipewire/pipewire.conf)|output=<br />
...<br />
context.properties = {<br />
...<br />
default.clock.rate = ''sample_rate''<br />
...<br />
}}<br />
<br />
PipeWire can also change output sample rates supported by your DAC. To configure, uncomment and set the line {{ic|1=default.clock.allowed-rates = [ 48000 ]}}, for example, {{ic|[ 44100 48000 88200 96000 ]}}. The sample rate follows the sample rate of the audio stream being played when the card is idle.<br />
<br />
To check out which output sample rate and sample format are the data sent to DAC (probably you need to change digits):<br />
cat /proc/asound/card0/pcm0p/sub0/hw_params<br />
To check out which input sample rate is used, change {{ic|pcm0p}} to {{ic|pcm0c}} ({{ic|c}} is short for "capture", {{ic|p}} is for "playback").<br />
<br />
==== Sound quality (resampling quality) ====<br />
<br />
If you used PulseAudio with {{ic|1=resample-method = speex-float-10}} or {{ic|soxr-vhq}}, then you might consider uncommenting and changing {{ic|1=resample.quality = 4}} to {{ic|10}} or the maximum {{ic|15}} in {{ic|stream.properties}} block in both {{ic|/etc/pipewire/client.conf}} and {{ic|/etc/pipewire/pipewire-pulse.conf}} (copy them from {{ic|/usr/share/pipewire/}} if they do not exist). Do not forget to restart PipeWire (without sudo): {{ic|systemctl --user restart pipewire.service pipewire-pulse.socket}} (never forget {{ic|pipewire-pulse.socket}} if you want your config changes to be applied).<br />
<br />
There is a very little quality difference between {{ic|10}} and {{ic|15}}, but the CPU load difference is 2-3x. And the latency difference between {{ic|4}}, {{ic|10}}, {{ic|15}} is yet to be investigated by anybody. {{ic|1= resample.quality = 15}} on 44100→48000 Hz on Ryzen 2600 causes {{ic|pipewire}} or {{ic|pipewire-pulse}} processes to cause 4.0% one CPU core load.<br />
<br />
You can compare resamplers here: https://src.infinitewave.ca/ (do not pay attention to anything above 18 KHz and over 120 dB). speex is listed as "Xiph.org Speex".<br />
<br />
PipeWire uses its own resampling algorithm called Spa. Like with SoX's {{ic|sox}}, Speex's {{ic|speexenc}}, PipeWire includes its standalone version: {{ic|spa-resample}}. Usage:<br />
spa-resample -q 15 -f s24 -r 48000 input16bit44100orAnythingElse.wav output24bit48000hz.wav<br />
<br />
It is probably somehow possible to use other resamplers by creating your own sink. Or just use a plugin in your music player (e.g., Qmmp has SoX plugin).<br />
<br />
==== External sound card not activated after reconnect ====<br />
<br />
Check {{ic|~/.config/pipewire-media-session/default-profile}} if there is any entry with default profile "off" and remove it. If that does not help, remove all files from {{ic|~/.config/pipewire-media-session/}} and restart PipeWire using {{ic|systemctl --user restart pipewire.service}}.<br />
<br />
==== No Sound or pactl info shows Failure: Connection refused ====<br />
<br />
It means applications are unable to connect to the PipeWire-Pulse service, confirm that {{ic|/etc/pipewire/pipewire-pulse.conf}} exists and is not empty and restart PipeWire-Pulse using {{ic|systemctl --user restart pipewire-pulse.service}}.<br />
<br />
If that does not fix it, run {{ic|strace -f -o /tmp/pipe.txt pactl info}} and pastebin {{ic|/tmp/pipe.txt}} while seeking help on IRC ([ircs://irc.oftc.net/pipewire #pipewire] on OFTC) or the mailing-lists.<br />
<br />
==== Low audio quality on Bluetooth ====<br />
<br />
In case Bluetooth playback stutters, check the [[unit status]] of the {{ic|pipewire.service}} user unit for errors similar as below:<br />
<br />
Feb 17 18:23:01 HOST pipewire[249297]: (bluez_input.18:54:CF:04:00:56.a2dp-sink-60) client too slow! rate:512/48000 pos:370688 status:triggered<br />
<br />
If they appear, check the currently selected codec using {{ic|pactl list sinks}} and try changing it by setting {{ic|bluez5.codecs}} to one of {{ic|sbc aac ldac aptx aptx_hd}}:<br />
<br />
{{hc|/etc/pipewire/media-session.d/bluez-monitor.conf (or ~/.config/pipewire/media-session.d/bluez-monitor.conf)|output=<br />
...<br />
properties = {<br />
...<br />
bluez5.codecs = [sbc]<br />
...<br />
}}<br />
<br />
Try enabling mSBC support (fixes mic on Sony 1000XM3, i.e. Headphones WH-1000XM3 and Earbuds WF-1000XM3):<br />
<br />
{{hc|/etc/pipewire/media-session.d/bluez-monitor.conf (or ~/.config/pipewire/media-session.d/bluez-monitor.conf)|output=<br />
...<br />
properties = {<br />
...<br />
bluez5.enable-msbc = true<br />
...<br />
}}<br />
<br />
Restart PipeWire by [[restart]]ing the {{ic|pipewire.service}} user unit for the changes to take effect.<br />
<br />
==== No devices detected after PipeWire update and reboot (git / >=0.3.23) ====<br />
<br />
As of commit [https://gitlab.freedesktop.org/pipewire/pipewire/-/commit/012a68f8ef33705f1a40ec8ac294b8cce7f6aa88 012a68f8], a new [[user unit]] has been added which is disabled by default, meaning there is no ''pipewire-media-session'' running on user login. Run this program on login by [[enabling]] the {{ic|pipewire-media-session.service}} user unit.<br />
<br />
If the user or the package manager have not sorted out the configuration file changes after update, then another instance of ''pipewire-media-session'' might be running in {{ic|pipewire.service}}. Verify whether this is the case by checking the [[unit status]] of the {{ic|pipewire.service}} user unit. If it shows ''pipewire'' and ''pipewire-media-session'' running, update your system and/or user configuration:<br />
<br />
{{hc|/etc/pipewire/pipewire.conf (or ~/.config/pipewire/pipewire.conf)|output=<br />
...<br />
context.exec = {<br />
...<br />
# Line below should be commented out<br />
#"/usr/bin/pipewire-media-session" = { args = "" }<br />
...<br />
}}<br />
<br />
==== Noticeable audio delay when starting playback ====<br />
<br />
This is caused by node suspension when inactive. It can be disabled by editing {{ic|/etc/pipewire/media-session.d/*-monitor.conf}} depending on where the delay occurs and changing property {{ic|session.suspend-timeout-seconds}} to 0 to disable or to experiment with other values and see what works. Alternatively you can comment out the line {{ic|suspend-node}} in {{ic|/etc/pipewire/media-session.d/media-session.conf}}. Restart both the {{ic|pipewire}} and {{ic|pipewire-pulse}} systemd services to apply these changes, or alternatively reboot.<br />
<br />
If you are using {{ic|wireplumber}} instead of {{ic|pipewire-media-session}}, then you can copy the example config file located at {{ic|/usr/share/wireplumber/main.lua.d/50-alsa-config.lua}} to {{ic|/etc/wireplumber/main.lua.d/50-alsa-config.lua}} and add the line {{ic|1=["session.suspend-timeout-seconds"] = 0}} to section {{ic|apply_properties}} at the end of the file.<br />
<br />
==== Audio cutting out when multiple streams start playing ====<br />
<br />
This problem can typically be diagnosed by running {{ic|journalctl --user -b -u pipewire-pulse}} and finding lines similar to:<br />
<br />
pipewire-pulse[21740]: pulse-server 0x56009b9d5de0: [Nightly] UNDERFLOW channel:0 offset:370676 underrun:940<br />
<br />
According to the [https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Troubleshooting#underrununderflow-and-broken-pipe-errors official PipeWire troubleshooting guide], to solve this problem edit {{ic|/etc/pipewire/media-session.d/alsa-monitor.conf}}, uncomment the line saying {{ic|1=api.alsa.headroom = 0}} and change its value to {{ic|1024}}.<br />
<br />
If you are using {{ic|wireplumber}} instead of {{ic|pipewire-media-session}}, then you can copy the example config file located at {{ic|/usr/share/wireplumber/main.lua.d/50-alsa-config.lua}} to {{ic|/etc/wireplumber/main.lua.d/50-alsa-config.lua}} and at the end of the file, under section {{ic|apply_properties}}, change {{ic|1=--["api.alsa.headroom"] = 0}} to {{ic|1=["api.alsa.headroom"] = 1024}}<br />
<br />
==== Audio is distorted ====<br />
<br />
* For microphones, try navigating to the card that is having issues after running {{ic|alsamixer}} and use the arrow keys to reduce any "Mic Boost" or "Internal Mic Boost" options.<br />
* Follow [[#Changing the sample rate]], reducing the sample rate to to {{ic|44100}} (44.1 kHz).<br />
<br />
==== Audio problems after standby ====<br />
<br />
If the sound is missing or otherwise garbled after waking the machine up from sleep, it might help to reinitialize ALSA:<br />
<br />
# alsactl init<br />
<br />
==== High latency with USB DACs (e.g. Schiit DACs) ====<br />
<br />
Changing sample rates or formats might help reduce latency with some DACs such as Schiit Hel 2.[https://www.reddit.com/r/osugame/comments/msifdd/usb_dacamp_and_audio_lag/]<br />
Using matching rules in ''pipewire-media-session'' we can set properties for devices.[https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Config-pipewire-media-session#matching-rules]<br />
<br />
Copy the default configuration of {{ic|alsa-monitor.conf}} for ''pipewire-media-session'' into either {{ic|/etc/pipewire/media-session.d}} or {{ic|~/.config/pipewire/media-session.d}}.<br />
Then append a new rule-block similar to the following one:<br />
<br />
{{hc|/etc/pipewire/media-session.d/alsa-monitor.conf (or ~/.config/pipewire/media-session.d/alsa-monitor.conf)|output=<br />
...<br />
rules = {<br />
...<br />
{<br />
matches = [<br />
{<br />
node.name = "alsa_output.<name of node>"<br />
}<br />
]<br />
actions = {<br />
update-props = {<br />
audio.format = "S24_3LE"<br />
audio.rate = 96000<br />
# Following value should be doubled until audio doesn't cut out or other issues stop occurring<br />
api.alsa.period-size = 128<br />
...<br />
}}<br />
<br />
{{ic|alsa_output.<name of node>}} node can be obtained using {{ic|pw-top}}.<br />
<br />
Your DAC might support a different format or sample rate. You can check what your DAC supports by querying [[ALSA]]:<br />
<br />
First get the card number of your DAC:<br />
<br />
{{hc|$ aplay -l|<br />
...<br />
card 3: S2 [Schiit Hel 2], device 0: USB Audio [USB Audio]<br />
Subdevices: 0/1<br />
Subdevice #0: subdevice #0<br />
...<br />
}}<br />
<br />
So in this example it would be card 3.<br />
Get all supported sample rates and formats:<br />
<br />
{{hc|$ cat /proc/asound/cardX/streamX|<br />
...<br />
Playback:<br />
...<br />
Interface 1<br />
Altset 1<br />
Format: S16_LE<br />
Channels: 2<br />
Endpoint: 0x05 (5 OUT) (ASYNC)<br />
Rates: 44100, 48000, 88200, 96000, 176400, 192000, 352800, 384000<br />
Data packet interval: 125 us<br />
Bits: 16<br />
...<br />
Interface 1<br />
Altset 2<br />
Format: S24_3LE<br />
Channels: 2<br />
Endpoint: 0x05 (5 OUT) (ASYNC)<br />
Rates: 44100, 48000, 88200, 96000, 176400, 192000, 352800, 384000<br />
Data packet interval: 125 us<br />
Bits: 24<br />
...<br />
Interface 1<br />
Altset 3<br />
Format: S32_LE<br />
Channels: 2<br />
Endpoint: 0x05 (5 OUT) (ASYNC)<br />
Rates: 44100, 48000, 88200, 96000, 176400, 192000, 352800, 384000<br />
Data packet interval: 125 us<br />
Bits: 32<br />
...<br />
...<br />
}}<br />
<br />
In this case {{ic|S16_LE, S24_3LE, S32_LE}} are the supported formats and {{ic|44100, 48000, 88200, 96000, 176400, 192000, 352800, 384000}} are the supported sample rates across all formats.<br />
<br />
=== Video ===<br />
<br />
==== OBS (etc.) display nothing, even if they ask for a window/screen ====<br />
<br />
If you are sure that you have {{Pkg|xdg-desktop-portal}} installed as well as either {{Pkg|xdg-desktop-portal-gtk}} or {{Pkg|xdg-desktop-portal-kde}}, check the running state of the daemons.<br />
<br />
In OBS, if everything is working, you should see this in {{ic|stdout}}:<br />
<br />
...<br />
info: [pipewire] desktop selected, setting up screencast<br />
info: [pipewire] created stream 0x5632d7456850<br />
info: [pipewire] playing stream…<br />
<br />
For multi-monitor setups the {{Pkg|slurp}} package will allow to capture of all the screens.<br />
<br />
== See also ==<br />
<br />
* [https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/home Wiki] — PipeWire Wiki on Freedesktop GitLab<br />
* [https://blogs.gnome.org/uraeus/2018/01/26/an-update-on-pipewire-the-multimedia-revolution-an-update/ Pipewire Update Blog Post] — Blog post from January 2018 outlining the state of PipeWire at the time<br />
* [https://blogs.gnome.org/uraeus/2020/09/04/pipewire-late-summer-update-2020/ PipeWire Late Summer Update 2020] — Blog post from September 2020</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=689656PCI passthrough via OVMF2021-07-28T19:28:51Z<p>NeoTheFox: Undo revision 689652 by NeoTheFox (talk) I made a mistake</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
[[zh-hans:PCI passthrough via OVMF]]<br />
{{Related articles start}}<br />
{{Related|Intel GVT-g}}<br />
{{Related|PCI passthrough via OVMF/Examples}}<br />
{{Related articles end}}<br />
<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of [[QEMU]], it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)].<br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=873&0_VTD=True List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Note|<br />
* IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
* VT-d stands for ''Intel Virtualization Technology for Directed I/O'' and should not be confused with VT-x ''Intel Virtualization Technology''. VT-x allows one hardware platform to function as multiple “virtual” platforms while VT-d improves security and reliability of the systems and also improves performance of I/O devices in virtualized environments.<br />
}}<br />
<br />
Using IOMMU opens to features like PCI passthrough and memory protection from faulty or malicious devices, see [[Wikipedia:Input-output memory management unit#Advantages]] and [https://www.quora.com/Memory-Management-computer-programming/Could-you-explain-IOMMU-in-plain-English Memory Management (computer programming): Could you explain IOMMU in plain English?].<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is supported by the CPU and enabled in the BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
Manually enable IOMMU support by setting the correct [[kernel parameter]] depending on the type of CPU in use:<br />
<br />
* For Intel CPUs (VT-d) set {{ic|1=intel_iommu=on}}. Since the kernel config option CONFIG_INTEL_IOMMU_DEFAULT_ON is not set in {{Pkg|linux}}.<br />
* For AMD CPUs (AMD-Vi), it's on if kernel detects IOMMU HW support from BIOS.<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check [[dmesg]] to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|# dmesg {{!}} grep -i -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do<br />
echo "IOMMU Group ${g##*/}:"<br />
for d in $g/devices/*; do<br />
echo -e "\t$(lspci -nns ${d##*/})"<br />
done;<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 1:<br />
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
IOMMU Group 2:<br />
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:0e31] (rev 04)<br />
IOMMU Group 4:<br />
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:0e2d] (rev 04)<br />
IOMMU Group 10:<br />
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:0e26] (rev 04)<br />
IOMMU Group 13:<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1:<br />
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in to your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the preferred method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{Warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this - your motherboard should be set to display using the host GPU.}}<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub did, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use.<br />
<br />
=== Binding vfio-pci via device ID ===<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13:<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|<br />
* You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.<br />
* If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.<br />
* Binding the audio device ({{ic|10de:0fbb}} in above's example) is optional. Libvirt is able to unbind it from the {{ic|snd_hda_intel}} driver on its own.<br />
}}<br />
<br />
Two methods exist for providing the device IDs. Specifying them via [[kernel parameters]] has the advantage of being able to easily edit, remove, or undo any breaking changes via your boot loader:<br />
<br />
vfio-pci.ids=10de:13c2,10de:0fbb<br />
<br />
Alternatively, the IDs may be added to a modprobe conf file. Since these conf files are embedded in the initramfs image, any changes require regenerating a new image each time:<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
=== Loading vfio-pci early ===<br />
<br />
==== mkinitcpio ====<br />
<br />
Since Arch's {{Pkg|linux}} has vfio-pci built as a module, we need to force it to load early before the graphics drivers have a chance to bind to the card. To ensure that, add {{ic|vfio_pci}}, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio_pci vfio vfio_iommu_type1 vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]].<br />
<br />
==== booster ====<br />
<br />
Similar to mkinitcpio you need to specify modules to load early:<br />
{{hc|/etc/booster.yaml|2=<br />
modules_force_load: vfio_pci,vfio,vfio_iommu_type1,vfio_virqfd<br />
}}<br />
<br />
and then [[Booster#Regenerate_booster_images|regenerate the initramfs]].<br />
<br />
==== dracut ====<br />
<br />
dracut's early loading mechanism is configured via kernel parameters. To load vfio-pci early, add both the [[#Binding vfio-pci via device ID|device ids]] and the following line to your [[kernel parameters]]:<br />
<br />
rd.driver.pre=vfio_pci<br />
<br />
We also need to add all the vfio drivers to the initramfs. Add the following file to {{ic|/etc/dracut.conf.d}}:<br />
<br />
{{hc|10-vfio.conf|2=<br />
add_drivers+=" vfio_pci vfio vfio_iommu_type1 vfio_virqfd "<br />
}}<br />
<br />
As with mkinitcpio, you must regenerate the initramfs. See [[dracut]] for more details.<br />
<br />
=== Verifying that the configuration worked ===<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|# dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in ''dmesg'' output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it is possible to use SeaBIOS to get similar results to an actual PCI passthrough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
{{Accuracy|{{Pkg|libvirt}} depends on ''ebtables'', not {{pkg|iptables-nft}} which is just a translation layer.}}<br />
<br />
Install {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|edk2-ovmf}}, and {{Pkg|virt-manager}}. For the default network connection, {{pkg|iptables-nft}} and {{pkg|dnsmasq}} are required.<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
You may also need to [https://wiki.libvirt.org/page/Networking#NAT_forwarding_.28aka_.22virtual_networks.22.29 activate the default libvirt network]:<br />
# virsh net-autostart default<br />
# virsh net-start default<br />
<br />
{{Note|The default libvirt network will only be listed if the virsh command is run as root.}}<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self-explanatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
However, you should pay special attention to the following steps:<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that:<br />
** Your hypervisor is running as a system session and not a user session. This can be verified [https://i.ibb.co/N1XZCdp/Deepin-Screenshot-select-area-20190125113216.png by clicking, then hovering] over the session in virt-manager. If you are accidentally running it as a user session, you must open a new connection by clicking "File" > "Add Connection..", then select the option from the drop-down menu station "QEMU/KVM" and not "QEMU/KVM user session".<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to either type it by hand or by using {{ic|virt-xml ''vmname'' --edit --cpu host-passthrough}}. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, it's easier to setup [[#Virtio disk]] before installing<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine. It is possible you will be dropped into the UEFI menu instead of starting the installation upon powering your VM for the first time. Sometimes the correct ISO file was not automatically detected and you will need to manually specify the drive to boot. By typing exit and navigating to "boot manager" you will enter a menu that allows you to choose between devices.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it is now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. For example, remove the following sections from your XML file:<br />
<br />
{{bc|1=<nowiki/><br />
<channel type="spicevmc"><br />
...<br />
</channel><br />
<input type="tablet" bus="usb"><br />
...<br />
</input><br />
<input type="mouse" bus="ps2"/><br />
<input type="keyboard" bus="ps2"/><br />
<graphics type="spice" autoport="yes"><br />
...<br />
</graphics><br />
<video><br />
<model type="qxl" .../><br />
...<br />
</video><br />
}}<br />
<br />
Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. This may be done by using {{ic|Add Hardware > USB Host Device}}.<br />
<br />
At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Video card driver virtualisation detection ===<br />
<br />
Video card drivers by AMD incorporate very basic virtual machine detection targeting Hyper-V extensions. Should this detection mechanism trigger the drivers will refuse to run (resulting in a black screen).<br />
<br />
It is therefore required to modify the reported Hyper-V vendor ID:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<features><br />
...<br />
<hyperv><br />
...<br />
<vendor_id state='on' value='randomid'/><br />
...<br />
</hyperv><br />
...<br />
</features><br />
...<br />
}}<br />
<br />
Nvidia guest drivers prior to version 465 exhibited a similar behaviour which resulted in a generic error 43 in the card's device manager status. Systems using these older drivers therefore also need the above modification. In addition, they also require hiding the KVM CPU leaf:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<features><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
</features><br />
...<br />
}}<br />
<br />
Note that the above steps do not equate 'hiding' the virtual machine from Windows or any drivers/programs running in the VM. Also, various other issues not related to any detection mechanism referred to here can also trigger error 43.<br />
<br />
=== Passing keyboard/mouse via Evdev ===<br />
<br />
If you do not have a spare mouse or keyboard to dedicate to your guest, and you do not want to suffer from the video overhead of Spice, you can setup evdev to share them between your Linux host and your virtual machine.<br />
<br />
{{Note|Press both left and right '''Ctrl''' keys at the same time to swap control between the host and the guest.}}<br />
<br />
First, find your keyboard and mouse devices in {{ic|/dev/input/by-id/}}. Only devices with {{ic|event}} in their name are valid. You may find multiple devices associated to your mouse or keyboard, so try {{ic|cat /dev/input/by-id/''device_id''}} and either hit some keys on the keyboard or wiggle your mouse to see if input comes through, if so you have got the right device. Now add those devices to your configuration:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<devices><br />
...<br />
<input type='evdev'><br />
<source dev='/dev/input/by-id/MOUSE_NAME'/><br />
</input><br />
<input type='evdev'><br />
<source dev='/dev/input/by-id/KEYBOARD_NAME' grab='all' repeat='on'/><br />
</input><br />
...<br />
</devices><br />
}}<br />
<br />
Replace {{ic|MOUSE_NAME}} and {{ic|KEYBOARD_NAME}} with your device path. Now you can startup the guest OS and test swapping control of your mouse and keyboard between the host and guest by pressing both the left and right control keys at the same time.<br />
<br />
You may also consider switching from PS/2 to Virtio inputs in your configurations. Add these two devices:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<input type='mouse' bus='virtio'/><br />
<input type='keyboard' bus='virtio'/><br />
...<br />
}}<br />
<br />
The virtio input devices will not actually be used until the guest drivers are installed. QEMU will continue to send key events to the PS2 devices until it detects the virtio input driver initialization. Note that the PS2 devices cannot be removed as they are an internal function of the emulated Q35/440FX chipsets.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate Linux/Windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. As such, the local CPU cache benefits (L1/L2/L3) are lost each time the host scheduler reschedules the virtual CPU thread on a different physical CPU. This can noticeably harm performance on the guest. CPU pinning aims to resolve this by limiting which physical CPUs the virtual CPUs are allowed to run on. The ideal setup is a one to one mapping such that the virtual CPU cores match physical CPU cores while taking hyperthreading/SMT into account.<br />
<br />
In addition, in some modern CPUs, groups of cores often share a common L3 cache. In such cases, care should be taken to pin exactly those physical cores that share a particular L3. Failing to do so might lead to cache evictions which could result in microstutters.<br />
<br />
{{Note|For certain users enabling CPU pinning may introduce stuttering and short hangs, especially with the MuQSS scheduler (present in linux-ck and linux-zen kernels). You might want to try disabling pinning first if you experience similar issues, which effectively trades maximum performance for responsiveness at all times.}}<br />
<br />
==== CPU topology ====<br />
<br />
Most modern CPUs support hardware multitasking, also known as hyper-threading on Intel CPUs or SMT on AMD CPUs. Hyper-threading/SMT is simply a very efficient way of running two threads on one CPU core at any given time. You will want to take into consideration that the CPU pinning you choose will greatly depend on what you do with your host while your VM is running.<br />
<br />
To find the topology for your CPU run {{ic|1=lscpu -e}}:<br />
<br />
{{Note|Pay special attention to the 4th column '''"CORE"''' as this shows the association of the Physical/Logical CPU cores as well as the 8th column '''"L3"''' which shows which cores are connected to which L3 cache.}}<br />
<br />
{{ic|lscpu -e}} on a 6c/12t Ryzen 5 1600:<br />
<br />
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ<br />
0 0 0 0 0:0:0:0 yes 3800.0000 1550.0000<br />
1 0 0 0 0:0:0:0 yes 3800.0000 1550.0000<br />
2 0 0 1 1:1:1:0 yes 3800.0000 1550.0000<br />
3 0 0 1 1:1:1:0 yes 3800.0000 1550.0000<br />
4 0 0 2 2:2:2:0 yes 3800.0000 1550.0000<br />
5 0 0 2 2:2:2:0 yes 3800.0000 1550.0000<br />
6 0 0 3 3:3:3:1 yes 3800.0000 1550.0000<br />
7 0 0 3 3:3:3:1 yes 3800.0000 1550.0000<br />
8 0 0 4 4:4:4:1 yes 3800.0000 1550.0000<br />
9 0 0 4 4:4:4:1 yes 3800.0000 1550.0000<br />
10 0 0 5 5:5:5:1 yes 3800.0000 1550.0000<br />
11 0 0 5 5:5:5:1 yes 3800.0000 1550.0000<br />
<br />
Considering the L3 mapping, it is recommended to pin and isolate CPUs 6–11. Pinning and isolating fewer than these (e.g. 8–11) would result in the host system making use of the L3 cache in core 6 and 7 which would eventually lead to cache evictions and therefore bad performance.<br />
<br />
{{Note|Ryzen 3000 ComboPi AGESA changes topology to match Intel example, even on prior generation CPUs. Above valid only on older AGESA.}}<br />
<br />
{{ic|lscpu -e}} on a 6c/12t Intel 8700k:<br />
<br />
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ<br />
0 0 0 0 0:0:0:0 yes 4600.0000 800.0000<br />
1 0 0 1 1:1:1:0 yes 4600.0000 800.0000<br />
2 0 0 2 2:2:2:0 yes 4600.0000 800.0000<br />
3 0 0 3 3:3:3:0 yes 4600.0000 800.0000<br />
4 0 0 4 4:4:4:0 yes 4600.0000 800.0000<br />
5 0 0 5 5:5:5:0 yes 4600.0000 800.0000<br />
6 0 0 0 0:0:0:0 yes 4600.0000 800.0000<br />
7 0 0 1 1:1:1:0 yes 4600.0000 800.0000<br />
8 0 0 2 2:2:2:0 yes 4600.0000 800.0000<br />
9 0 0 3 3:3:3:0 yes 4600.0000 800.0000<br />
10 0 0 4 4:4:4:0 yes 4600.0000 800.0000<br />
11 0 0 5 5:5:5:0 yes 4600.0000 800.0000<br />
<br />
Since all cores are connected to the same L3 in this example, it does not matter much how many CPUs you pin and isolate as long as you do it in the proper thread pairs. For instance, (0, 6), (1, 7), etc.<br />
<br />
As we see above, with AMD '''Core 0''' is sequential with '''CPU 0 & 1''', whereas Intel places '''Core 0''' on '''CPU 0 & 6'''.<br />
<br />
{{Tip|You can view your systems topology in diagram form, which may help some users. If you have {{Pkg|hwloc}} installed, run {{ic|lstopo}} to generate a helpful image of your CPU/Thread groupings.}}<br />
<br />
If you do not need all cores for the guest, it would then be preferable to leave at the very least one core for the host. Choosing which cores one to use for the host or guest should be based on the specific hardware characteristics of your CPU, however '''Core 0''' is a good choice for the host in most cases. If any cores are reserved for the host, it is recommended to pin the emulator and iothreads, if used, to the host cores rather than the VCPUs. This may improve performance and reduce latency for the guest since those threads will not pollute the cache or contend for scheduling with the guest VCPU threads. If all cores are passed to the guest, there is no need or benefit to pinning the emulator or iothreads.<br />
<br />
==== XML examples ====<br />
<br />
{{Note|Do not use the '''iothread''' lines from the XML examples shown below if you have not added an '''iothread''' to your disk controller. '''iothread''''s only work on '''virtio-scsi''' or '''virtio-blk''' devices.}}<br />
<br />
===== 4c/1t CPU w/o Hyperthreading Example =====<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='0'/><br />
<vcpupin vcpu='1' cpuset='1'/><br />
<vcpupin vcpu='2' cpuset='2'/><br />
<vcpupin vcpu='3' cpuset='3'/><br />
</cputune><br />
...<br />
}}<br />
<br />
===== 4c/2t Intel/AMD CPU example (after ComboPI AGESA bios update) =====<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<vcpu placement='static'>8</vcpu><br />
<iothreads>1</iothreads><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='8'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='9'/><br />
<vcpupin vcpu='4' cpuset='4'/><br />
<vcpupin vcpu='5' cpuset='10'/><br />
<vcpupin vcpu='6' cpuset='5'/><br />
<vcpupin vcpu='7' cpuset='11'/><br />
<emulatorpin cpuset='0,6'/><br />
<iothreadpin iothread='1' cpuset='0,6'/><br />
</cputune><br />
...<br />
<cpu mode='host-passthrough'><br />
<topology sockets='1' cores='4' threads='2'/><br />
</cpu><br />
...<br />
}}<br />
<br />
===== 4c/2t AMD CPU example (Before ComboPi AGESA bios update) =====<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<vcpu placement='static'>8</vcpu><br />
<iothreads>1</iothreads><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='3'/><br />
<vcpupin vcpu='2' cpuset='4'/><br />
<vcpupin vcpu='3' cpuset='5'/><br />
<vcpupin vcpu='4' cpuset='6'/><br />
<vcpupin vcpu='5' cpuset='7'/><br />
<vcpupin vcpu='6' cpuset='8'/><br />
<vcpupin vcpu='7' cpuset='9'/><br />
<emulatorpin cpuset='0-1'/><br />
<iothreadpin iothread='1' cpuset='0-1'/><br />
</cputune><br />
...<br />
<cpu mode='host-passthrough'><br />
<topology sockets='1' cores='4' threads='2'/><br />
</cpu><br />
...<br />
}}<br />
<br />
{{Note|If further CPU isolation is needed, consider using the '''isolcpus''' kernel command-line parameter on the unused physical/logical cores.}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, you may want to pin your VM threads across all of your cores, so that the VM can fully take advantage of the spare CPU time the host has available. Be aware that pinning all physical and logical cores of your CPU could induce latency in the guest VM.<br />
<br />
=== Huge memory pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information across multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4 GiB of memory divided into 4 KiB pages (which is the default size for normal pages) for a total of 1.04 million pages, meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession.<br />
<br />
==== Transparent huge pages ====<br />
<br />
QEMU will use 2MiB sized transparent huge pages automatically without any explicit configuration in QEMU or Libvirt, subject to some important caveats. When using VFIO the pages are locked in at boot time and transparent huge pages are allocated up front when the VM first boots. If the kernel memory is highly fragmented, or the VM is using a majority of the remaining free memory, it is likely that the kernel will not have enough 2MiB pages to fully satisfy the allocation. In such a case, it silently fails by using a mix of 2MiB and 4KiB pages. Since the pages are locked in VFIO mode, the kernel will not be able to convert those 4KiB pages to huge after the VM starts either. The number of available 2MiB huge pages available to THP is the same as via the [[#Dynamic huge pages]] mechanism described in the following sections.<br />
<br />
To check how much memory THP is using globally:<br />
<br />
{{hc|$ grep AnonHugePages /proc/meminfo|<br />
AnonHugePages: 8091648 kB<br />
}}<br />
<br />
To check a specific QEMU instance. QEMU's PID must be substituted in the grep command:<br />
<br />
{{hc|$ grep -P 'AnonHugePages:\s+(?!0)\d+' /proc/[PID]/smaps|<br />
AnonHugePages: 8087552 kB<br />
}}<br />
<br />
In this example, the VM was allocated 8388608KiB of memory, but only 8087552KiB was available via THP. The remaining 301056KiB are allocated as 4KiB pages. Aside from manually checking, there is no indication when partial allocations occur. As such, THP's effectiveness is very much dependent on the host system's memory fragmentation at the time of VM startup. If this trade off is unacceptable or strict guarantees are required, [[#Static huge pages]] is recommended.<br />
<br />
Arch kernels have THP compiled in and enabled by default with {{ic|1=/sys/kernel/mm/transparent_hugepage/enabled}} set to {{ic|1=madvise}} mode.<br />
<br />
Also when trying to allocate pages for {{ic|virsh}}, e.g. {{ic|1=virsh allocpages 2M 2048}}, you might need to free up cached RAM. This is because your system might have allocated RAM for cache resources over time. One possible solution is to drop the page cache of your system:<br />
<br />
{{ic|# echo 1 > /proc/sys/vm/drop_caches}}<br />
<br />
==== Static huge pages ====<br />
<br />
While transparent huge pages should work in the vast majority of cases, they can also be allocated statically during boot. This should only be needed to make use 1 GiB hugepages on machines that support it, since transparent huge pages normally only go up to 2 MiB.<br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4 GiBs worth of huge pages on a machine with 8 GiB of memory will only leave you with 4 GiB of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
{{Note|The described procedures have some drawbacks but will not necessarily net you a great performance advantage. According to [https://developers.redhat.com/blog/2021/04/27/benchmarking-transparent-versus-1gib-static-huge-page-performance-in-linux-virtual-machines#benchmarks Red Hat benchmarks], you shouldn't expect more than a 2% performance gain from this over [[#Transparent huge pages]].}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel command line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048 KiB per huge page creates 2 GiB worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GiB huge page support could be verified by {{ic|grep pdpe1gb /proc/cpuinfo}}. Setting 1 GiB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
}}<br />
<br />
==== Dynamic huge pages ====<br />
<br />
{{Accuracy|Need futher testing if this variant as effective as static one}}<br />
<br />
Hugepages could be allocated manually via {{ic|vm.nr_overcommit_hugepages}} [[sysctl]] parameter.<br />
<br />
{{hc|/etc/sysctl.d/10-kvm.conf|2=<br />
vm.nr_hugepages = 0<br />
vm.nr_overcommit_hugepages = ''num''<br />
}}<br />
<br />
Where {{ic|''num''}} - is the number of huge pages, which default size if 2 MiB.<br />
Pages will be automatically allocated, and freed after VM stops.<br />
<br />
More manual way:<br />
<br />
# echo ''num'' > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages<br />
# echo ''num'' > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages<br />
<br />
For 2 MiB and 1 GiB page size respectively.<br />
And they should be manually freed in the same way.<br />
<br />
It is hardly recommended to drop caches, compact memory and wait couple of seconds before starting VM, as there could be not enough free contiguous memory for required huge pages blocks. Especially after some uptime of the host system.<br />
<br />
# echo 3 > /proc/sys/vm/drop_caches<br />
# echo 1 > /proc/sys/vm/compact_memory<br />
<br />
Theoretically, 1 GiB pages works as 2 MiB. But practically - no guaranteed way was found to get contiguous 1 GiB memory blocks. Each consequent request of 1 GiB blocks lead to lesser and lesser dynamically allocated count.<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
{{Warning|Depending on your processor, it might actually be detrimental to performance to force the whole CPU to run at full frequency at all times. For instance, modern AMD processors (Zen 2 and Zen 3) depend heavily on being able to scale individual cores in separate core complexes for optimal thermal distribution. If the whole CPU is running at full frequency at all times, there's less headroom for invididual cores to clock high, resulting in worse performance for processes that aren't heavily multithreaded such as games. This should be benchmarked in your VM.}}<br />
<br />
=== Isolating pinned CPUs ===<br />
<br />
CPU pinning by itself will not prevent other host processes from running on the pinned CPUs. Properly isolating the pinned CPUs can reduce latency in the guest VM.<br />
<br />
==== With isolcpus kernel parameter ====<br />
<br />
In this example, let us assume you are using CPUs 4-7.<br />
Use the [[kernel parameters]] {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. For example:<br />
<br />
isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.removeddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this Removeddit mirror of a Reddit thread] for more info. ([https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ The original thread] is worthless because of deleted comments.)<br />
<br />
==== Dynamically isolating CPUs ====<br />
<br />
The isolcpus kernel parameter will permanently reserve CPU cores, even when the guest is not running. A more flexible alternative is to dynamically isolate CPUs when starting the guest. This can be achieved with the following alternatives:<br />
<br />
* {{AUR|cpuset-git}} ([https://www.redhat.com/archives/vfio-users/2016-September/msg00072.html vfio-users post], [https://rokups.github.io/#!pages/gaming-vm-performance.md blog post], [https://github.com/PassthroughPOST/VFIO-Tools/blob/master/libvirt_hooks/hooks/cset.sh example script])<br />
* {{AUR|vfio-isolate}}<br />
* systemd<br />
<br />
===== Example with systemd =====<br />
<br />
In this example, we assume a host with 12 CPUs, where CPUs 2-5 and 8-11 are [[PCI_passthrough_via_OVMF#CPU_pinning|pinned]] to the guest. Then run the following to isolate the host to CPUs 0, 1, 6, and 7:<br />
<br />
# systemctl set-property --runtime -- user.slice AllowedCPUs=0,1,6,7<br />
# systemctl set-property --runtime -- system.slice AllowedCPUs=0,1,6,7<br />
# systemctl set-property --runtime -- init.scope AllowedCPUs=0,1,6,7<br />
<br />
After shutting down the guest, run the following to reallocate all 12 CPUs back to the host:<br />
<br />
# systemctl set-property --runtime -- user.slice AllowedCPUs=0-11<br />
# systemctl set-property --runtime -- system.slice AllowedCPUs=0-11<br />
# systemctl set-property --runtime -- init.scope AllowedCPUs=0-11<br />
<br />
You can use a [https://libvirt.org/hooks.html libvirt hook] to automatically run the above at startup/shutdown of the guest like so:<br />
<br />
Create or edit {{ic|/etc/libvirt/hooks/qemu}} with the following content.<br />
<br />
{{hc|/etc/libvirt/hooks/qemu|2=<nowiki><br />
#!/bin/bash<br />
<br />
command=$2<br />
<br />
if [[ $command == "started" ]]; then<br />
systemctl set-property --runtime -- system.slice AllowedCPUs=0,1,6,7<br />
systemctl set-property --runtime -- user.slice AllowedCPUs=0,1,6,7<br />
systemctl set-property --runtime -- init.slice AllowedCPUs=0,1,6,7<br />
elif [[ $command == "release" ]]; then<br />
systemctl set-property --runtime -- system.slice AllowedCPUs=0-11<br />
systemctl set-property --runtime -- user.slice AllowedCPUs=0-11<br />
systemctl set-property --runtime -- init.slice AllowedCPUs=0-11<br />
fi<br />
</nowiki>}}<br />
<br />
Afterwards make it executable: {{ic|chmod +x /etc/libvirt/hooks/qemu}}.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and then start your VM. If you create some heavily multithreaded load on your host now, you should see that it keeps your chosen CPUs free from load while the VM can still make use of it. You should also see those CPUs automatically getting fully used by your host once you terminate the VM.<br />
<br />
More examples are contained in the following reddit threads: [https://www.reddit.com/r/VFIO/comments/ebe3l5/deprecated_isolcpus_workaround/fem8jgk] [https://www.reddit.com/r/VFIO/comments/gyem88/noob_question_how_to_isolate_cpu_cores/ftaqno1/] [https://www.reddit.com/r/VFIO/comments/ij25rg/splitting_ht_cores_between_host_and_vm/]<br />
<br />
Note that this requires systemd 244 or higher, and [[cgroups|cgroups v2]], which is now enabled by default.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Starting with QEMU 3.1 the TOPOEXT cpuid flag is disabled by default. In order to use hyperthreading (SMT) on AMD CPUs you need to manually enable it:<br />
<br />
<cpu mode='host-passthrough' check='none'><br />
<topology sockets='1' cores='4' threads='2'/><br />
<feature policy='require' name='topoext'/><br />
</cpu><br />
<br />
commit: https://git.qemu.org/?p=qemu.git;a=commit;h=7210a02c58572b2686a3a8d610c6628f87864aed<br />
<br />
=== Virtio disk ===<br />
<br />
{{Merge|QEMU#Installing virtio drivers|Off-topic.|section=Moving virtio disk section to QEMU}}<br />
<br />
The default disk types are SATA or IDE emulation out of the box. These controllers offer maximum compatibility but are not suited for efficient virtualization. Two accelerated models exist: {{ic|1=virtio-scsi}} for SCSI emulation and passthrough, or {{ic|1=virtio-blk}} for a more basic block device emulation.<br />
<br />
==== Drivers ====<br />
<br />
* Linux guests should support these out of the box on any modern kernel<br />
* macOS has {{ic|1=virtio-blk}} support starting in Mojave via {{ic|1=AppleVirtIO.kext}}<br />
* Windows needs the [https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/ Windows virtio drivers]. {{ic|1=virtio-scsi}} uses the {{ic|1=vioscsi}} driver. {{ic|1=virtio-blk}} uses the {{ic|1=viostor}} driver<br />
* Windows can be installed directly onto these disks by selecting 'load driver' on the installer disk selection menu. The windows iso and virtio driver iso should both be attached as regular SATA/IDE cdroms during the installation process<br />
* To switch boot disks to virtio on an existing Windows installation:<br />
** {{ic|1=virtio-blk}}: Add a temporary disk with bus {{ic|1=virtio}}, boot windows & load the driver for the disk, then shutdown and switch the boot disk disk bus to {{ic|1=virtio}}<br />
** {{ic|1=virtio-scsi}}: Add a scsi controller with model {{ic|1=virtio}}, boot windows & load the driver for the controller, then shutdown and switch the boot disk bus to {{ic|1=scsi}} (not virtio)<br />
<br />
==== Considerations ====<br />
<br />
* {{ic|1=virtio-scsi}} TRIM support is mature, all versions should support it. Traditionally, {{ic|1=virtio-scsi}} has been the preferred approach for this reason<br />
* {{ic|1=virtio-blk}} TRIM support is new, this requires requires qemu 4.0+, guest linux kernel 5.0+, guest windows drivers 0.1.173+<br />
* Thin provisioning works by enabling TRIM on a sparse image file: {{ic|1=discard='unmap'}}. Unused blocks will be freed and the disk usage will drop (works on both raw and qcow2). Actual on-disk size of a sparse image file may be checked with {{ic|1=du /path/to/disk.img}}<br />
* Thin provisioning can also work with block storage such as zfs zvols or thin lvm<br />
* Virt queue count will influence the number of threads inside the guest kernel used for IO processing, suggest using {{ic|1=queues='4'}} or more<br />
* Native mode ({{ic|1=io='native'}}) uses a single threaded model based on linux AIO, is a bit more CPU efficient but may have lower peak performance and does not allow host side caching to be used<br />
* Threaded mode ({{ic|1=io='threads'}}) will spawn dozens of threads on demand as the disk is used. This is less efficient but may perform better if there are enough host cores available to run them, and allows for host side caching to be used<br />
* Modern versions of libvirt will group the dynamic worker threads created when using threaded mode in with the iothread=1 cgroup for pinning purposes. Very old versions of libvirt left these in the emulator cgroup<br />
<br />
==== IO threads ====<br />
<br />
An IO thread is a dedicated thread for processing disk events, rather than using the main qemu emulator loop. This should not be confused with the worker threads spawned on demand with {{ic|1=io='threads'}}.<br />
<br />
* You can only use one iothread per disk controller. The thread must be assigned to a specific controller with {{ic|1=iothread='X'}} in the {{ic|1=<driver>}} tag. Furthermore, extra & unassigned iothreads will not be used and do nothing<br />
* In the case of {{ic|1=virtio-scsi}}, there is one controller for multiple scsi disks. The iothread is assigned on the controller: {{ic|1=<controller><driver iothread='X'>}}<br />
* In the case of {{ic|1=virtio-blk}}, each disk has its own controller. The iothread is assigned in the driver tag under the disk itself: {{ic|1=<disk><driver iothread='X'>}}<br />
* Since emulated disks incur a significant amount of CPU overhead, that can lead to vcpu stuttering under high disk load (especially high random IOPS). In this case it helps to pin the IO to different core(s) than your vcpus with {{ic|1=<iothreadpin>}}<br />
<br />
==== Examples with libvirt ====<br />
<br />
virtio-scsi + iothread + worker threads + host side writeback caching + full disk block device backend:<br />
<domain><br />
<devices><br />
<disk type='block' device='disk'><br />
<driver name='qemu' type='raw' cache='writeback' io='threads' discard='unmap'/><br />
<source dev='/dev/disk/by-id/ata-Samsung_SSD_840_EVO_1TB_S1D9NSAF206396F'/><br />
<target dev='sda' bus='scsi'/><br />
</disk><br />
<controller type='scsi' index='0' model='virtio-scsi'><br />
<driver iothread='1' queues='8'/><br />
</controller><br />
<br />
virtio-blk + iothread + native aio + no host caching + raw sparse image backend:<br />
<domain><br />
<devices><br />
<disk type='file' device='disk'><br />
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap' iothread='1' queues='8'/><br />
<source file='/var/lib/libvirt/images/pool/win10.img'/><br />
<target dev='vda' bus='virtio'/><br />
</disk><br />
<br />
Creating the iothreads:<br />
<domain><br />
<iothreads>1</iothreads><br />
<br />
Pinning iothreads:<br />
<domain><br />
<cputune><br />
<iothreadpin iothread='1' cpuset='0-1,6-7'/><br />
<br />
==== Example with virt-manager ====<br />
<br />
This will create a {{ic|1=virtio-blk}} device:<br />
# Open the VM preferences<br />
# Go to {{ic|Add Hardware > Storage}}<br />
# Create or choose a storage file<br />
# Select {{ic|Device Type: Disk device}} and {{ic|Bus type: VirtIO}}<br />
# Click Finish<br />
<br />
=== Virtio network ===<br />
<br />
The default NIC models rtl8139 or e1000 can be a bottleneck for gigabit+ speeds and have a significant amount of CPU overhead compared to {{ic|1=virtio-net}}.<br />
<br />
* Select {{ic|1=virtio}} as the model for the NIC with libvirt or use the {{ic|1=virtio-net-pci}} device in bare qemu<br />
* Windows needs the {{ic|1=NetKVM}} driver from [https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/ Windows virtio drivers]<br />
* Virtio uses vhost-net by default for in-kernel packet processing without exiting to userspace<br />
* Multiqueue can enabled for a further speedup with multiple connections but typically will not boost single stream speeds. For libvirt add {{ic|1=<driver queues='8'/>}} under the interface tag<br />
* Zero copy transmit may also be enabled on macvtap by setting the module parameter {{ic|1=vhost_net.experimental_zcopytx=1}} but this may actually have worse performance, see [https://github.com/torvalds/linux/commit/098eadce3c622c07b328d0a43dda379b38cf7c5e commit]<br />
<br />
Libvirt example with a bridge:<br />
<br />
<interface type='bridge'><br />
<mac address="52:54:00:6d:6e:2e"/><br />
<source bridge='br0'/><br />
<model type='virtio'/><br />
<driver queues='8'/><br />
</interface><br />
<br />
=== Further tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide].<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how vfio-pci uses your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthrough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/local/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|1=<br />
#!/bin/sh<br />
<br />
for i in /sys/bus/pci/devices/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" {{!}} sed -e "s/0$/1/")"<br />
USB="$(echo "$GPU" {{!}} sed -e "s/0$/2/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
if [ -d "$USB" ]; then<br />
echo "vfio-pci" > "$USB/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|1=<br />
#!/bin/sh<br />
<br />
DEVS="0000:03:00.0 0000:03:00.1"<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
}}<br />
<br />
==== Script installation ====<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}:<br />
<br />
# Add {{ic|modconf}} to the [[mkinitcpio#HOOKS|HOOKS]] array and {{ic|/usr/local/bin/vfio-pci-override.sh}} to the [[mkinitcpio#BINARIES and FILES|FILES]] array.<br />
<br />
Edit {{ic|/etc/modprobe.d/vfio.conf}}:<br />
<br />
# Add the following line: {{ic|install vfio-pci /usr/local/bin/vfio-pci-override.sh}}<br />
# [[Regenerate the initramfs]] and reboot.<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|This is related to VBIOS issues and should be moved into a separate section regarding VBIOS compatibility.|section=UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that is not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.io/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total mebibytes + 10<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MiB + 10 = 25.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 25.82 is bigger than 16 we should choose 32.<br />
<br />
Next create a configuration file to create the shared memory file on boot<br />
<br />
{{hc|/etc/tmpfiles.d/10-looking-glass.conf|2=<br />
f /dev/shm/looking-glass 0660 '''user''' kvm -<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Ask systemd-tmpfiles to create the shared memory file now without waiting to next boot<br />
<br />
# systemd-tmpfiles --create /etc/tmpfiles.d/10-looking-glass.conf<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to go into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download the signed driver [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/upstream-virtio/ from Red Hat].<br />
<br />
Once the driver is installed you must download a matching [https://looking-glass.io/downloads looking-glass-host] package that matches the client you will install from AUR, and install it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]. The recent version will automatically install a service that starts the daemon on boot. The logs of the host daemon are located at {{ic|%ProgramData%\Looking Glass (host)\looking-glass-host.txt}} on the guest system.<br />
<br />
==== Setting up the null video device ====<br />
<br />
(Retrieved from: https://looking-glass.io/docs/stable/install/#spice-server)<br />
<br />
If you would like to use Spice to give you keyboard and mouse input along with clipboard sync support, make sure you have a {{ic|1=<graphics type='spice'>}} device, then:<br />
<br />
* Find your {{ic|<video>}} device, and set {{ic|1=<model type='none'/>}}<br />
* If you cannot find it, make sure you have a {{ic|<graphics>}} device, save and edit again<br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you do not want to use Spice to control the guest mouse and keyboard you can disable the Spice server.<br />
<br />
$ looking-glass-client -s<br />
<br />
Additionally you may want to start Looking Glass Client as full screen, otherwise the image may be scaled down resulting in poor image fidelity.<br />
<br />
$ looking-glass-client -F<br />
<br />
Launch with the {{ic|--help}} option for further information.<br />
<br />
==== Additional information ====<br />
<br />
Refer to the [https://looking-glass.io/wiki/Installation upstream documentation] for further details.<br />
<br />
=== Swap peripherals to and from the Host ===<br />
<br />
Looking Glass includes a Spice client in order to control mouse movement on the Windows guest. However this may have too much latency for certain applications, such as gaming. An alternative method is passing through specific USB devices for minimal latency. This allows for switching the devices between host and guest.<br />
<br />
First create a .xml file for the device(s) you wish to pass-through, which libvirt will use to identify the device.<br />
<br />
{{hc|~/.VFIOinput/input_1.xml|2=<br />
<hostdev mode='subsystem' type='usb' managed='no'><br />
<source><br />
<vendor id='0x[Before Colon]'/><br />
<product id='0x[After Colon]'/><br />
</source><br />
</hostdev><br />
}}<br />
<br />
Replace [Before/After Colon] with the contents of the 'lsusb' command, specific to the device you want to pass-through.<br />
<br />
For instance my mouse is {{ic|Bus 005 Device 002: ID 1532:0037 Razer USA, Ltd}} so I would replace {{ic|vendor id}} with 1532, and {{ic|product id}} with 1037.<br />
<br />
Repeat this process for any additional USB devices you want to pass-through. If your mouse / keyboard has multiple entries in {{ic|lsusb}}, perhaps if it is wireless, then create additional xml files for each.<br />
<br />
{{Note|Do not forget to change the path & name of the script(s) above and below to match your user and specific system.}}<br />
<br />
Next a bash script file is needed to tell libvirt what to attach/detach the USB devices to the guest.<br />
<br />
{{hc|~/.VFIOinput/input_attach.sh|2=<br />
#!/bin/bash<br />
<br />
virsh attach-device [VM-Name] [USBdevice]<br />
}}<br />
<br />
Replace [VM-Name] with the name of your virtual machine, which can be seen under virt-manager. Additionally replace [USBdevice] with the '''full''' path to the .xml file for the device you wish to pass-through. Add additional lines for more than 1 device. For example here is my script:<br />
<br />
{{hc|~/.VFIOinput/input_attach.sh|2=<br />
#!/bin/bash<br />
<br />
virsh attach-device win10 /home/$USER/.VFIOinput/input_mouse.xml<br />
virsh attach-device win10 /home/$USER/.VFIOinput/input_keyboard.xml<br />
}}<br />
<br />
Next duplicate the script file and replace {{ic|attach-device}} with {{ic|detach-device}}. Ensure both scripts are executable with {{ic|chmod +x $script.sh}}<br />
<br />
This 2 script files can now be executed to attach or detach your USB devices from the host to the guest VM. It is important to note that they may need to be executed as root. To run the script from the Windows VM, one possibility is using [[PuTTY]] to [[SSH]] into the host, and execute the script. On Windows PuTTY comes with plink.exe which can execute singular commands over SSH before then logging out, instead of opening a SSH terminal, all in the background.<br />
<br />
{{hc|detach_devices.bat|2=<br />
"C:\Program Files\PuTTY\plink.exe" root@$HOST_IP -pw $ROOTPASSWORD /home/$USER/.VFIOinput/input_detach.sh<br />
}}<br />
<br />
Replace {{ic|$HOST_IP}} with the Host [[Network configuration#IP addresses|IP Address]] and $ROOTPASSWORD with the root password.<br />
<br />
{{warning|This method is insecure if somebody has access to your VM, since they could open the file and read your password. It is advisable to use [[SSH keys]] instead!}}<br />
<br />
You may also want to execute the script files using key binds. On Windows one option is [https://autohotkey.com/ Autohotkey], and on the Host [[Xbindkeys]]. Because of the need to run the scripts as root, you may also need to use [[Polkit]] or [[Sudo]] which can both be used to authenticate specific executables as able to run as root without needing a password.<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{pkg|linux-zen}} or {{AUR|linux-vfio}} package.<br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream,multifunction}} should break up as many devices as possible.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|edk2-ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/edk2-ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2-ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|<br />
* Make sure that {{ic|1=OVMF_CODE.fd}} is given as a command line parameter before {{ic|1=MY_VARS.fd}}. The boot sequence will fail otherwise.<br />
* QEMU's default SeaBIOS can be used instead of OVMF, but it is not recommended as it can cause issues with passthrough setups.<br />
}}<br />
<br />
It is recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing through other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting: [[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|1=$ for usb_ctrl in /sys/bus/pci/devices/*/usb*; do pci_path=${usb_ctrl%/*}; iommu_group=$(readlink $pci_path/iommu_group); echo "Bus $(cat $usb_ctrl/busnum) --> ${pci_path##*/} (IOMMU group ${iommu_group##*/})"; lsusb -s ${usb_ctrl#*/usb}:; echo; done|2=<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
{{Note|If your USB controller does not support resetting, is not in an isolated group, or is otherwise unable to be passed through then it may still be possible to accomplish similar results through [[udev]] rules. See [https://github.com/olavmrk/usb-libvirt-hotplug] which allows any device connected to specified USB ports to be automatically attached to a virtual machine.}}<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
An emulated audio setup consists of two components: An emulated sound device exposed to the guest and an audio backend connecting the sound device to the host's PulseAudio.<br />
<br />
Of the emulated sound devices available, two are of main interest: ICH9 and usb-audio. ICH9 features both output and input but is limited to stereo. usb-audio only features audio output but supports up to 6 channels in 5.1 configuration. For ICH9 remove any pre-existing audio backend in the {{ic|<devices>}} section and add:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<sound model='ich9'><br />
<codec type='micro'/><br />
<audio id='1'/><br />
</sound><br />
<audio id='1' type='pulseaudio' serverName='/run/user/1000/pulse/native'/><br />
}}<br />
<br />
Note the matching {{ic|id}} elements. The example above assumes a single-user system with user ID 1000. Use the {{ic|id}} command to find the correct ID. You can also use the {{ic|/tmp}} directory if you have multiple users accessing PulseAudio:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<audio id='1' type='pulseaudio' serverName='unix:/tmp/pulse-socket'/><br />
}}<br />
<br />
If you get crackling or distorted sound, try experimenting with some latency settings. The following example uses 20000 microseconds:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<audio id="1" type="pulseaudio" serverName="/run/user/1000/pulse/native"><br />
<input latency="20000"/><br />
<output latency="20000"/><br />
</audio><br />
}}<br />
<br />
You can also try disabling the software mixer included in QEMU. This should, in theory, be more efficient and allow for lower latencies since mixing will then take place on your host only:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<audio id="1" type="pulseaudio" serverName="/run/user/1000/pulse/native"><br />
<input mixingEngine="no"/><br />
<output mixingEngine="no"/><br />
</audio><br />
}}<br />
<br />
For usb-audio, the corresponding elements read<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<sound model='usb'><br />
<audio id='1'/><br />
</sound><br />
<audio id='1' type='pulseaudio' serverName='/run/user/1000/pulse/native'/><br />
}}<br />
<br />
However, if a 5.1 configuration is required the sound device needs to be configured via QEMU command line arguments:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
</devices><br />
<qemu:commandline><br />
<qemu:arg value='-device'/><br />
<qemu:arg value='usb-audio,id=sound0,audiodev=audio1,multi=on'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
The {{ic|audiodev}} tag has to be set to match the audio backend's {{ic|id}} element. {{ic|1=id='1'|2==}} corresponds to {{ic|audio1}} and so on. See [[#Libvirt versions below 7.3]] for further details how to directly pass QEMU arguments.<br />
<br />
{{Note|1=<nowiki/><br />
* You can have multiple audio backends, by simply specifying {{ic|<audio>}}/{{ic|-audiodev}} multiple times in your XML and by assigning them different ids. This can be useful for a use case of having two identical backends. With PulseAudio each backend is a separate stream and can be routed to different output devices on the host (using a pulse mixer app like {{Pkg|pavucontrol}} or {{Pkg|pulsemixer}}).<br />
* USB 3 emulation is needed in Libvirt/QEMU to enable the usb-audio.<br />
* It is recommended to enable MSI interrupts with a tool such as [https://forums.guru3d.com/threads/windows-line-based-vs-message-signaled-based-interrupts-msi-tool.378044/] on the ICH9 audio device to mitigate any crackling, stuttering, speedup, or no audio at all after VM restart.<br />
* If audio is crackling/stuttering/speedup etc. is still present you may want to adjust parameters such as {{ic|buffer-length}} and {{ic|timer-period}}, more information on these parameters and more can be found in the {{man|1|qemu}} manual.<br />
* Some audio chipsets such as [https://bugzilla.kernel.org/show_bug.cgi?id=195303 Realtek alc1220] may also have issues out of the box so do consider this when using any audio emulation with QEMU.<br />
* Improper pinning or heavy host usage without using [[#Isolating pinned CPUs|isolcpus]] can also influence sound bugs, especially while gaming in a VM.<br />
}}<br />
<br />
=== Passing VM audio to host via JACK and PipeWire ===<br />
<br />
It is also possible to pass VM audio to the host via JACK and PipeWire.<br />
<br />
First, make sure you have a working [[PipeWire]] setup with [[PipeWire#JACK_clients|JACK support]]. In addition to that, you need to install {{AUR|pipewire-jack-dropin}} as it will allow QEMU or libvirtd to easily access PipeWire's JACK sink. <br />
<br />
Next, you'll need to tell libvirt to run QEMU as your user:<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
As a final preparation, the XML scheme has to be extended to allow passing of environment variables. For this, modify the VM domain configuration<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
<domain type='kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
<domain type='kvm' xmlns:qemu='<nowiki>http://libvirt.org/schemas/domain/qemu/1.0</nowiki>'><br />
}}<br />
<br />
Then, you can add the actual audio config to your VM:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<devices><br />
...<br />
<audio id="1" type="jack"><br />
<input clientName="vm-win10" connectPorts="your-input"/><br />
<output clientName="vm-win10" connectPorts="your-output"/><br />
</audio><br />
</devices><br />
<qemu:commandline><br />
<qemu:env name="PIPEWIRE_RUNTIME_DIR" value="/run/user/1000"/><br />
<qemu:env name="PIPEWIRE_LATENCY" value="512/48000"/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
{{Note|Use a tool like {{Pkg|carla}} to figure out which input and outputs you want.}}<br />
<br />
Note the matching {{ic|id}} elements. Above's example assumes a single-user system with user ID 1000. Use the {{ic|id}} command to find the correct ID.<br />
<br />
You might have to play with the {{ic|PIPEWIRE_LATENCY}} values to get to the desired latency without crackling.<br />
<br />
=== Passing VM audio to host via Scream ===<br />
<br />
It is possible to pass VM audio through a bridged network such as the one provided by Libvirt or by adding a IVSHMEM device to the host by using a application called [https://github.com/duncanthrax/scream Scream]. This section will only cover using PulseAudio as a receiver on the host.<br />
See the project page for more details and instructions on other methods.<br />
<br />
==== Using Scream with a bridged network ====<br />
<br />
{{Note|<br />
* This is the ''preferred'' way to use this, although results may vary per user<br />
* It is recommend to use the [[#Virtio network]] adapter while using Scream, other virtual adapters provided by QEMU such as '''e1000e''' may lead to poor performance<br />
}}<br />
<br />
To use scream via your network you will want to find your bridge name via {{ic|1=ip a}}, in most cases it will be called '''br0''' or '''virbr0'''. Below is a example of the command needed to start the Scream application:<br />
<br />
$ scream -o pulse -i virbr0 &<br />
<br />
{{Warning| This will not work with a '''macvtap bridge''' as that does not allow host to guest communication, also make sure you have the proper firewall ports open for it to communicate with the VM}}<br />
<br />
==== Adding the IVSHMEM device to use Scream with IVSHMEM ====<br />
<br />
With the VM turned off, edit the machine configuration<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<devices><br />
...<br />
<shmem name='scream-ivshmem'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>2</size><br />
</shmem><br />
</devices><br />
...<br />
}}<br />
<br />
In the above configuration, the size of the IVSHMEM device is 2MB (the recommended amount). Change this as needed.<br />
<br />
Now refer to [[#Adding IVSHMEM Device to VM]] to configure the host to create the shared memory file on boot, replacing {{ic|looking-glass}} with {{ic|scream-ivshmem}}.<br />
<br />
===== Configuring the Windows guest for IVSHMEM =====<br />
<br />
The correct driver must be installed for the IVSHMEM device on the guest. <br />
See [[#Installing the IVSHMEM Host to Windows guest]]. Ignore the part about {{ic|looking-glass-host}}.<br />
<br />
Install the [https://github.com/duncanthrax/scream/releases Scream] virtual audio driver on the guest. <br />
If you have secure boot enabled for your VM, you may need to disable it. <br />
<br />
Using the registry editor, set the DWORD {{ic|HKLM\SYSTEM\CurrentControlSet\Services\Scream\Options\UseIVSHMEM}} to the size of the IVSHMEM device in MB.<br />
Note that scream identifies its IVSHMEM device using its size, so make sure there is only one device of that size.<br />
<br />
====== Configuring the host ======<br />
<br />
Install {{AUR|scream}}.<br />
<br />
Create a [[systemd/User|systemd user service]] to control the receiver:<br />
<br />
{{hc|~/.config/systemd/user/scream-ivshmem-pulse.service|2=<br />
[Unit]<br />
Description=Scream IVSHMEM pulse receiver<br />
After=pulseaudio.service<br />
Wants=pulseaudio.service<br />
<br />
[Service]<br />
Type=simple<br />
ExecStartPre=/usr/bin/truncate -s 0 /dev/shm/scream-ivshmem<br />
ExecStartPre=/usr/bin/dd if=/dev/zero of=/dev/shm/scream-ivshmem bs=1M count=2<br />
ExecStart=/usr/bin/scream -m /dev/shm/scream-ivshmem<br />
<br />
[Install]<br />
WantedBy=default.target<br />
}}<br />
<br />
Edit {{ic|1=count=2}} with the size of the IVSHMEM device in MiB.<br />
<br />
{{Tip|If you are using [[PipeWire]], replace {{ic|pulseaudio.service}} with {{ic|pipewire-pulse.service}}.}}<br />
<br />
Now start the service with:<br />
<br />
$ systemctl start --user scream-ivshmem-pulse<br />
<br />
To have it automatically start on next login, enable the service:<br />
<br />
$ systemctl enable --user scream-ivshmem-pulse<br />
<br />
=== Physical disk/partition ===<br />
<br />
Raw and qcow2 especially can have noticeable overhead for heavy IO. A whole disk or a partition may be used directly to bypass the filesystem and improve I/O performance. If you wish to dual boot the guest OS natively you would need to pass the entire disk without any partitioning. It is suggested to use /dev/disk/by- paths to refer to the disk since /dev/sdX entries can change between boots. To find out which disk/partition is associated with the one you would like to pass:<br />
<br />
{{hc|$ ls -l /dev/disk/by-id|<br />
ata-ST1000LM002-9VQ14L_Z0501SZ9 -> ../../sdd<br />
}}<br />
<br />
See [[#Virtio disk]] on how to add these with libvirt XML. You can also add the disk with Virt-Manager's '''Add Hardware''' menu and then type the disk you want in the '''Select or create custom storage''' box, e.g. '''/dev/disk/by-id/ata-ST1000LM002-9VQ14L_Z0501SZ9'''<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functional and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore recommended to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
For many reasons users may seek to see [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]].<br />
<br />
These examples offer a supplement to existing hardware compatibility lists. Additionally, if you have trouble configuring a certain mechanism in your setup, you might find these examples very valuable. Users there have described their setups in detail, and some have provided examples of their configuration files as well. <br />
<br />
We encourage those who successfully build their system from this resource to help improve it by contributing their builds. Due to the many different hardware manufacturers involved, the seemingly significant lack of sufficient documentation, as well as other issues due to the nature of this process, community contributions are necessary.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== QEMU 4.0: Unable to load graphics drivers/BSOD/Graphics stutter after driver install using Q35 ===<br />
<br />
Starting with QEMU 4.0, the Q35 machine type changes the default {{ic|kernel_irqchip}} from {{ic|off}} to {{ic|split}} which breaks some guest devices, such as nVidia graphics (the driver fails to load / black screen / code 43 / graphics stutters, usually when mouse moving). Switch to full KVM mode instead by adding {{ic|1=<ioapic driver='kvm'/>}} under libvirt's {{ic|<features>}} tag in your VM configuration or by adding {{ic|1=kernel_irqchip=on}} in the {{ic|-machine}} QEMU arg.<br />
<br />
=== QEMU 5.0: host-passthrough with kernel version 5.5 to 5.8.1 when using Zen 2 processors: Windows 10 BSOD loop 'KERNEL SECURITY CHECK FAILURE' ===<br />
<br />
{{Note|As of kernel version 5.8.2, disabling STIBP is not required anymore.}}<br />
<br />
Starting with QEMU 5.0 virtual machines running on Zen 2 and newer kernels than 5.4 will cause a BSOD loop of: 'KERNEL SECURITY CHECK FAILURE'. This can be fixed by either updating to kernel version 5.8.2 or higher, or disabling STIBP:<br />
<cpu mode='host-passthrough' ...><br />
...<br />
<feature policy='disable' name='amd-stibp'/><br />
...<br />
</cpu><br />
This requires libvirt 6.5 or higher. On older versions, several workarounds exist:<br />
* Switch CPU mode from {{ic|host-passthrough}} to {{ic|host-model}}. This only works on libvirt 6.4 or lower.<br />
* Manually patch {{Pkg|qemu}} in order to revert [https://github.com/qemu/qemu/commit/143c30d4d346831a09e59e9af45afdca0331e819 this] commit.<br />
* On qemu commandline, add {{ic|1=amd-stibp=off}} to the cpu flags string. This can also be invoked through libvirt via a {{ic|<qemu:commandline>}} entry.<br />
<br />
=== "Error 43: Driver failed to load" with mobile (Optimus/max-q) nvidia GPUs ===<br />
<br />
This error occurs because the Nvidia driver wants to check the status of the power supply. If no battery is present, the driver does not work. Whether Libvirt or Quemu, by default none of them provide the possibility to simulate a battery. This might also result in a reduced screen resolution and the Nvidia Desktop Manager refusing to load when right-clicking the desktop, saying it requires Windows 10, a compatible GPU and the Nvidia graphics driver.<br />
<br />
You can however create and add a custom acpi table file to the virtual machine which will do the work.<br />
<br />
First you have to create the custom acpi table file by pasting the following base64 string [https://base64.guru/converter/decode/file here] and save the result file as SSDT1.dat:<br />
{{bc|1=<br />
U1NEVKEAAAAB9EJPQ0hTAEJYUENTU0RUAQAAAElOVEwYEBkgoA8AFVwuX1NCX1BDSTAGABBMBi5f<br />
U0JfUENJMFuCTwVCQVQwCF9ISUQMQdAMCghfVUlEABQJX1NUQQCkCh8UK19CSUYApBIjDQELcBcL<br />
cBcBC9A5C1gCCywBCjwKPA0ADQANTElPTgANABQSX0JTVACkEgoEAAALcBcL0Dk=<br />
}}<br />
<br />
Next you must add the processed file to the main domain of the virtual machine:<br />
{{bc|<nowiki><br />
<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm"><br />
...<br />
<qemu:commandline><br />
<qemu:arg value="-acpitable"/><br />
<qemu:arg value="file=/path/to/your/SSDT1.dat"/><br />
</qemu:commandline><br />
</domain><br />
</nowiki>}}<br />
<br />
Make sure your XML file has the correct namespace in the {{ic|<domain>}} tag as visible above, otherwise the XML verification will fail.<br />
<br />
[https://www.reddit.com/r/VFIO/comments/ebo2uk/nvidia_geforce_rtx_2060_mobile_success_qemu_ovmf/ Source]<br />
<br />
=== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ===<br />
<br />
{{Expansion|This error is actually related to the boot_vgs issue and should be merged together with everything else concerning GPU ROMs.|section=UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check ''dmesg'' for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
In addition try adding kernel parameter {{ic|1=pci=realloc}} which also [https://github.com/Dunedan/mbp-2016-linux/issues/60#issuecomment-396311301 helps with hotplugging issues].<br />
<br />
=== UEFI (OVMF) compatibility in VBIOS ===<br />
<br />
{{Remove|Flashing you guest GPU for the purpose of a GPU passthrough is '''never''' good advice. A full section should be dedicated to VBIOS compatibility.|section= UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it is not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it is stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it is pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
On a linux guest you can use modinfo to see if there is option to enable MSI (for example: "modinfo snd_hda_intel |grep msi"). If there is, one can enable it by adding the relevant option to a custom omdprobe file - in "/etc/modprobe.d/snd-hda-intel.conf" inserting "options snd-hda-intel enable_msi=1"<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
A UI tool called [https://github.com/CHEF-KOCH/MSI-utility MSI Utility (FOSS Version 2)]{{Dead link|2021|05|17|status=404}} works with Windows 10 64-bit and simplifies the process.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read [https://www.kernel.org/doc/html/latest/x86/intel-iommu.html#graphics-problems intel-iommu.html] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X does not start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== Host lockup after VM shutdown ===<br />
<br />
This issue seems to primarily affect users running a Windows 10 guest and usually after the VM has been run for a prolonged period of time: the host will experience multiple CPU core lockups (see [https://bbs.archlinux.org/viewtopic.php?id=206050&p=2]). To fix this try enabling Message Signal Interrupts on the GPU passed through to the guest. A good guide for how to do this can be found in [https://forums.guru3d.com/threads/windows-line-based-vs-message-signaled-based-interrupts.378044/]. You can also download this application for windows here [https://github.com/TechtonicSoftware/MSIInturruptEnabler] that should make the process easier.<br />
<br />
=== Host lockup if guest is left running during sleep ===<br />
<br />
VFIO-enabled virtual machines tend to become unstable if left running through a sleep/wakeup cycle and have been known to cause the host machine to lockup when an attempt is then made to shut them down. In order to avoid this, one can simply prevent the host from going into sleep while the guest is running using the following libvirt hook script and systemd unit. The hook file needs executable permissions to work.<br />
<br />
{{hc|/etc/libvirt/hooks/qemu|2=<br />
#!/bin/bash<br />
<br />
OBJECT="$1"<br />
OPERATION="$2"<br />
SUBOPERATION="$3"<br />
EXTRA_ARG="$4"<br />
<br />
case "$OPERATION" in<br />
"prepare")<br />
systemctl start libvirt-nosleep@"$OBJECT"<br />
;;<br />
"release")<br />
systemctl stop libvirt-nosleep@"$OBJECT"<br />
;;<br />
esac<br />
}}<br />
<br />
{{hc|/etc/systemd/system/libvirt-nosleep@.service|2=<br />
[Unit]<br />
Description=Preventing sleep while libvirt domain "%i" is running<br />
<br />
[Service]<br />
Type=simple<br />
ExecStart=/usr/bin/systemd-inhibit --what=sleep --why="Libvirt domain \"%i\" is running" --who=%U --mode=block sleep infinity<br />
}}<br />
<br />
=== Cannot boot after upgrading ovmf ===<br />
<br />
If you cannot boot after upgrading from {{Pkg|ovmf}}{{Broken package link|replaced by {{Pkg|edk2-ovmf}}}} version 1:r23112.018432f0ce-1 then you need to remove the old {{ic|*VARS.fd}} file in {{ic|/var/lib/libvirt/qemu/nvram/}}:<br />
<br />
# mv /var/lib/libvirt/qemu/nvram/vmname_VARS.fd /var/lib/libvirt/qemu/nvram/vmname_VARS.fd.old<br />
<br />
See {{Bug|57825}} for further details.<br />
<br />
=== QEMU via cli pulseaudio stuttering/delay ===<br />
<br />
Using following flags for the audio device and chipset might help if you are running into the stuttering/delay audio issues when running QEMU via cli:<br />
<br />
qemu-system-x86_64 \<br />
-machine pc-i440fx-3.0 \<br />
-device hda-micro \<br />
-soundhw hda \<br />
-...<br />
<br />
As noted in [[#QEMU 3.0 audio changes|QEMU 3.0 audio changes]]{{Broken section link}} the specified chipset will include a series of audio patches.<br />
<br />
Setting {{ic|QEMU_AUDIO_TIMER_PERIOD}} to values higher than 100 might also help (did not test value lower than 100).<br />
<br />
=== Bluescreen at boot since Windows 10 1803 ===<br />
<br />
Since Windows 10 1803 there is a problem when you are using "host-passthrough" as cpu model. The machine cannot boot and is either boot looping or you get a bluescreen.<br />
You can workaround this by:<br />
<br />
# echo 1 > /sys/module/kvm/parameters/ignore_msrs<br />
<br />
To make it permanently you can create a modprobe file {{ic|kvm.conf}}:<br />
<br />
options kvm ignore_msrs=1<br />
<br />
To prevent clogging up ''dmesg'' with "ignored rdmsr" messages you can additionally add:<br />
<br />
options kvm report_ignored_msrs=0<br />
<br />
=== AMD Ryzen / BIOS updates (AGESA) yields "Error: internal error: Unknown PCI header type ‘127’" ===<br />
<br />
AMD users have been experiencing breakage of their KVM setups after updating the BIOS on their motherboard. There is a kernel [https://clbin.com/VCiYJ patch], (see [[Kernel/Arch Build System]] for instruction on compiling kernels with custom patches) that can resolve the issue as of now (7/28/19), but this is not the first time AMD has made an error of this very nature, so take this into account if you are considering updating your BIOS in the future as a VFIO user.<br />
<br />
=== AMD GPU not resetting properly yielding "Error: internal error: Unknown PCI header type ‘127’" (Separate issue from the one above) ===<br />
<br />
Passing through an AMD GPU may result into a problem known as the "AMD reset bug". Upon power cycling the guest, the GPU does not properly reset its state which causes the device to malfunction until the host is also rebooted. This is usually paired with a "code 43" driver error in a Windows guest, and the message "Error: internal error: Unknown PCI header type '127'" in the libvirt log on the host.<br />
<br />
In the past, this meant having to use work-arounds to manually reset the GPU, or resorting to the use of kernel patches that were unlikely to land in upstream. Currently, the recommended solution that does not require patching of the kernel is to install {{AUR|vendor-reset-git}} or {{AUR|vendor-reset-dkms-git}} and making sure the 'vendor-reset' kernel module is loaded before booting the guest.<br />
<br />
{{Note| Make sure you do not have any of the AMD reset bug kernel patches installed if you are using {{AUR|vendor-reset-git}} or {{AUR|vendor-reset-dkms-git}}.}}<br />
<br />
=== Host crashes when hotplugging Nvidia card with USB ===<br />
<br />
If attempting to hotplug an Nvidia card with a USB port, you may have to blacklist the {{ic|i2c_nvidia_gpu}} driver. Do this by adding the line {{ic|blacklist i2c_nvidia_gpu}} to {{ic|/etc/modprobe.d/blacklist.conf}}.<br />
<br />
=== Host unable to boot and stuck in black screen after enabling vfio ===<br />
<br />
If debug kernel messages during boot are enabled by adding with the {{ic|debug ignore_loglevel}} [[kernel parameters]], you may see boot stuck with the last message similar to<br />
<br />
vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none<br />
<br />
This can be resolved by disconnecting the passthroughed GPU with your moniter. You may reconnect the passthroughed GPU to a moniter after host is booted.<br />
<br />
=== AER errors when passing through PCIe USB hub ===<br />
<br />
In some cases passing through a PCIe USB hub, such as one connected to the guest GPU, might fail with AER errors similar to the following:<br />
<br />
kernel: pcieport 0000:00:01.1: AER: Uncorrected (Non-Fatal) error received: 0000:00:01.1<br />
kernel: pcieport 0000:00:01.1: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)<br />
kernel: pcieport 0000:00:01.1: AER: device [8086:1905] error status/mask=00100000/00000000<br />
kernel: pcieport 0000:00:01.1: AER: [20] UnsupReq (First)<br />
kernel: pcieport 0000:00:01.1: AER: TLP Header: 00000000 00000000 00000000 00000000<br />
kernel: pcieport 0000:00:01.1: AER: device recovery successful<br />
<br />
=== Reserved Memory Region Reporting (RMRR) Conflict ===<br />
<br />
If you run into an issue passing through a device because of the BIOS's usage of RMRR, like the error below.<br />
<br />
vfio-pci 0000:01:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.<br />
<br />
You can try the patches here: https://github.com/kiler129/relax-intel-rmrr<br />
<br />
== See also ==<br />
<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://www.reddit.com/r/VFIO /r/VFIO: A subreddit focused on vfio]<br />
* [https://github.com/intel/gvt-linux/wiki/GVTd_Setup_Guide GVT-d: passthrough of an entire integrated GPU]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=689652PCI passthrough via OVMF2021-07-28T18:52:41Z<p>NeoTheFox: /* Passing VM audio to host via JACK and PipeWire */ pipewire-jack is in the repos now</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
[[zh-hans:PCI passthrough via OVMF]]<br />
{{Related articles start}}<br />
{{Related|Intel GVT-g}}<br />
{{Related|PCI passthrough via OVMF/Examples}}<br />
{{Related articles end}}<br />
<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of [[QEMU]], it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)].<br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=873&0_VTD=True List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Note|<br />
* IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
* VT-d stands for ''Intel Virtualization Technology for Directed I/O'' and should not be confused with VT-x ''Intel Virtualization Technology''. VT-x allows one hardware platform to function as multiple “virtual” platforms while VT-d improves security and reliability of the systems and also improves performance of I/O devices in virtualized environments.<br />
}}<br />
<br />
Using IOMMU opens to features like PCI passthrough and memory protection from faulty or malicious devices, see [[Wikipedia:Input-output memory management unit#Advantages]] and [https://www.quora.com/Memory-Management-computer-programming/Could-you-explain-IOMMU-in-plain-English Memory Management (computer programming): Could you explain IOMMU in plain English?].<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is supported by the CPU and enabled in the BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
Manually enable IOMMU support by setting the correct [[kernel parameter]] depending on the type of CPU in use:<br />
<br />
* For Intel CPUs (VT-d) set {{ic|1=intel_iommu=on}}. Since the kernel config option CONFIG_INTEL_IOMMU_DEFAULT_ON is not set in {{Pkg|linux}}.<br />
* For AMD CPUs (AMD-Vi), it's on if kernel detects IOMMU HW support from BIOS.<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check [[dmesg]] to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|# dmesg {{!}} grep -i -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do<br />
echo "IOMMU Group ${g##*/}:"<br />
for d in $g/devices/*; do<br />
echo -e "\t$(lspci -nns ${d##*/})"<br />
done;<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 1:<br />
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
IOMMU Group 2:<br />
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:0e31] (rev 04)<br />
IOMMU Group 4:<br />
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:0e2d] (rev 04)<br />
IOMMU Group 10:<br />
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:0e26] (rev 04)<br />
IOMMU Group 13:<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1:<br />
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in to your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the preferred method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{Warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this - your motherboard should be set to display using the host GPU.}}<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub did, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use.<br />
<br />
=== Binding vfio-pci via device ID ===<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13:<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|<br />
* You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.<br />
* If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.<br />
* Binding the audio device ({{ic|10de:0fbb}} in above's example) is optional. Libvirt is able to unbind it from the {{ic|snd_hda_intel}} driver on its own.<br />
}}<br />
<br />
Two methods exist for providing the device IDs. Specifying them via [[kernel parameters]] has the advantage of being able to easily edit, remove, or undo any breaking changes via your boot loader:<br />
<br />
vfio-pci.ids=10de:13c2,10de:0fbb<br />
<br />
Alternatively, the IDs may be added to a modprobe conf file. Since these conf files are embedded in the initramfs image, any changes require regenerating a new image each time:<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
=== Loading vfio-pci early ===<br />
<br />
==== mkinitcpio ====<br />
<br />
Since Arch's {{Pkg|linux}} has vfio-pci built as a module, we need to force it to load early before the graphics drivers have a chance to bind to the card. To ensure that, add {{ic|vfio_pci}}, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio_pci vfio vfio_iommu_type1 vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]].<br />
<br />
==== booster ====<br />
<br />
Similar to mkinitcpio you need to specify modules to load early:<br />
{{hc|/etc/booster.yaml|2=<br />
modules_force_load: vfio_pci,vfio,vfio_iommu_type1,vfio_virqfd<br />
}}<br />
<br />
and then [[Booster#Regenerate_booster_images|regenerate the initramfs]].<br />
<br />
==== dracut ====<br />
<br />
dracut's early loading mechanism is configured via kernel parameters. To load vfio-pci early, add both the [[#Binding vfio-pci via device ID|device ids]] and the following line to your [[kernel parameters]]:<br />
<br />
rd.driver.pre=vfio_pci<br />
<br />
We also need to add all the vfio drivers to the initramfs. Add the following file to {{ic|/etc/dracut.conf.d}}:<br />
<br />
{{hc|10-vfio.conf|2=<br />
add_drivers+=" vfio_pci vfio vfio_iommu_type1 vfio_virqfd "<br />
}}<br />
<br />
As with mkinitcpio, you must regenerate the initramfs. See [[dracut]] for more details.<br />
<br />
=== Verifying that the configuration worked ===<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|# dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in ''dmesg'' output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it is possible to use SeaBIOS to get similar results to an actual PCI passthrough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
{{Accuracy|{{Pkg|libvirt}} depends on ''ebtables'', not {{pkg|iptables-nft}} which is just a translation layer.}}<br />
<br />
Install {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|edk2-ovmf}}, and {{Pkg|virt-manager}}. For the default network connection, {{pkg|iptables-nft}} and {{pkg|dnsmasq}} are required.<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
You may also need to [https://wiki.libvirt.org/page/Networking#NAT_forwarding_.28aka_.22virtual_networks.22.29 activate the default libvirt network]:<br />
# virsh net-autostart default<br />
# virsh net-start default<br />
<br />
{{Note|The default libvirt network will only be listed if the virsh command is run as root.}}<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self-explanatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
However, you should pay special attention to the following steps:<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that:<br />
** Your hypervisor is running as a system session and not a user session. This can be verified [https://i.ibb.co/N1XZCdp/Deepin-Screenshot-select-area-20190125113216.png by clicking, then hovering] over the session in virt-manager. If you are accidentally running it as a user session, you must open a new connection by clicking "File" > "Add Connection..", then select the option from the drop-down menu station "QEMU/KVM" and not "QEMU/KVM user session".<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to either type it by hand or by using {{ic|virt-xml ''vmname'' --edit --cpu host-passthrough}}. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, it's easier to setup [[#Virtio disk]] before installing<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine. It is possible you will be dropped into the UEFI menu instead of starting the installation upon powering your VM for the first time. Sometimes the correct ISO file was not automatically detected and you will need to manually specify the drive to boot. By typing exit and navigating to "boot manager" you will enter a menu that allows you to choose between devices.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it is now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. For example, remove the following sections from your XML file:<br />
<br />
{{bc|1=<nowiki/><br />
<channel type="spicevmc"><br />
...<br />
</channel><br />
<input type="tablet" bus="usb"><br />
...<br />
</input><br />
<input type="mouse" bus="ps2"/><br />
<input type="keyboard" bus="ps2"/><br />
<graphics type="spice" autoport="yes"><br />
...<br />
</graphics><br />
<video><br />
<model type="qxl" .../><br />
...<br />
</video><br />
}}<br />
<br />
Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. This may be done by using {{ic|Add Hardware > USB Host Device}}.<br />
<br />
At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Video card driver virtualisation detection ===<br />
<br />
Video card drivers by AMD incorporate very basic virtual machine detection targeting Hyper-V extensions. Should this detection mechanism trigger the drivers will refuse to run (resulting in a black screen).<br />
<br />
It is therefore required to modify the reported Hyper-V vendor ID:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<features><br />
...<br />
<hyperv><br />
...<br />
<vendor_id state='on' value='randomid'/><br />
...<br />
</hyperv><br />
...<br />
</features><br />
...<br />
}}<br />
<br />
Nvidia guest drivers prior to version 465 exhibited a similar behaviour which resulted in a generic error 43 in the card's device manager status. Systems using these older drivers therefore also need the above modification. In addition, they also require hiding the KVM CPU leaf:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<features><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
</features><br />
...<br />
}}<br />
<br />
Note that the above steps do not equate 'hiding' the virtual machine from Windows or any drivers/programs running in the VM. Also, various other issues not related to any detection mechanism referred to here can also trigger error 43.<br />
<br />
=== Passing keyboard/mouse via Evdev ===<br />
<br />
If you do not have a spare mouse or keyboard to dedicate to your guest, and you do not want to suffer from the video overhead of Spice, you can setup evdev to share them between your Linux host and your virtual machine.<br />
<br />
{{Note|Press both left and right '''Ctrl''' keys at the same time to swap control between the host and the guest.}}<br />
<br />
First, find your keyboard and mouse devices in {{ic|/dev/input/by-id/}}. Only devices with {{ic|event}} in their name are valid. You may find multiple devices associated to your mouse or keyboard, so try {{ic|cat /dev/input/by-id/''device_id''}} and either hit some keys on the keyboard or wiggle your mouse to see if input comes through, if so you have got the right device. Now add those devices to your configuration:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<devices><br />
...<br />
<input type='evdev'><br />
<source dev='/dev/input/by-id/MOUSE_NAME'/><br />
</input><br />
<input type='evdev'><br />
<source dev='/dev/input/by-id/KEYBOARD_NAME' grab='all' repeat='on'/><br />
</input><br />
...<br />
</devices><br />
}}<br />
<br />
Replace {{ic|MOUSE_NAME}} and {{ic|KEYBOARD_NAME}} with your device path. Now you can startup the guest OS and test swapping control of your mouse and keyboard between the host and guest by pressing both the left and right control keys at the same time.<br />
<br />
You may also consider switching from PS/2 to Virtio inputs in your configurations. Add these two devices:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<input type='mouse' bus='virtio'/><br />
<input type='keyboard' bus='virtio'/><br />
...<br />
}}<br />
<br />
The virtio input devices will not actually be used until the guest drivers are installed. QEMU will continue to send key events to the PS2 devices until it detects the virtio input driver initialization. Note that the PS2 devices cannot be removed as they are an internal function of the emulated Q35/440FX chipsets.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate Linux/Windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. As such, the local CPU cache benefits (L1/L2/L3) are lost each time the host scheduler reschedules the virtual CPU thread on a different physical CPU. This can noticeably harm performance on the guest. CPU pinning aims to resolve this by limiting which physical CPUs the virtual CPUs are allowed to run on. The ideal setup is a one to one mapping such that the virtual CPU cores match physical CPU cores while taking hyperthreading/SMT into account.<br />
<br />
In addition, in some modern CPUs, groups of cores often share a common L3 cache. In such cases, care should be taken to pin exactly those physical cores that share a particular L3. Failing to do so might lead to cache evictions which could result in microstutters.<br />
<br />
{{Note|For certain users enabling CPU pinning may introduce stuttering and short hangs, especially with the MuQSS scheduler (present in linux-ck and linux-zen kernels). You might want to try disabling pinning first if you experience similar issues, which effectively trades maximum performance for responsiveness at all times.}}<br />
<br />
==== CPU topology ====<br />
<br />
Most modern CPUs support hardware multitasking, also known as hyper-threading on Intel CPUs or SMT on AMD CPUs. Hyper-threading/SMT is simply a very efficient way of running two threads on one CPU core at any given time. You will want to take into consideration that the CPU pinning you choose will greatly depend on what you do with your host while your VM is running.<br />
<br />
To find the topology for your CPU run {{ic|1=lscpu -e}}:<br />
<br />
{{Note|Pay special attention to the 4th column '''"CORE"''' as this shows the association of the Physical/Logical CPU cores as well as the 8th column '''"L3"''' which shows which cores are connected to which L3 cache.}}<br />
<br />
{{ic|lscpu -e}} on a 6c/12t Ryzen 5 1600:<br />
<br />
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ<br />
0 0 0 0 0:0:0:0 yes 3800.0000 1550.0000<br />
1 0 0 0 0:0:0:0 yes 3800.0000 1550.0000<br />
2 0 0 1 1:1:1:0 yes 3800.0000 1550.0000<br />
3 0 0 1 1:1:1:0 yes 3800.0000 1550.0000<br />
4 0 0 2 2:2:2:0 yes 3800.0000 1550.0000<br />
5 0 0 2 2:2:2:0 yes 3800.0000 1550.0000<br />
6 0 0 3 3:3:3:1 yes 3800.0000 1550.0000<br />
7 0 0 3 3:3:3:1 yes 3800.0000 1550.0000<br />
8 0 0 4 4:4:4:1 yes 3800.0000 1550.0000<br />
9 0 0 4 4:4:4:1 yes 3800.0000 1550.0000<br />
10 0 0 5 5:5:5:1 yes 3800.0000 1550.0000<br />
11 0 0 5 5:5:5:1 yes 3800.0000 1550.0000<br />
<br />
Considering the L3 mapping, it is recommended to pin and isolate CPUs 6–11. Pinning and isolating fewer than these (e.g. 8–11) would result in the host system making use of the L3 cache in core 6 and 7 which would eventually lead to cache evictions and therefore bad performance.<br />
<br />
{{Note|Ryzen 3000 ComboPi AGESA changes topology to match Intel example, even on prior generation CPUs. Above valid only on older AGESA.}}<br />
<br />
{{ic|lscpu -e}} on a 6c/12t Intel 8700k:<br />
<br />
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ<br />
0 0 0 0 0:0:0:0 yes 4600.0000 800.0000<br />
1 0 0 1 1:1:1:0 yes 4600.0000 800.0000<br />
2 0 0 2 2:2:2:0 yes 4600.0000 800.0000<br />
3 0 0 3 3:3:3:0 yes 4600.0000 800.0000<br />
4 0 0 4 4:4:4:0 yes 4600.0000 800.0000<br />
5 0 0 5 5:5:5:0 yes 4600.0000 800.0000<br />
6 0 0 0 0:0:0:0 yes 4600.0000 800.0000<br />
7 0 0 1 1:1:1:0 yes 4600.0000 800.0000<br />
8 0 0 2 2:2:2:0 yes 4600.0000 800.0000<br />
9 0 0 3 3:3:3:0 yes 4600.0000 800.0000<br />
10 0 0 4 4:4:4:0 yes 4600.0000 800.0000<br />
11 0 0 5 5:5:5:0 yes 4600.0000 800.0000<br />
<br />
Since all cores are connected to the same L3 in this example, it does not matter much how many CPUs you pin and isolate as long as you do it in the proper thread pairs. For instance, (0, 6), (1, 7), etc.<br />
<br />
As we see above, with AMD '''Core 0''' is sequential with '''CPU 0 & 1''', whereas Intel places '''Core 0''' on '''CPU 0 & 6'''.<br />
<br />
{{Tip|You can view your systems topology in diagram form, which may help some users. If you have {{Pkg|hwloc}} installed, run {{ic|lstopo}} to generate a helpful image of your CPU/Thread groupings.}}<br />
<br />
If you do not need all cores for the guest, it would then be preferable to leave at the very least one core for the host. Choosing which cores one to use for the host or guest should be based on the specific hardware characteristics of your CPU, however '''Core 0''' is a good choice for the host in most cases. If any cores are reserved for the host, it is recommended to pin the emulator and iothreads, if used, to the host cores rather than the VCPUs. This may improve performance and reduce latency for the guest since those threads will not pollute the cache or contend for scheduling with the guest VCPU threads. If all cores are passed to the guest, there is no need or benefit to pinning the emulator or iothreads.<br />
<br />
==== XML examples ====<br />
<br />
{{Note|Do not use the '''iothread''' lines from the XML examples shown below if you have not added an '''iothread''' to your disk controller. '''iothread''''s only work on '''virtio-scsi''' or '''virtio-blk''' devices.}}<br />
<br />
===== 4c/1t CPU w/o Hyperthreading Example =====<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='0'/><br />
<vcpupin vcpu='1' cpuset='1'/><br />
<vcpupin vcpu='2' cpuset='2'/><br />
<vcpupin vcpu='3' cpuset='3'/><br />
</cputune><br />
...<br />
}}<br />
<br />
===== 4c/2t Intel/AMD CPU example (after ComboPI AGESA bios update) =====<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<vcpu placement='static'>8</vcpu><br />
<iothreads>1</iothreads><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='8'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='9'/><br />
<vcpupin vcpu='4' cpuset='4'/><br />
<vcpupin vcpu='5' cpuset='10'/><br />
<vcpupin vcpu='6' cpuset='5'/><br />
<vcpupin vcpu='7' cpuset='11'/><br />
<emulatorpin cpuset='0,6'/><br />
<iothreadpin iothread='1' cpuset='0,6'/><br />
</cputune><br />
...<br />
<cpu mode='host-passthrough'><br />
<topology sockets='1' cores='4' threads='2'/><br />
</cpu><br />
...<br />
}}<br />
<br />
===== 4c/2t AMD CPU example (Before ComboPi AGESA bios update) =====<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<vcpu placement='static'>8</vcpu><br />
<iothreads>1</iothreads><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='3'/><br />
<vcpupin vcpu='2' cpuset='4'/><br />
<vcpupin vcpu='3' cpuset='5'/><br />
<vcpupin vcpu='4' cpuset='6'/><br />
<vcpupin vcpu='5' cpuset='7'/><br />
<vcpupin vcpu='6' cpuset='8'/><br />
<vcpupin vcpu='7' cpuset='9'/><br />
<emulatorpin cpuset='0-1'/><br />
<iothreadpin iothread='1' cpuset='0-1'/><br />
</cputune><br />
...<br />
<cpu mode='host-passthrough'><br />
<topology sockets='1' cores='4' threads='2'/><br />
</cpu><br />
...<br />
}}<br />
<br />
{{Note|If further CPU isolation is needed, consider using the '''isolcpus''' kernel command-line parameter on the unused physical/logical cores.}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, you may want to pin your VM threads across all of your cores, so that the VM can fully take advantage of the spare CPU time the host has available. Be aware that pinning all physical and logical cores of your CPU could induce latency in the guest VM.<br />
<br />
=== Huge memory pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information across multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4 GiB of memory divided into 4 KiB pages (which is the default size for normal pages) for a total of 1.04 million pages, meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession.<br />
<br />
==== Transparent huge pages ====<br />
<br />
QEMU will use 2MiB sized transparent huge pages automatically without any explicit configuration in QEMU or Libvirt, subject to some important caveats. When using VFIO the pages are locked in at boot time and transparent huge pages are allocated up front when the VM first boots. If the kernel memory is highly fragmented, or the VM is using a majority of the remaining free memory, it is likely that the kernel will not have enough 2MiB pages to fully satisfy the allocation. In such a case, it silently fails by using a mix of 2MiB and 4KiB pages. Since the pages are locked in VFIO mode, the kernel will not be able to convert those 4KiB pages to huge after the VM starts either. The number of available 2MiB huge pages available to THP is the same as via the [[#Dynamic huge pages]] mechanism described in the following sections.<br />
<br />
To check how much memory THP is using globally:<br />
<br />
{{hc|$ grep AnonHugePages /proc/meminfo|<br />
AnonHugePages: 8091648 kB<br />
}}<br />
<br />
To check a specific QEMU instance. QEMU's PID must be substituted in the grep command:<br />
<br />
{{hc|$ grep -P 'AnonHugePages:\s+(?!0)\d+' /proc/[PID]/smaps|<br />
AnonHugePages: 8087552 kB<br />
}}<br />
<br />
In this example, the VM was allocated 8388608KiB of memory, but only 8087552KiB was available via THP. The remaining 301056KiB are allocated as 4KiB pages. Aside from manually checking, there is no indication when partial allocations occur. As such, THP's effectiveness is very much dependent on the host system's memory fragmentation at the time of VM startup. If this trade off is unacceptable or strict guarantees are required, [[#Static huge pages]] is recommended.<br />
<br />
Arch kernels have THP compiled in and enabled by default with {{ic|1=/sys/kernel/mm/transparent_hugepage/enabled}} set to {{ic|1=madvise}} mode.<br />
<br />
Also when trying to allocate pages for {{ic|virsh}}, e.g. {{ic|1=virsh allocpages 2M 2048}}, you might need to free up cached RAM. This is because your system might have allocated RAM for cache resources over time. One possible solution is to drop the page cache of your system:<br />
<br />
{{ic|# echo 1 > /proc/sys/vm/drop_caches}}<br />
<br />
==== Static huge pages ====<br />
<br />
While transparent huge pages should work in the vast majority of cases, they can also be allocated statically during boot. This should only be needed to make use 1 GiB hugepages on machines that support it, since transparent huge pages normally only go up to 2 MiB.<br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4 GiBs worth of huge pages on a machine with 8 GiB of memory will only leave you with 4 GiB of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
{{Note|The described procedures have some drawbacks but will not necessarily net you a great performance advantage. According to [https://developers.redhat.com/blog/2021/04/27/benchmarking-transparent-versus-1gib-static-huge-page-performance-in-linux-virtual-machines#benchmarks Red Hat benchmarks], you shouldn't expect more than a 2% performance gain from this over [[#Transparent huge pages]].}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel command line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048 KiB per huge page creates 2 GiB worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GiB huge page support could be verified by {{ic|grep pdpe1gb /proc/cpuinfo}}. Setting 1 GiB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
}}<br />
<br />
==== Dynamic huge pages ====<br />
<br />
{{Accuracy|Need futher testing if this variant as effective as static one}}<br />
<br />
Hugepages could be allocated manually via {{ic|vm.nr_overcommit_hugepages}} [[sysctl]] parameter.<br />
<br />
{{hc|/etc/sysctl.d/10-kvm.conf|2=<br />
vm.nr_hugepages = 0<br />
vm.nr_overcommit_hugepages = ''num''<br />
}}<br />
<br />
Where {{ic|''num''}} - is the number of huge pages, which default size if 2 MiB.<br />
Pages will be automatically allocated, and freed after VM stops.<br />
<br />
More manual way:<br />
<br />
# echo ''num'' > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages<br />
# echo ''num'' > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages<br />
<br />
For 2 MiB and 1 GiB page size respectively.<br />
And they should be manually freed in the same way.<br />
<br />
It is hardly recommended to drop caches, compact memory and wait couple of seconds before starting VM, as there could be not enough free contiguous memory for required huge pages blocks. Especially after some uptime of the host system.<br />
<br />
# echo 3 > /proc/sys/vm/drop_caches<br />
# echo 1 > /proc/sys/vm/compact_memory<br />
<br />
Theoretically, 1 GiB pages works as 2 MiB. But practically - no guaranteed way was found to get contiguous 1 GiB memory blocks. Each consequent request of 1 GiB blocks lead to lesser and lesser dynamically allocated count.<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
{{Warning|Depending on your processor, it might actually be detrimental to performance to force the whole CPU to run at full frequency at all times. For instance, modern AMD processors (Zen 2 and Zen 3) depend heavily on being able to scale individual cores in separate core complexes for optimal thermal distribution. If the whole CPU is running at full frequency at all times, there's less headroom for invididual cores to clock high, resulting in worse performance for processes that aren't heavily multithreaded such as games. This should be benchmarked in your VM.}}<br />
<br />
=== Isolating pinned CPUs ===<br />
<br />
CPU pinning by itself will not prevent other host processes from running on the pinned CPUs. Properly isolating the pinned CPUs can reduce latency in the guest VM.<br />
<br />
==== With isolcpus kernel parameter ====<br />
<br />
In this example, let us assume you are using CPUs 4-7.<br />
Use the [[kernel parameters]] {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. For example:<br />
<br />
isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.removeddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this Removeddit mirror of a Reddit thread] for more info. ([https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ The original thread] is worthless because of deleted comments.)<br />
<br />
==== Dynamically isolating CPUs ====<br />
<br />
The isolcpus kernel parameter will permanently reserve CPU cores, even when the guest is not running. A more flexible alternative is to dynamically isolate CPUs when starting the guest. This can be achieved with the following alternatives:<br />
<br />
* {{AUR|cpuset-git}} ([https://www.redhat.com/archives/vfio-users/2016-September/msg00072.html vfio-users post], [https://rokups.github.io/#!pages/gaming-vm-performance.md blog post], [https://github.com/PassthroughPOST/VFIO-Tools/blob/master/libvirt_hooks/hooks/cset.sh example script])<br />
* {{AUR|vfio-isolate}}<br />
* systemd<br />
<br />
===== Example with systemd =====<br />
<br />
In this example, we assume a host with 12 CPUs, where CPUs 2-5 and 8-11 are [[PCI_passthrough_via_OVMF#CPU_pinning|pinned]] to the guest. Then run the following to isolate the host to CPUs 0, 1, 6, and 7:<br />
<br />
# systemctl set-property --runtime -- user.slice AllowedCPUs=0,1,6,7<br />
# systemctl set-property --runtime -- system.slice AllowedCPUs=0,1,6,7<br />
# systemctl set-property --runtime -- init.scope AllowedCPUs=0,1,6,7<br />
<br />
After shutting down the guest, run the following to reallocate all 12 CPUs back to the host:<br />
<br />
# systemctl set-property --runtime -- user.slice AllowedCPUs=0-11<br />
# systemctl set-property --runtime -- system.slice AllowedCPUs=0-11<br />
# systemctl set-property --runtime -- init.scope AllowedCPUs=0-11<br />
<br />
You can use a [https://libvirt.org/hooks.html libvirt hook] to automatically run the above at startup/shutdown of the guest like so:<br />
<br />
Create or edit {{ic|/etc/libvirt/hooks/qemu}} with the following content.<br />
<br />
{{hc|/etc/libvirt/hooks/qemu|2=<nowiki><br />
#!/bin/bash<br />
<br />
command=$2<br />
<br />
if [[ $command == "started" ]]; then<br />
systemctl set-property --runtime -- system.slice AllowedCPUs=0,1,6,7<br />
systemctl set-property --runtime -- user.slice AllowedCPUs=0,1,6,7<br />
systemctl set-property --runtime -- init.slice AllowedCPUs=0,1,6,7<br />
elif [[ $command == "release" ]]; then<br />
systemctl set-property --runtime -- system.slice AllowedCPUs=0-11<br />
systemctl set-property --runtime -- user.slice AllowedCPUs=0-11<br />
systemctl set-property --runtime -- init.slice AllowedCPUs=0-11<br />
fi<br />
</nowiki>}}<br />
<br />
Afterwards make it executable: {{ic|chmod +x /etc/libvirt/hooks/qemu}}.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and then start your VM. If you create some heavily multithreaded load on your host now, you should see that it keeps your chosen CPUs free from load while the VM can still make use of it. You should also see those CPUs automatically getting fully used by your host once you terminate the VM.<br />
<br />
More examples are contained in the following reddit threads: [https://www.reddit.com/r/VFIO/comments/ebe3l5/deprecated_isolcpus_workaround/fem8jgk] [https://www.reddit.com/r/VFIO/comments/gyem88/noob_question_how_to_isolate_cpu_cores/ftaqno1/] [https://www.reddit.com/r/VFIO/comments/ij25rg/splitting_ht_cores_between_host_and_vm/]<br />
<br />
Note that this requires systemd 244 or higher, and [[cgroups|cgroups v2]], which is now enabled by default.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Starting with QEMU 3.1 the TOPOEXT cpuid flag is disabled by default. In order to use hyperthreading (SMT) on AMD CPUs you need to manually enable it:<br />
<br />
<cpu mode='host-passthrough' check='none'><br />
<topology sockets='1' cores='4' threads='2'/><br />
<feature policy='require' name='topoext'/><br />
</cpu><br />
<br />
commit: https://git.qemu.org/?p=qemu.git;a=commit;h=7210a02c58572b2686a3a8d610c6628f87864aed<br />
<br />
=== Virtio disk ===<br />
<br />
{{Merge|QEMU#Installing virtio drivers|Off-topic.|section=Moving virtio disk section to QEMU}}<br />
<br />
The default disk types are SATA or IDE emulation out of the box. These controllers offer maximum compatibility but are not suited for efficient virtualization. Two accelerated models exist: {{ic|1=virtio-scsi}} for SCSI emulation and passthrough, or {{ic|1=virtio-blk}} for a more basic block device emulation.<br />
<br />
==== Drivers ====<br />
<br />
* Linux guests should support these out of the box on any modern kernel<br />
* macOS has {{ic|1=virtio-blk}} support starting in Mojave via {{ic|1=AppleVirtIO.kext}}<br />
* Windows needs the [https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/ Windows virtio drivers]. {{ic|1=virtio-scsi}} uses the {{ic|1=vioscsi}} driver. {{ic|1=virtio-blk}} uses the {{ic|1=viostor}} driver<br />
* Windows can be installed directly onto these disks by selecting 'load driver' on the installer disk selection menu. The windows iso and virtio driver iso should both be attached as regular SATA/IDE cdroms during the installation process<br />
* To switch boot disks to virtio on an existing Windows installation:<br />
** {{ic|1=virtio-blk}}: Add a temporary disk with bus {{ic|1=virtio}}, boot windows & load the driver for the disk, then shutdown and switch the boot disk disk bus to {{ic|1=virtio}}<br />
** {{ic|1=virtio-scsi}}: Add a scsi controller with model {{ic|1=virtio}}, boot windows & load the driver for the controller, then shutdown and switch the boot disk bus to {{ic|1=scsi}} (not virtio)<br />
<br />
==== Considerations ====<br />
<br />
* {{ic|1=virtio-scsi}} TRIM support is mature, all versions should support it. Traditionally, {{ic|1=virtio-scsi}} has been the preferred approach for this reason<br />
* {{ic|1=virtio-blk}} TRIM support is new, this requires requires qemu 4.0+, guest linux kernel 5.0+, guest windows drivers 0.1.173+<br />
* Thin provisioning works by enabling TRIM on a sparse image file: {{ic|1=discard='unmap'}}. Unused blocks will be freed and the disk usage will drop (works on both raw and qcow2). Actual on-disk size of a sparse image file may be checked with {{ic|1=du /path/to/disk.img}}<br />
* Thin provisioning can also work with block storage such as zfs zvols or thin lvm<br />
* Virt queue count will influence the number of threads inside the guest kernel used for IO processing, suggest using {{ic|1=queues='4'}} or more<br />
* Native mode ({{ic|1=io='native'}}) uses a single threaded model based on linux AIO, is a bit more CPU efficient but may have lower peak performance and does not allow host side caching to be used<br />
* Threaded mode ({{ic|1=io='threads'}}) will spawn dozens of threads on demand as the disk is used. This is less efficient but may perform better if there are enough host cores available to run them, and allows for host side caching to be used<br />
* Modern versions of libvirt will group the dynamic worker threads created when using threaded mode in with the iothread=1 cgroup for pinning purposes. Very old versions of libvirt left these in the emulator cgroup<br />
<br />
==== IO threads ====<br />
<br />
An IO thread is a dedicated thread for processing disk events, rather than using the main qemu emulator loop. This should not be confused with the worker threads spawned on demand with {{ic|1=io='threads'}}.<br />
<br />
* You can only use one iothread per disk controller. The thread must be assigned to a specific controller with {{ic|1=iothread='X'}} in the {{ic|1=<driver>}} tag. Furthermore, extra & unassigned iothreads will not be used and do nothing<br />
* In the case of {{ic|1=virtio-scsi}}, there is one controller for multiple scsi disks. The iothread is assigned on the controller: {{ic|1=<controller><driver iothread='X'>}}<br />
* In the case of {{ic|1=virtio-blk}}, each disk has its own controller. The iothread is assigned in the driver tag under the disk itself: {{ic|1=<disk><driver iothread='X'>}}<br />
* Since emulated disks incur a significant amount of CPU overhead, that can lead to vcpu stuttering under high disk load (especially high random IOPS). In this case it helps to pin the IO to different core(s) than your vcpus with {{ic|1=<iothreadpin>}}<br />
<br />
==== Examples with libvirt ====<br />
<br />
virtio-scsi + iothread + worker threads + host side writeback caching + full disk block device backend:<br />
<domain><br />
<devices><br />
<disk type='block' device='disk'><br />
<driver name='qemu' type='raw' cache='writeback' io='threads' discard='unmap'/><br />
<source dev='/dev/disk/by-id/ata-Samsung_SSD_840_EVO_1TB_S1D9NSAF206396F'/><br />
<target dev='sda' bus='scsi'/><br />
</disk><br />
<controller type='scsi' index='0' model='virtio-scsi'><br />
<driver iothread='1' queues='8'/><br />
</controller><br />
<br />
virtio-blk + iothread + native aio + no host caching + raw sparse image backend:<br />
<domain><br />
<devices><br />
<disk type='file' device='disk'><br />
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap' iothread='1' queues='8'/><br />
<source file='/var/lib/libvirt/images/pool/win10.img'/><br />
<target dev='vda' bus='virtio'/><br />
</disk><br />
<br />
Creating the iothreads:<br />
<domain><br />
<iothreads>1</iothreads><br />
<br />
Pinning iothreads:<br />
<domain><br />
<cputune><br />
<iothreadpin iothread='1' cpuset='0-1,6-7'/><br />
<br />
==== Example with virt-manager ====<br />
<br />
This will create a {{ic|1=virtio-blk}} device:<br />
# Open the VM preferences<br />
# Go to {{ic|Add Hardware > Storage}}<br />
# Create or choose a storage file<br />
# Select {{ic|Device Type: Disk device}} and {{ic|Bus type: VirtIO}}<br />
# Click Finish<br />
<br />
=== Virtio network ===<br />
<br />
The default NIC models rtl8139 or e1000 can be a bottleneck for gigabit+ speeds and have a significant amount of CPU overhead compared to {{ic|1=virtio-net}}.<br />
<br />
* Select {{ic|1=virtio}} as the model for the NIC with libvirt or use the {{ic|1=virtio-net-pci}} device in bare qemu<br />
* Windows needs the {{ic|1=NetKVM}} driver from [https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/ Windows virtio drivers]<br />
* Virtio uses vhost-net by default for in-kernel packet processing without exiting to userspace<br />
* Multiqueue can enabled for a further speedup with multiple connections but typically will not boost single stream speeds. For libvirt add {{ic|1=<driver queues='8'/>}} under the interface tag<br />
* Zero copy transmit may also be enabled on macvtap by setting the module parameter {{ic|1=vhost_net.experimental_zcopytx=1}} but this may actually have worse performance, see [https://github.com/torvalds/linux/commit/098eadce3c622c07b328d0a43dda379b38cf7c5e commit]<br />
<br />
Libvirt example with a bridge:<br />
<br />
<interface type='bridge'><br />
<mac address="52:54:00:6d:6e:2e"/><br />
<source bridge='br0'/><br />
<model type='virtio'/><br />
<driver queues='8'/><br />
</interface><br />
<br />
=== Further tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide].<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how vfio-pci uses your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthrough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/local/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|1=<br />
#!/bin/sh<br />
<br />
for i in /sys/bus/pci/devices/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" {{!}} sed -e "s/0$/1/")"<br />
USB="$(echo "$GPU" {{!}} sed -e "s/0$/2/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
if [ -d "$USB" ]; then<br />
echo "vfio-pci" > "$USB/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|1=<br />
#!/bin/sh<br />
<br />
DEVS="0000:03:00.0 0000:03:00.1"<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
}}<br />
<br />
==== Script installation ====<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}:<br />
<br />
# Add {{ic|modconf}} to the [[mkinitcpio#HOOKS|HOOKS]] array and {{ic|/usr/local/bin/vfio-pci-override.sh}} to the [[mkinitcpio#BINARIES and FILES|FILES]] array.<br />
<br />
Edit {{ic|/etc/modprobe.d/vfio.conf}}:<br />
<br />
# Add the following line: {{ic|install vfio-pci /usr/local/bin/vfio-pci-override.sh}}<br />
# [[Regenerate the initramfs]] and reboot.<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|This is related to VBIOS issues and should be moved into a separate section regarding VBIOS compatibility.|section=UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that is not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.io/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total mebibytes + 10<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MiB + 10 = 25.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 25.82 is bigger than 16 we should choose 32.<br />
<br />
Next create a configuration file to create the shared memory file on boot<br />
<br />
{{hc|/etc/tmpfiles.d/10-looking-glass.conf|2=<br />
f /dev/shm/looking-glass 0660 '''user''' kvm -<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Ask systemd-tmpfiles to create the shared memory file now without waiting to next boot<br />
<br />
# systemd-tmpfiles --create /etc/tmpfiles.d/10-looking-glass.conf<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to go into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download the signed driver [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/upstream-virtio/ from Red Hat].<br />
<br />
Once the driver is installed you must download a matching [https://looking-glass.io/downloads looking-glass-host] package that matches the client you will install from AUR, and install it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]. The recent version will automatically install a service that starts the daemon on boot. The logs of the host daemon are located at {{ic|%ProgramData%\Looking Glass (host)\looking-glass-host.txt}} on the guest system.<br />
<br />
==== Setting up the null video device ====<br />
<br />
(Retrieved from: https://looking-glass.io/docs/stable/install/#spice-server)<br />
<br />
If you would like to use Spice to give you keyboard and mouse input along with clipboard sync support, make sure you have a {{ic|1=<graphics type='spice'>}} device, then:<br />
<br />
* Find your {{ic|<video>}} device, and set {{ic|1=<model type='none'/>}}<br />
* If you cannot find it, make sure you have a {{ic|<graphics>}} device, save and edit again<br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you do not want to use Spice to control the guest mouse and keyboard you can disable the Spice server.<br />
<br />
$ looking-glass-client -s<br />
<br />
Additionally you may want to start Looking Glass Client as full screen, otherwise the image may be scaled down resulting in poor image fidelity.<br />
<br />
$ looking-glass-client -F<br />
<br />
Launch with the {{ic|--help}} option for further information.<br />
<br />
==== Additional information ====<br />
<br />
Refer to the [https://looking-glass.io/wiki/Installation upstream documentation] for further details.<br />
<br />
=== Swap peripherals to and from the Host ===<br />
<br />
Looking Glass includes a Spice client in order to control mouse movement on the Windows guest. However this may have too much latency for certain applications, such as gaming. An alternative method is passing through specific USB devices for minimal latency. This allows for switching the devices between host and guest.<br />
<br />
First create a .xml file for the device(s) you wish to pass-through, which libvirt will use to identify the device.<br />
<br />
{{hc|~/.VFIOinput/input_1.xml|2=<br />
<hostdev mode='subsystem' type='usb' managed='no'><br />
<source><br />
<vendor id='0x[Before Colon]'/><br />
<product id='0x[After Colon]'/><br />
</source><br />
</hostdev><br />
}}<br />
<br />
Replace [Before/After Colon] with the contents of the 'lsusb' command, specific to the device you want to pass-through.<br />
<br />
For instance my mouse is {{ic|Bus 005 Device 002: ID 1532:0037 Razer USA, Ltd}} so I would replace {{ic|vendor id}} with 1532, and {{ic|product id}} with 1037.<br />
<br />
Repeat this process for any additional USB devices you want to pass-through. If your mouse / keyboard has multiple entries in {{ic|lsusb}}, perhaps if it is wireless, then create additional xml files for each.<br />
<br />
{{Note|Do not forget to change the path & name of the script(s) above and below to match your user and specific system.}}<br />
<br />
Next a bash script file is needed to tell libvirt what to attach/detach the USB devices to the guest.<br />
<br />
{{hc|~/.VFIOinput/input_attach.sh|2=<br />
#!/bin/bash<br />
<br />
virsh attach-device [VM-Name] [USBdevice]<br />
}}<br />
<br />
Replace [VM-Name] with the name of your virtual machine, which can be seen under virt-manager. Additionally replace [USBdevice] with the '''full''' path to the .xml file for the device you wish to pass-through. Add additional lines for more than 1 device. For example here is my script:<br />
<br />
{{hc|~/.VFIOinput/input_attach.sh|2=<br />
#!/bin/bash<br />
<br />
virsh attach-device win10 /home/$USER/.VFIOinput/input_mouse.xml<br />
virsh attach-device win10 /home/$USER/.VFIOinput/input_keyboard.xml<br />
}}<br />
<br />
Next duplicate the script file and replace {{ic|attach-device}} with {{ic|detach-device}}. Ensure both scripts are executable with {{ic|chmod +x $script.sh}}<br />
<br />
This 2 script files can now be executed to attach or detach your USB devices from the host to the guest VM. It is important to note that they may need to be executed as root. To run the script from the Windows VM, one possibility is using [[PuTTY]] to [[SSH]] into the host, and execute the script. On Windows PuTTY comes with plink.exe which can execute singular commands over SSH before then logging out, instead of opening a SSH terminal, all in the background.<br />
<br />
{{hc|detach_devices.bat|2=<br />
"C:\Program Files\PuTTY\plink.exe" root@$HOST_IP -pw $ROOTPASSWORD /home/$USER/.VFIOinput/input_detach.sh<br />
}}<br />
<br />
Replace {{ic|$HOST_IP}} with the Host [[Network configuration#IP addresses|IP Address]] and $ROOTPASSWORD with the root password.<br />
<br />
{{warning|This method is insecure if somebody has access to your VM, since they could open the file and read your password. It is advisable to use [[SSH keys]] instead!}}<br />
<br />
You may also want to execute the script files using key binds. On Windows one option is [https://autohotkey.com/ Autohotkey], and on the Host [[Xbindkeys]]. Because of the need to run the scripts as root, you may also need to use [[Polkit]] or [[Sudo]] which can both be used to authenticate specific executables as able to run as root without needing a password.<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{pkg|linux-zen}} or {{AUR|linux-vfio}} package.<br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream,multifunction}} should break up as many devices as possible.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|edk2-ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/edk2-ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2-ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|<br />
* Make sure that {{ic|1=OVMF_CODE.fd}} is given as a command line parameter before {{ic|1=MY_VARS.fd}}. The boot sequence will fail otherwise.<br />
* QEMU's default SeaBIOS can be used instead of OVMF, but it is not recommended as it can cause issues with passthrough setups.<br />
}}<br />
<br />
It is recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing through other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting: [[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|1=$ for usb_ctrl in /sys/bus/pci/devices/*/usb*; do pci_path=${usb_ctrl%/*}; iommu_group=$(readlink $pci_path/iommu_group); echo "Bus $(cat $usb_ctrl/busnum) --> ${pci_path##*/} (IOMMU group ${iommu_group##*/})"; lsusb -s ${usb_ctrl#*/usb}:; echo; done|2=<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
{{Note|If your USB controller does not support resetting, is not in an isolated group, or is otherwise unable to be passed through then it may still be possible to accomplish similar results through [[udev]] rules. See [https://github.com/olavmrk/usb-libvirt-hotplug] which allows any device connected to specified USB ports to be automatically attached to a virtual machine.}}<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
An emulated audio setup consists of two components: An emulated sound device exposed to the guest and an audio backend connecting the sound device to the host's PulseAudio.<br />
<br />
Of the emulated sound devices available, two are of main interest: ICH9 and usb-audio. ICH9 features both output and input but is limited to stereo. usb-audio only features audio output but supports up to 6 channels in 5.1 configuration. For ICH9 remove any pre-existing audio backend in the {{ic|<devices>}} section and add:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<sound model='ich9'><br />
<codec type='micro'/><br />
<audio id='1'/><br />
</sound><br />
<audio id='1' type='pulseaudio' serverName='/run/user/1000/pulse/native'/><br />
}}<br />
<br />
Note the matching {{ic|id}} elements. The example above assumes a single-user system with user ID 1000. Use the {{ic|id}} command to find the correct ID. You can also use the {{ic|/tmp}} directory if you have multiple users accessing PulseAudio:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<audio id='1' type='pulseaudio' serverName='unix:/tmp/pulse-socket'/><br />
}}<br />
<br />
If you get crackling or distorted sound, try experimenting with some latency settings. The following example uses 20000 microseconds:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<audio id="1" type="pulseaudio" serverName="/run/user/1000/pulse/native"><br />
<input latency="20000"/><br />
<output latency="20000"/><br />
</audio><br />
}}<br />
<br />
You can also try disabling the software mixer included in QEMU. This should, in theory, be more efficient and allow for lower latencies since mixing will then take place on your host only:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<audio id="1" type="pulseaudio" serverName="/run/user/1000/pulse/native"><br />
<input mixingEngine="no"/><br />
<output mixingEngine="no"/><br />
</audio><br />
}}<br />
<br />
For usb-audio, the corresponding elements read<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<sound model='usb'><br />
<audio id='1'/><br />
</sound><br />
<audio id='1' type='pulseaudio' serverName='/run/user/1000/pulse/native'/><br />
}}<br />
<br />
However, if a 5.1 configuration is required the sound device needs to be configured via QEMU command line arguments:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
</devices><br />
<qemu:commandline><br />
<qemu:arg value='-device'/><br />
<qemu:arg value='usb-audio,id=sound0,audiodev=audio1,multi=on'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
The {{ic|audiodev}} tag has to be set to match the audio backend's {{ic|id}} element. {{ic|1=id='1'|2==}} corresponds to {{ic|audio1}} and so on. See [[#Libvirt versions below 7.3]] for further details how to directly pass QEMU arguments.<br />
<br />
{{Note|1=<nowiki/><br />
* You can have multiple audio backends, by simply specifying {{ic|<audio>}}/{{ic|-audiodev}} multiple times in your XML and by assigning them different ids. This can be useful for a use case of having two identical backends. With PulseAudio each backend is a separate stream and can be routed to different output devices on the host (using a pulse mixer app like {{Pkg|pavucontrol}} or {{Pkg|pulsemixer}}).<br />
* USB 3 emulation is needed in Libvirt/QEMU to enable the usb-audio.<br />
* It is recommended to enable MSI interrupts with a tool such as [https://forums.guru3d.com/threads/windows-line-based-vs-message-signaled-based-interrupts-msi-tool.378044/] on the ICH9 audio device to mitigate any crackling, stuttering, speedup, or no audio at all after VM restart.<br />
* If audio is crackling/stuttering/speedup etc. is still present you may want to adjust parameters such as {{ic|buffer-length}} and {{ic|timer-period}}, more information on these parameters and more can be found in the {{man|1|qemu}} manual.<br />
* Some audio chipsets such as [https://bugzilla.kernel.org/show_bug.cgi?id=195303 Realtek alc1220] may also have issues out of the box so do consider this when using any audio emulation with QEMU.<br />
* Improper pinning or heavy host usage without using [[#Isolating pinned CPUs|isolcpus]] can also influence sound bugs, especially while gaming in a VM.<br />
}}<br />
<br />
=== Passing VM audio to host via JACK and PipeWire ===<br />
<br />
It is also possible to pass VM audio to the host via JACK and PipeWire.<br />
<br />
First, make sure you have a working [[PipeWire]] setup with [[PipeWire#JACK_clients|JACK support]]. In addition to that, you need to install {{Pkg|pipewire-jack}} as it will allow QEMU or libvirtd to easily access PipeWire's JACK sink. <br />
<br />
Next, you'll need to tell libvirt to run QEMU as your user:<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
As a final preparation, the XML scheme has to be extended to allow passing of environment variables. For this, modify the VM domain configuration<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
<domain type='kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
<domain type='kvm' xmlns:qemu='<nowiki>http://libvirt.org/schemas/domain/qemu/1.0</nowiki>'><br />
}}<br />
<br />
Then, you can add the actual audio config to your VM:<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<nowiki/><br />
<devices><br />
...<br />
<audio id="1" type="jack"><br />
<input clientName="vm-win10" connectPorts="your-input"/><br />
<output clientName="vm-win10" connectPorts="your-output"/><br />
</audio><br />
</devices><br />
<qemu:commandline><br />
<qemu:env name="PIPEWIRE_RUNTIME_DIR" value="/run/user/1000"/><br />
<qemu:env name="PIPEWIRE_LATENCY" value="512/48000"/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
{{Note|Use a tool like {{Pkg|carla}} to figure out which input and outputs you want.}}<br />
<br />
Note the matching {{ic|id}} elements. Above's example assumes a single-user system with user ID 1000. Use the {{ic|id}} command to find the correct ID.<br />
<br />
You might have to play with the {{ic|PIPEWIRE_LATENCY}} values to get to the desired latency without crackling.<br />
<br />
=== Passing VM audio to host via Scream ===<br />
<br />
It is possible to pass VM audio through a bridged network such as the one provided by Libvirt or by adding a IVSHMEM device to the host by using a application called [https://github.com/duncanthrax/scream Scream]. This section will only cover using PulseAudio as a receiver on the host.<br />
See the project page for more details and instructions on other methods.<br />
<br />
==== Using Scream with a bridged network ====<br />
<br />
{{Note|<br />
* This is the ''preferred'' way to use this, although results may vary per user<br />
* It is recommend to use the [[#Virtio network]] adapter while using Scream, other virtual adapters provided by QEMU such as '''e1000e''' may lead to poor performance<br />
}}<br />
<br />
To use scream via your network you will want to find your bridge name via {{ic|1=ip a}}, in most cases it will be called '''br0''' or '''virbr0'''. Below is a example of the command needed to start the Scream application:<br />
<br />
$ scream -o pulse -i virbr0 &<br />
<br />
{{Warning| This will not work with a '''macvtap bridge''' as that does not allow host to guest communication, also make sure you have the proper firewall ports open for it to communicate with the VM}}<br />
<br />
==== Adding the IVSHMEM device to use Scream with IVSHMEM ====<br />
<br />
With the VM turned off, edit the machine configuration<br />
<br />
{{hc|$ virsh edit ''vmname''|2=<br />
...<br />
<devices><br />
...<br />
<shmem name='scream-ivshmem'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>2</size><br />
</shmem><br />
</devices><br />
...<br />
}}<br />
<br />
In the above configuration, the size of the IVSHMEM device is 2MB (the recommended amount). Change this as needed.<br />
<br />
Now refer to [[#Adding IVSHMEM Device to VM]] to configure the host to create the shared memory file on boot, replacing {{ic|looking-glass}} with {{ic|scream-ivshmem}}.<br />
<br />
===== Configuring the Windows guest for IVSHMEM =====<br />
<br />
The correct driver must be installed for the IVSHMEM device on the guest. <br />
See [[#Installing the IVSHMEM Host to Windows guest]]. Ignore the part about {{ic|looking-glass-host}}.<br />
<br />
Install the [https://github.com/duncanthrax/scream/releases Scream] virtual audio driver on the guest. <br />
If you have secure boot enabled for your VM, you may need to disable it. <br />
<br />
Using the registry editor, set the DWORD {{ic|HKLM\SYSTEM\CurrentControlSet\Services\Scream\Options\UseIVSHMEM}} to the size of the IVSHMEM device in MB.<br />
Note that scream identifies its IVSHMEM device using its size, so make sure there is only one device of that size.<br />
<br />
====== Configuring the host ======<br />
<br />
Install {{AUR|scream}}.<br />
<br />
Create a [[systemd/User|systemd user service]] to control the receiver:<br />
<br />
{{hc|~/.config/systemd/user/scream-ivshmem-pulse.service|2=<br />
[Unit]<br />
Description=Scream IVSHMEM pulse receiver<br />
After=pulseaudio.service<br />
Wants=pulseaudio.service<br />
<br />
[Service]<br />
Type=simple<br />
ExecStartPre=/usr/bin/truncate -s 0 /dev/shm/scream-ivshmem<br />
ExecStartPre=/usr/bin/dd if=/dev/zero of=/dev/shm/scream-ivshmem bs=1M count=2<br />
ExecStart=/usr/bin/scream -m /dev/shm/scream-ivshmem<br />
<br />
[Install]<br />
WantedBy=default.target<br />
}}<br />
<br />
Edit {{ic|1=count=2}} with the size of the IVSHMEM device in MiB.<br />
<br />
{{Tip|If you are using [[PipeWire]], replace {{ic|pulseaudio.service}} with {{ic|pipewire-pulse.service}}.}}<br />
<br />
Now start the service with:<br />
<br />
$ systemctl start --user scream-ivshmem-pulse<br />
<br />
To have it automatically start on next login, enable the service:<br />
<br />
$ systemctl enable --user scream-ivshmem-pulse<br />
<br />
=== Physical disk/partition ===<br />
<br />
Raw and qcow2 especially can have noticeable overhead for heavy IO. A whole disk or a partition may be used directly to bypass the filesystem and improve I/O performance. If you wish to dual boot the guest OS natively you would need to pass the entire disk without any partitioning. It is suggested to use /dev/disk/by- paths to refer to the disk since /dev/sdX entries can change between boots. To find out which disk/partition is associated with the one you would like to pass:<br />
<br />
{{hc|$ ls -l /dev/disk/by-id|<br />
ata-ST1000LM002-9VQ14L_Z0501SZ9 -> ../../sdd<br />
}}<br />
<br />
See [[#Virtio disk]] on how to add these with libvirt XML. You can also add the disk with Virt-Manager's '''Add Hardware''' menu and then type the disk you want in the '''Select or create custom storage''' box, e.g. '''/dev/disk/by-id/ata-ST1000LM002-9VQ14L_Z0501SZ9'''<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functional and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore recommended to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
For many reasons users may seek to see [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]].<br />
<br />
These examples offer a supplement to existing hardware compatibility lists. Additionally, if you have trouble configuring a certain mechanism in your setup, you might find these examples very valuable. Users there have described their setups in detail, and some have provided examples of their configuration files as well. <br />
<br />
We encourage those who successfully build their system from this resource to help improve it by contributing their builds. Due to the many different hardware manufacturers involved, the seemingly significant lack of sufficient documentation, as well as other issues due to the nature of this process, community contributions are necessary.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== QEMU 4.0: Unable to load graphics drivers/BSOD/Graphics stutter after driver install using Q35 ===<br />
<br />
Starting with QEMU 4.0, the Q35 machine type changes the default {{ic|kernel_irqchip}} from {{ic|off}} to {{ic|split}} which breaks some guest devices, such as nVidia graphics (the driver fails to load / black screen / code 43 / graphics stutters, usually when mouse moving). Switch to full KVM mode instead by adding {{ic|1=<ioapic driver='kvm'/>}} under libvirt's {{ic|<features>}} tag in your VM configuration or by adding {{ic|1=kernel_irqchip=on}} in the {{ic|-machine}} QEMU arg.<br />
<br />
=== QEMU 5.0: host-passthrough with kernel version 5.5 to 5.8.1 when using Zen 2 processors: Windows 10 BSOD loop 'KERNEL SECURITY CHECK FAILURE' ===<br />
<br />
{{Note|As of kernel version 5.8.2, disabling STIBP is not required anymore.}}<br />
<br />
Starting with QEMU 5.0 virtual machines running on Zen 2 and newer kernels than 5.4 will cause a BSOD loop of: 'KERNEL SECURITY CHECK FAILURE'. This can be fixed by either updating to kernel version 5.8.2 or higher, or disabling STIBP:<br />
<cpu mode='host-passthrough' ...><br />
...<br />
<feature policy='disable' name='amd-stibp'/><br />
...<br />
</cpu><br />
This requires libvirt 6.5 or higher. On older versions, several workarounds exist:<br />
* Switch CPU mode from {{ic|host-passthrough}} to {{ic|host-model}}. This only works on libvirt 6.4 or lower.<br />
* Manually patch {{Pkg|qemu}} in order to revert [https://github.com/qemu/qemu/commit/143c30d4d346831a09e59e9af45afdca0331e819 this] commit.<br />
* On qemu commandline, add {{ic|1=amd-stibp=off}} to the cpu flags string. This can also be invoked through libvirt via a {{ic|<qemu:commandline>}} entry.<br />
<br />
=== "Error 43: Driver failed to load" with mobile (Optimus/max-q) nvidia GPUs ===<br />
<br />
This error occurs because the Nvidia driver wants to check the status of the power supply. If no battery is present, the driver does not work. Whether Libvirt or Quemu, by default none of them provide the possibility to simulate a battery. This might also result in a reduced screen resolution and the Nvidia Desktop Manager refusing to load when right-clicking the desktop, saying it requires Windows 10, a compatible GPU and the Nvidia graphics driver.<br />
<br />
You can however create and add a custom acpi table file to the virtual machine which will do the work.<br />
<br />
First you have to create the custom acpi table file by pasting the following base64 string [https://base64.guru/converter/decode/file here] and save the result file as SSDT1.dat:<br />
{{bc|1=<br />
U1NEVKEAAAAB9EJPQ0hTAEJYUENTU0RUAQAAAElOVEwYEBkgoA8AFVwuX1NCX1BDSTAGABBMBi5f<br />
U0JfUENJMFuCTwVCQVQwCF9ISUQMQdAMCghfVUlEABQJX1NUQQCkCh8UK19CSUYApBIjDQELcBcL<br />
cBcBC9A5C1gCCywBCjwKPA0ADQANTElPTgANABQSX0JTVACkEgoEAAALcBcL0Dk=<br />
}}<br />
<br />
Next you must add the processed file to the main domain of the virtual machine:<br />
{{bc|<nowiki><br />
<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm"><br />
...<br />
<qemu:commandline><br />
<qemu:arg value="-acpitable"/><br />
<qemu:arg value="file=/path/to/your/SSDT1.dat"/><br />
</qemu:commandline><br />
</domain><br />
</nowiki>}}<br />
<br />
Make sure your XML file has the correct namespace in the {{ic|<domain>}} tag as visible above, otherwise the XML verification will fail.<br />
<br />
[https://www.reddit.com/r/VFIO/comments/ebo2uk/nvidia_geforce_rtx_2060_mobile_success_qemu_ovmf/ Source]<br />
<br />
=== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ===<br />
<br />
{{Expansion|This error is actually related to the boot_vgs issue and should be merged together with everything else concerning GPU ROMs.|section=UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check ''dmesg'' for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
In addition try adding kernel parameter {{ic|1=pci=realloc}} which also [https://github.com/Dunedan/mbp-2016-linux/issues/60#issuecomment-396311301 helps with hotplugging issues].<br />
<br />
=== UEFI (OVMF) compatibility in VBIOS ===<br />
<br />
{{Remove|Flashing you guest GPU for the purpose of a GPU passthrough is '''never''' good advice. A full section should be dedicated to VBIOS compatibility.|section= UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it is not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it is stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it is pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
On a linux guest you can use modinfo to see if there is option to enable MSI (for example: "modinfo snd_hda_intel |grep msi"). If there is, one can enable it by adding the relevant option to a custom omdprobe file - in "/etc/modprobe.d/snd-hda-intel.conf" inserting "options snd-hda-intel enable_msi=1"<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
A UI tool called [https://github.com/CHEF-KOCH/MSI-utility MSI Utility (FOSS Version 2)]{{Dead link|2021|05|17|status=404}} works with Windows 10 64-bit and simplifies the process.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read [https://www.kernel.org/doc/html/latest/x86/intel-iommu.html#graphics-problems intel-iommu.html] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X does not start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== Host lockup after VM shutdown ===<br />
<br />
This issue seems to primarily affect users running a Windows 10 guest and usually after the VM has been run for a prolonged period of time: the host will experience multiple CPU core lockups (see [https://bbs.archlinux.org/viewtopic.php?id=206050&p=2]). To fix this try enabling Message Signal Interrupts on the GPU passed through to the guest. A good guide for how to do this can be found in [https://forums.guru3d.com/threads/windows-line-based-vs-message-signaled-based-interrupts.378044/]. You can also download this application for windows here [https://github.com/TechtonicSoftware/MSIInturruptEnabler] that should make the process easier.<br />
<br />
=== Host lockup if guest is left running during sleep ===<br />
<br />
VFIO-enabled virtual machines tend to become unstable if left running through a sleep/wakeup cycle and have been known to cause the host machine to lockup when an attempt is then made to shut them down. In order to avoid this, one can simply prevent the host from going into sleep while the guest is running using the following libvirt hook script and systemd unit. The hook file needs executable permissions to work.<br />
<br />
{{hc|/etc/libvirt/hooks/qemu|2=<br />
#!/bin/bash<br />
<br />
OBJECT="$1"<br />
OPERATION="$2"<br />
SUBOPERATION="$3"<br />
EXTRA_ARG="$4"<br />
<br />
case "$OPERATION" in<br />
"prepare")<br />
systemctl start libvirt-nosleep@"$OBJECT"<br />
;;<br />
"release")<br />
systemctl stop libvirt-nosleep@"$OBJECT"<br />
;;<br />
esac<br />
}}<br />
<br />
{{hc|/etc/systemd/system/libvirt-nosleep@.service|2=<br />
[Unit]<br />
Description=Preventing sleep while libvirt domain "%i" is running<br />
<br />
[Service]<br />
Type=simple<br />
ExecStart=/usr/bin/systemd-inhibit --what=sleep --why="Libvirt domain \"%i\" is running" --who=%U --mode=block sleep infinity<br />
}}<br />
<br />
=== Cannot boot after upgrading ovmf ===<br />
<br />
If you cannot boot after upgrading from {{Pkg|ovmf}}{{Broken package link|replaced by {{Pkg|edk2-ovmf}}}} version 1:r23112.018432f0ce-1 then you need to remove the old {{ic|*VARS.fd}} file in {{ic|/var/lib/libvirt/qemu/nvram/}}:<br />
<br />
# mv /var/lib/libvirt/qemu/nvram/vmname_VARS.fd /var/lib/libvirt/qemu/nvram/vmname_VARS.fd.old<br />
<br />
See {{Bug|57825}} for further details.<br />
<br />
=== QEMU via cli pulseaudio stuttering/delay ===<br />
<br />
Using following flags for the audio device and chipset might help if you are running into the stuttering/delay audio issues when running QEMU via cli:<br />
<br />
qemu-system-x86_64 \<br />
-machine pc-i440fx-3.0 \<br />
-device hda-micro \<br />
-soundhw hda \<br />
-...<br />
<br />
As noted in [[#QEMU 3.0 audio changes|QEMU 3.0 audio changes]]{{Broken section link}} the specified chipset will include a series of audio patches.<br />
<br />
Setting {{ic|QEMU_AUDIO_TIMER_PERIOD}} to values higher than 100 might also help (did not test value lower than 100).<br />
<br />
=== Bluescreen at boot since Windows 10 1803 ===<br />
<br />
Since Windows 10 1803 there is a problem when you are using "host-passthrough" as cpu model. The machine cannot boot and is either boot looping or you get a bluescreen.<br />
You can workaround this by:<br />
<br />
# echo 1 > /sys/module/kvm/parameters/ignore_msrs<br />
<br />
To make it permanently you can create a modprobe file {{ic|kvm.conf}}:<br />
<br />
options kvm ignore_msrs=1<br />
<br />
To prevent clogging up ''dmesg'' with "ignored rdmsr" messages you can additionally add:<br />
<br />
options kvm report_ignored_msrs=0<br />
<br />
=== AMD Ryzen / BIOS updates (AGESA) yields "Error: internal error: Unknown PCI header type ‘127’" ===<br />
<br />
AMD users have been experiencing breakage of their KVM setups after updating the BIOS on their motherboard. There is a kernel [https://clbin.com/VCiYJ patch], (see [[Kernel/Arch Build System]] for instruction on compiling kernels with custom patches) that can resolve the issue as of now (7/28/19), but this is not the first time AMD has made an error of this very nature, so take this into account if you are considering updating your BIOS in the future as a VFIO user.<br />
<br />
=== AMD GPU not resetting properly yielding "Error: internal error: Unknown PCI header type ‘127’" (Separate issue from the one above) ===<br />
<br />
Passing through an AMD GPU may result into a problem known as the "AMD reset bug". Upon power cycling the guest, the GPU does not properly reset its state which causes the device to malfunction until the host is also rebooted. This is usually paired with a "code 43" driver error in a Windows guest, and the message "Error: internal error: Unknown PCI header type '127'" in the libvirt log on the host.<br />
<br />
In the past, this meant having to use work-arounds to manually reset the GPU, or resorting to the use of kernel patches that were unlikely to land in upstream. Currently, the recommended solution that does not require patching of the kernel is to install {{AUR|vendor-reset-git}} or {{AUR|vendor-reset-dkms-git}} and making sure the 'vendor-reset' kernel module is loaded before booting the guest.<br />
<br />
{{Note| Make sure you do not have any of the AMD reset bug kernel patches installed if you are using {{AUR|vendor-reset-git}} or {{AUR|vendor-reset-dkms-git}}.}}<br />
<br />
=== Host crashes when hotplugging Nvidia card with USB ===<br />
<br />
If attempting to hotplug an Nvidia card with a USB port, you may have to blacklist the {{ic|i2c_nvidia_gpu}} driver. Do this by adding the line {{ic|blacklist i2c_nvidia_gpu}} to {{ic|/etc/modprobe.d/blacklist.conf}}.<br />
<br />
=== Host unable to boot and stuck in black screen after enabling vfio ===<br />
<br />
If debug kernel messages during boot are enabled by adding with the {{ic|debug ignore_loglevel}} [[kernel parameters]], you may see boot stuck with the last message similar to<br />
<br />
vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none<br />
<br />
This can be resolved by disconnecting the passthroughed GPU with your moniter. You may reconnect the passthroughed GPU to a moniter after host is booted.<br />
<br />
=== AER errors when passing through PCIe USB hub ===<br />
<br />
In some cases passing through a PCIe USB hub, such as one connected to the guest GPU, might fail with AER errors similar to the following:<br />
<br />
kernel: pcieport 0000:00:01.1: AER: Uncorrected (Non-Fatal) error received: 0000:00:01.1<br />
kernel: pcieport 0000:00:01.1: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)<br />
kernel: pcieport 0000:00:01.1: AER: device [8086:1905] error status/mask=00100000/00000000<br />
kernel: pcieport 0000:00:01.1: AER: [20] UnsupReq (First)<br />
kernel: pcieport 0000:00:01.1: AER: TLP Header: 00000000 00000000 00000000 00000000<br />
kernel: pcieport 0000:00:01.1: AER: device recovery successful<br />
<br />
=== Reserved Memory Region Reporting (RMRR) Conflict ===<br />
<br />
If you run into an issue passing through a device because of the BIOS's usage of RMRR, like the error below.<br />
<br />
vfio-pci 0000:01:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.<br />
<br />
You can try the patches here: https://github.com/kiler129/relax-intel-rmrr<br />
<br />
== See also ==<br />
<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://www.reddit.com/r/VFIO /r/VFIO: A subreddit focused on vfio]<br />
* [https://github.com/intel/gvt-linux/wiki/GVTd_Setup_Guide GVT-d: passthrough of an entire integrated GPU]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=ReadyMedia&diff=687323ReadyMedia2021-07-09T10:32:15Z<p>NeoTheFox: /* Running minidlnad without systemd */ use a more appropriate cache folder</p>
<hr />
<div>[[Category:Media servers]]<br />
[[es:ReadyMedia]]<br />
[[ja:ReadyMedia]]<br />
[https://sourceforge.net/projects/minidlna ReadyMedia] (previously '''MiniDLNA''') is server software with the aim of being fully compliant with [[wikipedia:Digital_Living_Network_Alliance|DLNA]]/[[wikipedia:Universal_Plug_and_Play|UPnP]] clients. The MiniDNLA daemon serves media files (music, pictures, and video) to clients on a network. Example clients include applications such as {{Pkg|totem}} and [[Kodi]], and devices such as portable media players, Smartphones, Televisions, and gaming systems (such as PS3 and Xbox 360).<br />
<br />
ReadyMedia is a simple, lightweight alternative to [[MediaTomb]], but has fewer features. It does not have a web interface for administration and must be configured by editing a text file.<br />
<br />
== Installation ==<br />
<br />
Install the {{Pkg|minidlna}} package.<br />
<br />
If you want to use an unofficial branch which supports transcoding, install the {{AUR|readymedia-transcode-git}} package.<br />
<br />
== Configuration and starting ==<br />
<br />
{{Warning|1=The current version of ReadyMedia has a serious bug: The {{ic|-i}} command-line option and {{ic|network_interface}} configuration option used to bind to a specific IP address does not apply to the HTTP server. Bug reported in 2017 to [https://sourceforge.net/p/minidlna/bugs/297/ upstream]. The webserver remains accessible on all interfaces, potentially creating a security problem if running on a publicly accessible host.}}<br />
<br />
By default, minidlna runs as a system service (alternatively, you can [[#Running minidlnad without systemd|run it manually]]). It can be configured in {{ic|/etc/minidlna.conf}}. Set the following necessary settings:<br />
<br />
{{bc|1=#network_interface=eth0 # Self-discovers if commented (at times necessary to set)<br />
media_dir=A,/home/user/Music # Mounted Media_Collection drive directories<br />
media_dir=P,/home/user/Pictures # Use A, P, and V to restrict media 'type' in directory<br />
media_dir=V,/home/user/Videos<br />
friendly_name=Media Server # Optional<br />
inotify=yes # 'no' for less resources, restart required for new media<br />
presentation_url=<nowiki>http://www.mylan/index.php</nowiki> # or use your device static IP http://192.168.0.14:8200/}}<br />
<br />
MiniDLNA runs as the ''minidlna'' user by default. To change the user it runs as, see [[#Running minidlna as your own user]] below.<br />
<br />
The minidlna service can be managed by {{ic|minidlna.service}} [[systemd#Using units|using systemd]].<br />
<br />
=== Automatic Media_DB Update ===<br />
<br />
Kernel adds one [[wikipedia:Inotify|inotify]] watch per each folder/subfolder in ''Media_Collection'' Directories set in {{ic|/etc/minidlna.conf}} to monitor changes thus allowing MiniDLNA to update ''Media_DB'' in real time. When MiniDLNA is run as a regular user, it does not have the ability to change the kernel's ''inotify'' limits. If default number of ''inotify'' watches is non-sufficient to have MiniDLNA monitor all your media folders, increase ''inotify'' watches through {{ic|sysctl}} (100000 should be enough for most uses):<br />
<br />
# sysctl fs.inotify.max_user_watches=100000<br />
<br />
To have it permanently changed, add to {{ic|/etc/sysctl.d/90-inotify.conf}}<br />
<br />
# Increase inotify max watchs per user for local minidlna<br />
fs.inotify.max_user_watches = 100000<br />
<br />
''inotify'' performance may depend on device type. Some do not rescan media drives on a consistent basis or at all. If files are added/deleted to monitored media directories, they may not be noticed until the device DLNA client is restarted.<br />
<br />
Check ''inotify'' updates via MiniDLNA ''presentation_url'' by comparing files count. If it does not change, make sure the user running MiniDLNA has ''rw'' access to the DB folder. If the issue persists, copy or download new files first to a non-watched by ''inotify'' '''Downloads''' folder on the same drive, and then move them to appropriate media folders, since lengthy media files copying or downloading may confuse ''inotify''.<br />
<br />
You can also clean or rebuild MiniDLNA DB manually after stopping MiniDLNA daemon, or analyze its debug output (Ctrl+C to exit):<br />
<br />
[[Stop]] the MiniDLNA daemon.<br />
<br />
To rebuild Media_DB forcibly:<br />
# minidlnad -R<br />
Stop the daemon after rebuilding Media_DB e.g. {{Ic|killall minidlnad}}.<br />
<br />
To run in debug mode:<br />
# minidlnad -d<br />
{{ic|Ctrl+C}} to exit it.<br />
<br />
=== Troubleshooting service autostart ===<br />
<br />
Sometimes the minidlna daemon fails to start while booting. [[NetworkManager#Enable NetworkManager Wait Online]] solves this issue. See {{Bug|35325}}<br />
<br />
=== Running minidlna as your own user ===<br />
<br />
Change {{ic|user}} line in {{ic|/etc/minidlna.conf}}, and change the {{ic|db_dir}} and {{ic|log_dir}} options to directories that are writeable by that user. If you are [[#Running minidlnad without systemd|running without systemd]], this is all you need.<br />
<br />
If you are using the systemd service, [[edit]] {{ic|minidlna.service}} as it will by default run as the {{ic|minidlna}} user. To do so, create a {{ic|/etc/systemd/system/minidlna.service.d/run-as-user.conf}} [[drop-in file]]:<br />
<br />
{{hc|/etc/systemd/system/minidlna.service.d/run-as-user.conf|2=<br />
[Service]<br />
User=''user''<br />
Group=''group''<br />
}}<br />
<br />
{{Note|<br />
* Add {{ic|1=ProtectHome=off}} in the {{ic|[Service]}} section if you want minidlna to be able to read/write to {{ic|/home/''user''/}}, including the {{ic|db_dir}} or any {{ic|media_dir}}.<br />
* The {{ic|log_dir}} is not used by the systemd service. Instead, run {{ic|journalctl -u minidlna.service}} as root to view the logs.<br />
}}<br />
<br />
Copy the {{ic|/usr/lib/tmpfiles.d/minidlna.conf}} file to {{ic|/etc/tmpfiles.d/minidlna.conf}} and replace root with your own user and group:<br />
<br />
=== Running minidlnad without systemd ===<br />
<br />
Alternatively to a system service, you can run minidlna manually. This can be useful if you want to share media but do not have administrator access to the machine.<br />
<br />
Create the necessary files and directories locally and edit the configuration:<br />
<br />
$ install -Dm644 /etc/minidlna.conf ~/.config/minidlna/minidlna.conf<br />
$ $EDITOR minidlna.conf<br />
<br />
Configuring should be as above, specifically:<br />
<br />
media_dir=/home/$USER/dir<br />
db_dir=/home/$USER/.cache/minidlna<br />
log_dir=/home/$USER/.config/minidlna<br />
<br />
You can now start minidlna with the following command:<br />
<br />
$ minidlnad -f /home/$USER/.config/minidlna/minidlna.conf -P /home/$USER/.config/minidlna/minidlna.pid<br />
<br />
To autostart it at login, add the previous line to {{ic|~/.bash_profile}}.<br />
<br />
== Other aspects ==<br />
<br />
Other aspects and MiniDLNA limitations may need to be considered beforehand to ensure satisfaction from its performance.<br />
<br />
=== Firewall ===<br />
<br />
If using a firewall the the ssdp (1900/udp) and trivnet1 (8200/tcp) ports will need to be opened. For example, this can be done with arno's iptables firewall by editing {{ic|firewall.conf}} and opening the ports by doing:<br />
<br />
OPEN_TCP="8200"<br />
OPEN_UDP="1900"<br />
<br />
==== iptables ====<br />
<br />
In addition to 1900/udp and 8200/tcp, minidlna may also require allowing muliticasts. An overall configuration for [[iptables]] may therefore look like the following snippet, assuming there are separate chains which handel [[Simple_stateful_firewall#Creating_necessary_chains|TCP, UDP]] and IGMP.<br />
<br />
-A TCP -m tcp --dport 8200 -m comment --comment minidlna -j ACCEPT<br />
-A UDP -d 239.255.255.250/32 -m udp --dport 1900 -m comment --comment "upnp dlna" -j ACCEPT<br />
-A IGMP -d 224.0.0.1/32 -m comment --comment "igmp membership queries, upnp/dlna" -j ACCEPT<br />
-A IGMP -d 239.0.0.0/8 -m comment --comment "igmp multicast, upnp/dlna" -j ACCEPT<br />
<br />
=== File System and Localization ===<br />
<br />
When keeping MiniDLNA ''Media_DB'' on an external drive accessible in both Linux and Windows, choose proper [[file system]] for it. '''NTFS''' preserves in Windows its Linux defaults: ''rw'' access for ''root'' user and ''UTF8'' font ''encoding'' for file names, so media titles in your language are readable when browsing ''Media_DB'' in terminal and media players, since most support ''UTF8''. If you prefer '''Vfat''' (FAT32) for better USB drive compatibility with older players when hooked directly, or your ''Media_Collection'' drive is Vfat and has folder & file names in your local language, MiniDLNA can transcode them to ''UTF8'' charset while scanning folders to ''Media_DB''. Add to ''Media_Collection'' and ''Media_DB'' drives' mount options your FS language [[wikipedia:Code_page|codepage]] for transcoding to short DOS file names, and [[wikipedia:Character_encoding|iocharset]] for converting long file names to your terminal's locale, i.g. ''codepage=cp866,iocharset=utf8 (or ISO-8859-5)''. Set ''rw'' permissions for all users, since ''Vfat'' does not preserve Linux access permissions:<br />
<br />
{{bc|1=UUID=6140-75F7 /media/MyDrive/Media_DB vfat user,rw,async,noatime,umask=111,dmask=000,codepage=cp866,iocharset=utf8 0 0}}<br />
<br />
While your ''iocharset'' would be present in the system with a matching [[locale]], if your terminal or player supports only short file names, check if the set ''codepage'' is also present and enabled (like ru_RU.CP866), i.e. was included in system config when ArchLinux release was compiled, or consider recompiling the release to add it: <br />
<br />
ls /usr/share/fonts/encodings<br />
<br />
MiniDLNA lists ''Movies'' and ''Photos'' by file name in its DB, and ''Music'' entries by [http://id3lib.sourceforge.net/id3/ ID3 tags] instead of file names. If Music collection was not tagged in UTF8 but in a local charset, MiniDLNA might not identify and transcode it correctly to UTF8 for display in media players, or the original tags ''codepage(s)'' may be absent in your system, so the tags will not be readable even when media file names are. In this case consider re-tagging your collection to ''UTF-16BE'' or ''UTF-8'' encoding with an '''ID3 Tag Converter'''.<br />
<br />
Picking the "right" ''file system'' for your ''Media_Collection'' is a trade-off: XFS and EXT4 show fast read/write for HDs and lower CPU load critical for small [https://archlinuxarm.org/platforms/armv5/pogoplug-series-4 Plug Computers] with attached storage. NTFS is most compatible with Windows when plugging a drive directly for faster copy, while network file systems like Samba, NFS or iSCSI allow import to Windows any Linux FS with slower data copy. As file fragmentation affects playback, store your Movies on a non-system drive formatted in XFS (prevents fragments), NTFS (fragment resistant and easy to defrag), or EXT4 (uses large file ''extents''), and [https://trac.transmissionbt.com/ticket/849 avoid] EXT3 or [[wikipedia:File_Allocation_Table#Fragmentation|less]] resistant FAT32. For smaller Flash drives with seldom fragmented Music and Photo files, VFAT (FAT32) and EXT4 show faster writes with less CPU load, but EXT4 may affect memory wear due to journaling, and less compatible with media players. Proper drive partitioning, block [https://lwn.net/Articles/428584/ alignment] and ''mount options'' (i.e. ''async,noatime''...- choice depends on file system and memory type) can greatly [https://linux-howto-guide.blogspot.ca/2009/10/increase-usb-flash-drive-write-speed.html accelerate] flash and HD drive speed among other [[wikipedia:Logical_Disk_Manager#Advantages_of_using_a_1-MiB_alignment_boundary|advantages]].<br />
<br />
=== Media Handling ===<br />
<br />
MiniDLNA is aimed for small devices, so does not generate movie thumbnails to lower CPU load and DB built time. It uses either thumbs in the same folder with movie if any, or extracts them where present from media containers like MP4 or MKV with embedded Album Art tags, but not AVI. One can add thumbs (JPG 160x160 pxl or less) to media folders with a '''Thumbnail Maker''', and miniDLNA will link them to media files after rescan. Larger thumbs will be resized and stored in ''Media_DB'' that slows scan. At one movie per folder, follow thumb naming rules in ''minidlna.conf''. For multiple show episodes per folder, each thumb name should match its episode name without ext. (''<file>.cover.jpg'' or ''<file>.jpg''). To handle ''MS Album Art'' thumb names with GUID, add * to the end ''"AlbumArt_{*".jpg'' . MiniDLNA will list on screen only chosen media type (i.e. Movies), but will not other files in the same folder.<br />
<br />
When viewing photos, progressive and/or lossless compression JPG may not be supported by your player via DLNA. Also resize photos to "suggested photo size" by the player's docs for problem free image slideshow. DLNA spec restricts image type to JPG or PNG, and max size to 4096 x 4096 pixels - and that is if the DLNA server implementation supports the LARGE format. The next size limit down (MEDIUM) is 1024 x 768, so resizing may help to show photos correctly.<br />
<br />
To decrease system load, MiniDLNA does not transcode on the fly unsupported media files into supported by your player formats. When building ''Media_DB'', it might not correctly identify whether certain formats are supported by your player, which may play via UPnP a broader formats choice. DLNA standard is quite limiting UPnP subset in media containers and codec ''profiles'' allowed. If you do not see on TV screen or cannot play some media files listed in ''Media_DB'', check if your HD started spinning or try connecting to your media player via USB for their playback. MiniDLNA might not support choosing audio tracks, subtitles, disk chapters, list sorting, and other advanced playback features for your player model.<br />
<br />
== Building a media server ==<br />
<br />
Media served could be based on lightweight and cheap system like development board (Raspberry Pi, CubeBoard, etc.). You do not even need to put X Server on this board.<br />
<br />
=== Automount external drives ===<br />
<br />
This is very useful if you want to automate the server. See [[udisks#Mount helpers]] for more information.<br />
<br />
=== Issues ===<br />
<br />
Media server based on MiniDLNA could face the drive re-scan issue. Ex.: external HDD you have plugged will be scanned each time again and again. This happens due to MiniDLNA removes DB records for unplugged drive. If your drive plugged all the time it is not a problem, but if you have "pluggable" media library on large external drives this could take a big while till you start watching your video.<br />
<br />
One can resolve the rescan issue by using [https://code.google.com/p/minidlna-fastscan this minidlna fork]. It creates a metadata file next to each video file. This can significantly decrease the scan time for large media.<br />
<br />
== Troubleshooting ==<br />
<br />
=== Server not visible on Wireless behind a router ===<br />
<br />
On some network configurations when the machine hosting MiniDLNA server is connected to the router through Ethernet, there may be problems accessing MiniDLNA server on WiFi (same router). To solve this, make sure that "Multicast Isolation" is turned off on the router. For example, on ADB / Pirelli P.RG EA4202N router, connect to the configuration page, then Settings->Bridge and VLAN->Bridge List->click edit on Bridge Ethernet WiFi->set Multicast Isolation to No->Apply.<br />
<br />
=== Media directory not accessible ===<br />
<br />
Please note that the default systemd service file enforces the parameter {{ic|1=ProtectHome=on}}. If you intend to share files that reside within the {{ic|/home/}} file system you may want to lessen that restriction.<br />
You can achieve this by updating the [[systemd]] unit override file.<br />
{{hc|/etc/systemd/system/minidlna.service.d/override.conf|<nowiki>[Service]<br />
ProtectHome=read-only</nowiki>}}<br />
<br />
=== DLNA server stops being visible after some time when being shared on a bridge device ===<br />
<br />
If you are using ReadyMedia to "broadcast" on a bridged device (such as an OpenVPN device bridged to an Ethernet device), the server may stop being seen by the clients after some time (which may vary from a few seconds to half a day).<br />
In order to solve this you need to disable 'multicast snooping'. You can do it instantly with the following command:<br />
<br />
# echo 0 >> /sys/devices/virtual/net/br0/bridge/multicast_snooping<br />
<br />
This should make the server visible to the clients *immediately*, but the change will be lost on reboot.<br />
If this works, you can make it a permanent change by using a systemd service file. Edit the file {{ic|/etc/systemd/system/multicast_snooping.service}} with the following content:<br />
<br />
[Unit]<br />
Description=Set multicast snoop to off<br />
After=network-online.target<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/bin/bash -c "echo 0 >> /sys/devices/virtual/net/br0/bridge/multicast_snooping"<br />
RemainAfterExit=true<br />
ExecStop=/usr/bin/bash -c "echo 1 >> /sys/devices/virtual/net/br0/bridge/multicast_snooping"<br />
StandardOutput=journal<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
Now all you have to do is [[enable]] {{ic|multicast_snooping.service}}.<br />
<br />
This approach should disable multicast_snooping on every boot.</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Lenovo_ThinkPad_T14s_(AMD)_Gen_1&diff=671972Lenovo ThinkPad T14s (AMD) Gen 12021-05-19T19:15:48Z<p>NeoTheFox: /* Backlight */ warn about "quiet" option</p>
<hr />
<div>[[Category:Lenovo]]<br />
[[ja:Lenovo ThinkPad Thinkpad T14s (AMD) Gen 1]]<br />
{{Laptop style|The hardware table needs to be adjusted, there is no function keys table. {{ic|lspci}}/{{ic|lsusb}} dumps do not belong into laptop pages}}<br />
{{Related articles start}}<br />
{{Related|Lenovo ThinkPad T495s}}<br />
{{Related|Lenovo ThinkPad T495}}<br />
{{Related articles end}}<br />
<br />
{| class="wikitable" style="float: right; clear: right"<br />
! Device !! Status<br />
|-<br />
| [[AMDGPU|AMD graphics]] || {{Yes}} <br />
|-<br />
| [[Wireless]] || {{Yes}}<br />
|-<br />
| [[ALSA]] || {{Yes}}<br />
|-<br />
| [[TrackPoint]] || {{Yes}}<br />
|-<br />
| [[Touchpad]] || {{Yes}}<br />
|-<br />
| [[Webcam]] || {{Yes}}<br />
|-<br />
| [[Fingerprint Reader]] || {{Yes}}<br />
|-<br />
| [[ThinkPad mobile Internet|Mobile Broadband]] || [[Xmm7360-pci]]<br />
|-<br />
| [[Bluetooth]] || {{Yes}}<br />
|-<br />
| [[smartcards|Smartcard Reader]] || {{Yes}}<br />
|-<br />
|}<br />
<br />
This article covers the installation and configuration of Arch Linux on a Lenovo Thinkpad T14s (AMD) Gen 1 laptop. Almost everything seems to work pretty much out the box with kernel >=5.8.0. Yet untested: wwan<br />
<br />
For a general overview of laptop-related articles and recommendations, see [[Laptop]].<br />
<br />
== Hardware ==<br />
<br />
AMD Ryzen 7 PRO 4750U Processor<br />
<br />
Using kernel 5.8.0.<br />
<br />
Release Date: 20202<br />
Product Name: Thinkpad T14s (AMD) Gen 1<br />
Machine Type: 20UH, 20UJ<br />
BIOS: 1.30 (Jan 2021)<br />
<br />
{{ic|lspci}} returns something like:<br />
<br />
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Root Complex<br />
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Renoir IOMMU<br />
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge<br />
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge<br />
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge<br />
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge<br />
00:02.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge<br />
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge<br />
00:02.7 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge<br />
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge<br />
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus<br />
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 51)<br />
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)<br />
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0<br />
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1<br />
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2<br />
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3<br />
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4<br />
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5<br />
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6<br />
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7<br />
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983<br />
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0e)<br />
02:00.1 Serial controller: Realtek Semiconductor Co., Ltd. Device 816a (rev 0e)<br />
02:00.2 Serial controller: Realtek Semiconductor Co., Ltd. Device 816b (rev 0e)<br />
02:00.3 IPMI Interface: Realtek Semiconductor Co., Ltd. Device 816c (rev 0e)<br />
02:00.4 USB controller: Realtek Semiconductor Co., Ltd. Device 816d (rev 0e)<br />
03:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)<br />
04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS522A PCI Express Card Reader (rev 01)<br />
05:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)<br />
06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Renoir (rev d1)<br />
06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 1637<br />
06:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor<br />
06:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1<br />
06:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1<br />
06:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor (rev 01)<br />
06:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller<br />
<br />
{{ic|lsusb}} returns something like:<br />
<br />
Bus 007 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub<br />
Bus 006 Device 003: ID 8087:0029 Intel Corp. AX200 Bluetooth<br />
Bus 006 Device 002: ID 06cb:00bd Synaptics, Inc. Prometheus MIS Touch Fingerprint Reader<br />
Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub<br />
Bus 004 Device 002: ID 058f:9540 Alcor Micro Corp. AU9540 Smartcard Reader<br />
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub<br />
Bus 002 Device 002: ID 13d3:5405 IMC Networks Integrated Camera<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
== AMD Graphics ==<br />
<br />
The opensource [[AMDGPU]] drivers work correctly with no special modification.<br />
<br />
{{Warning|1=Hardware Acceleration in latest Chromium seems [https://bbs.archlinux.org/viewtopic.php?id=258797 have rendering issue].}}<br />
<br />
To enable [[Chromium]] hardware accelerated video decoding, install {{Pkg|libva-mesa-driver}}, {{Pkg|vulkan-radeon}} and {{Pkg|mesa-vdpau}}, create a file {{ic|~/.config/chromium-flags.conf}} and paste the following:<br />
<br />
--ignore-gpu-blocklist<br />
--ignore-gpu-blacklist<br />
--use-gl=egl<br />
--enable-gpu-rasterization<br />
--enable-zero-copy<br />
--disable-gpu-driver-bug-workarounds<br />
<br />
Go to {{ic|chrome://flags}} find {{ic|Vulkan}} and enable it.<br />
<br />
Restart [[Chromium]] and then verify {{ic|chrome://gpu}} has the following output:<br />
<br />
Graphics Feature Status<br />
Canvas: Hardware accelerated<br />
Flash: Hardware accelerated<br />
Flash Stage3D: Hardware accelerated<br />
Flash Stage3D Baseline profile: Hardware accelerated<br />
Compositing: Hardware accelerated<br />
Multiple Raster Threads: Enabled<br />
Out-of-process Rasterization: Hardware accelerated<br />
OpenGL: Enabled<br />
Hardware Protected Video Decode: Hardware accelerated<br />
Rasterization: Hardware accelerated on all pages<br />
Skia Renderer: Enabled<br />
Video Decode: Hardware accelerated<br />
Vulkan: Enabled<br />
WebGL: Hardware accelerated<br />
WebGL2: Hardware accelerated <br />
<br />
For more infomation see [[Chromium#Force GPU acceleration]] and [https://chromium.googlesource.com/chromium/src/+/master/docs/gpu/vaapi.md#vaapi-on-linux vaapi on linux].<br />
<br />
{{Warning|Hardware Acceleration is incompatible with [[Chromium]] builtin PDF reader.}}<br />
<br />
== Fingerprint Sensor ==<br />
<br />
The fingerprint sensor works with some recent firmware and software updates.<br />
<br />
# Use [[fwupd]] to install the latest firmware for "Synaptics Prometheus Fingerprint Reader". The update might have to be done manually as the released firmware is in testing; or you could [https://github.com/fwupd/fwupd/wiki/LVFS-Testing-remote enable the testing remote in fwupd] to allow automated upgrade. The relevant firmwares are [https://fwupd.org/lvfs/devices/com.synaptics.prometheus.firmware Prometheus Fingerprint Reader] and [https://fwupd.org/lvfs/devices/com.synaptics.prometheus.config Prometheus Fingerprint Reader Configuration].<br />
# {{pkg|fprintd}} ≥ 1.90.1 and {{pkg|libfprint}} ≥ 1.90.1 are required. Alternatively, use the latest Git master through {{AUR|fprintd-libfprint2}} and {{AUR|libfprint-git}}.<br />
<br />
[[fprint]] has more details on how to setup the fingerprint, for [[PAM]]-based authentication for example.<br />
<br />
If the fingerprint reader is not detected by [[fwupd]] but is detected by {{ic|lsusb}} you will need to reset the fingerprint reader in the Bios.<br />
<br />
== Backlight ==<br />
{{Warning | If {{ic|quiet}} kernel option is set you may have to add {{ic|1=fbcon=nodefer}} or systemd-backlight will fail on boot}} <br />
<br />
Backlight works correctly by manipulating the values, between 0-255, inside {{ic|/sys/class/backlight/amdgpu_bl0/brightness}} or using a backlight managing utility.<br />
<br />
Before kernel 5.8.6, {{ic|systemd-backlight@backlight:acpi_video0.service}} required [[systemd#Using_units|masking]] as it failed on boot.<br />
<br />
== Smartcard Reader ==<br />
<br />
Seems to work and read cards. Following instructions from [[smartcards]].<br />
<br />
== Updating Firmware ==<br />
<br />
Updating the firmware using the [[fwupd]] utility works as long as all the relevant options are enabled in the Bios (Enable Windows Update UEFI Update). A BIOS update Bootable CD iso that is OS agnostic may be downloaded from Lenovo support[https://pcsupport.lenovo.com/].<br />
<br />
{{Merge|Flashing BIOS from Linux#Bootable optical disk emulation|The steps below duplicate content from the aforementioned article/section. Consider removing this section or at the very least, replacing it with a wikilink to the other article/section.}}<br />
<br />
{{Warning|This may brick your device! Be careful!}}<br />
<br />
A charged battery and connected AC power is required to run the update tool successfully. Avoid having any addons/docks plugged in when performing the BIOS update.<br />
<br />
The bootable image needs to be extracted from the iso using {{AUR|geteltorito}} to create a bootable USB.<br />
<br />
1. Use geteltorito to extract bootable image: {{ic|geteltorito -o image_usb.img downloaded_bios_update.iso}}<br />
<br />
2. Copy image to USB drive with [[dd]]: {{ic|sudo dd if᐀image_usb.img of᐀/dev/sdX && sync}}<br />
<br />
3. Reboot. Boot from USB drive.<br />
<br />
4. Perform BIOS update following instructions on screen.<br />
<br />
== Power Mangement ==<br />
<br />
=== Enabling S3 Sleep ===<br />
<br />
For Suspend to work as expected, enter UEFI/BIOS setup, enter the page "Config -> Power" and set the option "Sleep State" to "Linux".<br />
<br />
This will enable S3 sleep as opposed to Windows' "Modern Standby Mode", which Arch Linux does not support at this time.<br />
(Despite the warning in the setting's description, Windows will continue to work; it will just sleep in S3 mode, rather than the "Modern Standby" mode.)<br />
See the [[Lenovo_ThinkPad_X1_Yoga_(Gen_3)#Power_management|wiki page on Thinkpad X1 Yoga]] for some more background on S3 sleep vs Arch Linux.<br />
<br />
== WWAN ==<br />
<br />
The only supported WWAN card is the Fibocom L850-GL, a PCIe card without any Linux drivers (and most probably never will get any, it's an Apple owned chipset now.). All the existing WWAN workaround involve flashing a Sierra Wireless (Qualcomm) WWAN to escape the BIOS whitelist by enumerating slowly. This however does not work in the T14s, the card is entirely disabled and not even visible in ''lsusb''. It is possible that the USB lanes are disabled on the M.2 port.<br />
<br />
There exists an alpha-quality driver for the Fibocom L850-GL [https://github.com/xmm7360/xmm7360-pci here]. The L860 is not proposed in the T14s AMD but is in the T14s Intel and a linux mainline driver is being worked on by Lenovo for it.<br />
<br />
== WiFi ==<br />
<br />
The onboard wireless card is Intel AX200, and it may have microcode issues when used as-is. A possible fix for WiFi disconnects is turning on iwlwifi antenna agreggation on by creating a [[modprobe]] config<br />
<br />
{{hc|/etc/modprobe.d/iwlwifi.conf|2=<br />
options iwlwifi 11n_disable=8<br />
}}<br />
<br />
and rebooting. Look in [[Network configuration/Wireless#iwlwifi]] for details.<br />
<br />
== Install videos ==<br />
<br />
* [https://www.youtube.com/playlist?list=PLiKgVPlhUNuy_AnTdTxCVGsMgidyGuCcQ T14s Arch video playlist]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Talk:Lenovo_ThinkPad_T14s_(AMD)_Gen_1&diff=664363Talk:Lenovo ThinkPad T14s (AMD) Gen 12021-04-26T10:25:02Z<p>NeoTheFox: suggestion on merging</p>
<hr />
<div>I suggest to merge [[Lenovo Thinkpad T14 (AMD) Gen 1]] and [[Lenovo Thinkpad T14s (AMD) Gen 1]] into one page because tiny differences between both models are just in chassis dimensions, battery, cooling system, etc.<br />
[[User:Dmnc|Dmnc]] ([[User talk:Dmnc|talk]]) 07:54, 10 March 2021 (UTC)<br />
:It might be better to merge this article with [[Lenovo ThinkPad X13 Gen 1 (AMD)]], as these two laptops share the same exact motherboard, while regular T14 has a different one. [[User:NeoTheFox|NeoTheFox]] ([[User talk:NeoTheFox|talk]]) 10:24, 26 April 2021 (UTC)</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Lenovo_ThinkPad_T14s_(AMD)_Gen_1&diff=634722Lenovo ThinkPad T14s (AMD) Gen 12020-09-08T17:10:07Z<p>NeoTheFox: Added iwlwifi info</p>
<hr />
<div>[[Category:Lenovo]]<br />
[[ja:Lenovo ThinkPad Thinkpad T14s (AMD) Gen 1]]<br />
{{Related articles start}}<br />
{{Related|Lenovo ThinkPad T495s}}<br />
{{Related|Lenovo ThinkPad T495}}<br />
{{Related articles end}}<br />
<br />
{| class="wikitable" style="float: right; clear: right"<br />
| '''Device''' || '''Status''' <br />
|-<br />
| [[AMDGPU|AMD graphics]] || {{Yes}} <br />
|-<br />
| [[Wireless]] || {{Yes}}<br />
|-<br />
| [[ALSA]] || {{Yes}}<br />
|-<br />
| [[TrackPoint]] || {{Yes}}<br />
|-<br />
| [[Touchpad]] || {{Yes}}<br />
|-<br />
| [[Webcam]] || {{Yes}}<br />
|-<br />
| [[Fingerprint Reader]] || {{Yes}}<br />
|-<br />
| [[ThinkPad mobile Internet|Mobile Broadband]] || {{No}}<br />
|-<br />
| [[Bluetooth]] || {{Yes}}<br />
|-<br />
| [[smartcards|Smartcard Reader]] || {{Yes}}<br />
|-<br />
|}<br />
<br />
This article covers the installation and configuration of Arch Linux on a Lenovo Thinkpad T14s (AMD) Gen 1 laptop. Almost everything seems to work pretty much out the box with kernel >=5.8.0. Yet untested: wwan<br />
<br />
For a general overview of laptop-related articles and recommendations, see [[Laptop]].<br />
<br />
== Hardware ==<br />
<br />
AMD Ryzen 7 PRO 4750U Processor<br />
<br />
Using kernel 5.8.0.<br />
<br />
Release Date: 20202<br />
Product Name: Thinkpad T14s (AMD) Gen 1<br />
SKU Number: 20UH, 20UJ (20UHCTO1WW)<br />
BIOS: 1.05<br />
<br />
{{ic|lspci}} returns something like:<br />
<br />
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Root Complex<br />
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Renoir IOMMU<br />
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge<br />
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge<br />
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge<br />
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge<br />
00:02.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge<br />
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge<br />
00:02.7 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge<br />
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge<br />
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus<br />
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 51)<br />
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)<br />
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0<br />
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1<br />
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2<br />
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3<br />
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4<br />
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5<br />
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6<br />
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7<br />
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983<br />
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0e)<br />
02:00.1 Serial controller: Realtek Semiconductor Co., Ltd. Device 816a (rev 0e)<br />
02:00.2 Serial controller: Realtek Semiconductor Co., Ltd. Device 816b (rev 0e)<br />
02:00.3 IPMI Interface: Realtek Semiconductor Co., Ltd. Device 816c (rev 0e)<br />
02:00.4 USB controller: Realtek Semiconductor Co., Ltd. Device 816d (rev 0e)<br />
03:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)<br />
04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS522A PCI Express Card Reader (rev 01)<br />
05:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)<br />
06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Renoir (rev d1)<br />
06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 1637<br />
06:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor<br />
06:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1<br />
06:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1<br />
06:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor (rev 01)<br />
06:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller<br />
<br />
<br />
{{ic|lsusb}} returns something like:<br />
<br />
Bus 007 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub<br />
Bus 006 Device 003: ID 8087:0029 Intel Corp. AX200 Bluetooth<br />
Bus 006 Device 002: ID 06cb:00bd Synaptics, Inc. Prometheus MIS Touch Fingerprint Reader<br />
Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub<br />
Bus 004 Device 002: ID 058f:9540 Alcor Micro Corp. AU9540 Smartcard Reader<br />
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub<br />
Bus 002 Device 002: ID 13d3:5405 IMC Networks Integrated Camera<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
== AMD Graphics ==<br />
The opensource [[AMDGPU]] drivers work correctly with no special modification.<br />
<br />
To enable [[Chromium]] hardware accelerated video decoding, install {{Pkg|libva-mesa-driver}}, {{Pkg|vulkan-radeon}} and {{Pkg|mesa-vdpau}}, create a file {{ic|~/.config/chromium-flags.conf}} and paste the following:<br />
<br />
--ignore-gpu-blocklist<br />
--ignore-gpu-blacklist<br />
--use-gl=egl<br />
--enable-gpu-rasterization<br />
--enable-zero-copy<br />
--disable-gpu-driver-bug-workarounds<br />
<br />
Go to {{ic|chrome://flags}} find {{ic|Vulkan}} and enable it.<br />
<br />
Restart [[Chromium]] and then verify {{ic|chrome://gpu}} has the following output:<br />
<br />
Graphics Feature Status<br />
Canvas: Hardware accelerated<br />
Flash: Hardware accelerated<br />
Flash Stage3D: Hardware accelerated<br />
Flash Stage3D Baseline profile: Hardware accelerated<br />
Compositing: Hardware accelerated<br />
Multiple Raster Threads: Enabled<br />
Out-of-process Rasterization: Hardware accelerated<br />
OpenGL: Enabled<br />
Hardware Protected Video Decode: Hardware accelerated<br />
Rasterization: Hardware accelerated on all pages<br />
Skia Renderer: Enabled<br />
Video Decode: Hardware accelerated<br />
Vulkan: Enabled<br />
WebGL: Hardware accelerated<br />
WebGL2: Hardware accelerated <br />
<br />
For more infomation see [[Chromium#Force_GPU_acceleration]] and [https://chromium.googlesource.com/chromium/src/+/master/docs/gpu/vaapi.md#vaapi-on-linux vaapi on linux].<br />
<br />
{{Warning|Hardware Acceleration is incompatible with [[Chromium]] builtin PDF reader.}}<br />
<br />
== Fingerprint Sensor ==<br />
<br />
The fingerprint sensor works with some recent firmware and software updates.<br />
<br />
# Use [[fwupd]] to install the latest firmware for "Synaptics Prometheus Fingerprint Reader". The update might have to be done manually as the released firmware is in testing; or you could [https://github.com/fwupd/fwupd/wiki/LVFS-Testing-remote enable the testing remote in fwupd] to allow automated upgrade. The relevant firmwares are [https://fwupd.org/lvfs/devices/com.synaptics.prometheus.firmware Prometheus Fingerprint Reader] and [https://fwupd.org/lvfs/devices/com.synaptics.prometheus.config Prometheus Fingerprint Reader Configuration].<br />
# {{pkg|fprintd}} ≥ 1.90.1 and {{pkg|libfprint}} ≥ 1.90.1 are required. Alternatively, use the latest Git master through {{AUR|fprintd-libfprint2}} and {{AUR|libfprint-git}}.<br />
<br />
[[fprint]] has more details on how to setup the fingerprint, for [[PAM]]-based authentication for example.<br />
<br />
If the fingerprint reader isn't detected by [[fwupd]] but is detected by {{ic|lsusb}} you will need to reset the fingerprint reader in the Bios.<br />
<br />
== Backlight ==<br />
Backlight works correctly by manipulating the values, between 0-255, inside {{ic|/sys/class/backlight/amdgpu_bl0/brightness}} or using a backlight managing utility.<br />
<br />
Before kernel 5.8.6, {{ic|systemd-backlight@backlight:acpi_video0.service}} required [[systemd#Using_units|masking]] as it failed on boot.<br />
<br />
== Smartcard Reader ==<br />
Seems to work and read cards. Following instructions from [[smartcards]].<br />
<br />
== Updating Firmware ==<br />
Updating the firmware using the [[fwupd]] utility works as long as all the relevant options are enabled in the Bios (Enable Windows Update UEFI Update). A BIOS update Bootable CD iso that is OS agnostic may be downloaded from Lenovo support[https://pcsupport.lenovo.com/].<br />
<br />
{{Warning|This may brick your device! Be careful!}}<br />
<br />
A charged battery and connected ac power is required to run the update tool successfully. Avoid having any addons/docks plugged in when performing the BIOS update.<br />
<br />
The bootable image needs to be extracted from the iso using {{AUR|geteltorito}} to create a bootable USB.<br />
<br />
1. Use geteltorito to extract bootable image: {{ic|geteltorito -o image_usb.img downloaded_bios_update.iso}}<br />
<br />
2. Copy image to USB drive with dd: {{ic|sudo dd if᐀image_usb.img of᐀/dev/sdX && sync}}<br />
<br />
3. Reboot. Boot from USB drive.<br />
<br />
4. Perform BIOS update following instructions on screen.<br />
<br />
== WWAN ==<br />
<br />
The only supported WWAN card is the Fibocom L850-GL, a PCIe card without any Linux drivers (and most probably never will get any, it's an Apple owned chipset now.). All the existing WWAN workaround involve flashing a Sierra Wireless (Qualcomm) WWAN to escape the BIOS whitelist by enumerating slowly. This however doesn't work in the T14s, the card is entirely disabled and not even visible in ''lsusb''. It is possible that the USB lanes are disabled on the M.2 port.<br />
<br />
There exists an alpha-quality driver for the Fibocom L850-GL [https://github.com/xmm7360/xmm7360-pci here]. The L860 is not proposed in the T14s AMD but is in the T14s Intel and a linux mainline driver is being worked on by Lenovo for it.<br />
<br />
== WiFi ==<br />
<br />
The onboard wireless card is Intel AX200, and it may have microcode issues when used as-is. A possible fix for WiFi disconnects is turning on iwlwifi antenna agreggation on by creating a [[modprobe]] config<br />
<br />
{{hc|/etc/modprobe.d/iwlwifi.conf|2=<br />
options iwlwifi 11n_disable=8<br />
}}<br />
<br />
and rebooting. Look in [[Network_configuration/Wireless#iwlwifi]] for details.<br />
<br />
== Install videos ==<br />
<br />
* [https://www.youtube.com/playlist?list=PLiKgVPlhUNuy_AnTdTxCVGsMgidyGuCcQ T14s Arch video playlist]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Self-encrypting_drives&diff=625584Self-encrypting drives2020-07-16T17:31:14Z<p>NeoTheFox: sedutil-sleep fork for waking up from S3 suspend</p>
<hr />
<div>[[Category:Storage]]<br />
[[Category:Data-at-rest encryption]]<br />
[[ja:自己暗号化ドライブ]]<br />
{{Related articles start}}<br />
{{Related|Data-at-rest encryption}}<br />
{{Related|Trusted Platform Module}}<br />
{{Related articles end}}<br />
<br />
{{Style|Don't use "/" instead of proper logical conjunctions such as "and" or "or".}}<br />
<br />
[[wikipedia:Hardware-based full disk encryption|Hardware-based full-disk encryption]] (FDE) is now available from many hard disk (HDD) vendors, becoming increasingly common especially for [[solid state drive]]s. The term "self-encrypting drive" (SED) is now common when referring to HDDs or SSDs with built-in full-disk encryption. [[Wikipedia:Opal_Storage_Specification|OPAL]] is a set of specifications for self-encrypting drives developed by the Trusted Computing Group. <br />
<br />
== Overview ==<br />
Many of the self-encrypting SSDs/HDDs available today implement the [https://trustedcomputinggroup.org/wp-content/uploads/TCG_Storage-Opal_SSC_v2.01_rev1.00.pdf OPAL 2.0] and [https://trustedcomputinggroup.org/wp-content/uploads/TCG_Storage-SSC_Enterprise-v1.01_r1.00.pdf Enterprise] standards developed by the Trusted Computing Group (TCG). Enterprise SAS versions of the TCG standard are called &quot;TCG Enterprise&quot; drives. The hardware manufactured according to the standards is [http://www.wavesys.com/self-encrypting-drive-compatibility-list labeled] accordingly. <br />
<br />
Unlocking of the drive can be done during operating system runtime using software utilities, in a pre-boot authentication environment, or with a [[#BIOS based ATA-password]] on power up.<br />
<br />
{{Tip|"Encryption" in the context of this page refers to hardware-based encryption. See [[Data-at-rest encryption#Block device encryption]] and [[Data-at-rest encryption#Stacked filesystem encryption]] for software-based encryption.}}<br />
<br />
Refer to the [[#Advantages]] and [[#Disadvantages]] sections to better understand and decide if hardware-based full-disk encryption is what you want.<br />
<br />
=== Key management technical implementation ===<br />
<br />
{{Tip|This section is important to understand the core concepts behind effective key management, and secure erasure of the disk.}}<br />
<br />
Key management takes place within the disk controller and encryption keys are usually 128 or 256 bit Advanced Encryption Standard (AES).<br />
<br />
{{Expansion|The Opal standards are full of acronyms. The sedutil refers to them in the [https://github.com/Drive-Trust-Alliance/sedutil/wiki/FAQ FAQ] and [https://github.com/Drive-Trust-Alliance/sedutil/wiki/Command-Syntax command syntax]. It would be useful to name the important acronyms (e.g. SID, MSID, locking SP, admin SP) in this section, because they are intrinsic to the key-management defined per standard. (See also [[w:Key escrow]]).}}<br />
<br />
Self-encrypting drives adhering to the TCG OPAL 2.0 standard specification (almost all modern self-encrypting drives) implement key management via an authentication key, and a 2nd-level data encryption key. The data encryption key is the key against which the data is actually encrypted/decrypeted. The authentication key is the user-facing 1st-level password/passphrase which decrypts the data encryption key (which in turn decrypts the data). This approach has specific advantages:<br />
<br />
* Allows the user to change the passphrase ''without'' losing the existing encrypted data on the disk<br />
** This improves security, as it is fast and easy to respond to security threats and revoke a compromised passphrase<br />
* Facilitates near-instant and cryptographically secure full disk erasure.<br />
<br />
For those who are familiar; this concept is similar to the LUKS key management layer often used in a [[dm-crypt]] deployment. Using LUKS, the user can effectively have up to 8 different key-files / passphrases to decrypt the ''encryption key'', which in turn decrypts the underlying data. This approach allows the user to revoke / change these key-files / passphrases as required ''without'' needing to re-encrypt the data, as the 2nd-level encryption key ''is unchanged'' (itself being re-encrypted by the new passphrase).<br />
<br />
In fact, in drives featuring full-disk encryption, data is ''always'' encrypted with the data encryption key when stored to disk, even if there is no password set (e.g. a new drive). Manufacturers do this to make it easier for users who are not able to, or do not wish to enable the security features of the self-encrypting drive. This can be thought of as all drives by default having a zero-length password that transparently encrypts/decrypts the data ''always'' (similar to how passwordless SSH keys provide (somewhat) secure access without user intervention).<br />
<br />
''The key point to note'' is that if at a later stage the user wishes to ''"enable"'' encryption, they can configure the passphrase (authentication key), which will then be used to encrypt the ''existing'' data encryption key (thus prompting for passphrase before decrypting the data encryption key in future). However, as the existing data encryption key ''will not'' be changed (regenerated), this in effect locks the drive, while preserving the existing encrypted data on the disk.<br />
<br />
=== Advantages ===<br />
<br />
* Easier to setup (compared to software-based encryption)<br />
* Notably transparent to the user, except for initial bootup authentication<br />
* Data-at-Rest protection<br />
* Increased performance (CPU is freed up from encryption/decryption calculations)<br />
* The main CPU and RAM are eliminated as possible attack targets<br />
* Optimally fast and [[#Secure disk erasure]] (sanitation) (regardless of disk size)<br />
* Protection from alternative boot methods due to the possibility to encrypt the MBR, rendering the drive inaccessible before pre-boot authentication<br />
<br />
=== Disadvantages ===<br />
<br />
* Constant-power exploits<br />
:Typical self-encrypting drives, once unlocked, will remain unlocked as long as power is provided. This vulnerability can be exploited by means of altering the environment external to the drive, without cutting power, in effect keeping the drive in an unlocked state. For example, it has been shown (by researchers at University of Erlangen-Nuremberg) that it is possible to reboot the computer into an attacker-controlled operating system without cutting power to the drive. The researchers have also demonstrated moving the drive to another computer without cutting power.[https://www1.cs.fau.de/sed]<br />
* Key-in-memory exploits<br />
:When the system is powered down into S3 ("sleep") mode, the drive is powered down, but the drive keeps access to the encryption key in its internal memory (NVRAM) to allow for a resume ("wake"). This is necessary because for system booted with an arbitrary operating system there is no standard mechanism to prompt the user to re-enter the pre-boot decryption passphrase again. An attacker (with physical access to the drive) can leverage this to access the drive. Taking together known exploits the researchers summarize "we were able to break hardware-based full-disk encryption on eleven [of twelve] of those systems provided they were running or in standby mode".[https://www1.cs.fau.de/sed] Note, however, S3 ("sleep") is '''not''' currently supported by sedutil (the current available toolset for managing a TCG OPAL 2.0 self-encrypting drives via Linux)<br />
* Compromised firmware<br />
:The firmware of the drive may be compromised (backdoor) and data sent to it thus potentially compromised (decryptable by the malicious third party in question, provided access to physical drive is achievable). A study demonstrated methods for compromising device firmware, as well as applying invalid passwords to access data on OPAL devices.[https://www.ru.nl/publish/pages/909275/draft-paper_1.pdf] If data is encrypted by the operating system (e.g. dm-crypt), the encryption key is unknown to the compromised drive, thus circumventing this attack vector entirely.<br />
<br />
== Linux support ==<br />
<br />
{{Expansion|1=Kernel 4.11 gained {{ic|BLK_SED_OPAL}} self-encrypting drives support.[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=455a7b238cd6bc68c4a550cbbd37c1e22b64f71c]}}<br />
<br />
''msed'' and ''OpalTool'', the two known Open Source code bases available for self-encrypting drives support on Linux, have both been retired, and their development efforts officially merged to form ''[https://github.com/Drive-Trust-Alliance/sedutil sedutil]'', under the umbrella of The [https://www.drivetrust.com Drive Trust Alliance (DTA)]. [https://github.com/Drive-Trust-Alliance/sedutil sedutil] is &quot;''an Open Source (GPLv3) effort to make Self Encrypting Drive technology freely available to everyone.''&quot;<br />
<br />
[[Install]] the {{AUR|sedutil}} package, which contains the ''sedutil-cli'' tool, and helper scripts to create a custom pre-boot authorization (PBA) image based off the ''current'' OS in use (e.g. for setting a custom console keymap). Alternatively, you can install ''sedutil'' solely for acquiring the ''sedutil-cli'' toolset, but download and use the [https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions precompiled PBA image (for BIOS or UEFI)] provided by the Drive Trust Alliance.<br />
<br />
{{Note|UEFI support currently requires that Secure Boot be turned off.}}<br />
<br />
{{ic|libata.allow_tpm}} '''must''' be set to {{ic|1}} (true) in order to use sedutil. Either add {{ic|1=libata.allow_tpm=1}} to the [[kernel parameters]], or by setting {{ic|/sys/module/libata/parameters/allow_tpm}} to {{ic|1}} on a running system.<br />
<br />
== Encrypting the root (boot) drive ==<br />
{{Out of date|Some of the information in this section has changed, and may not work correctly for sedutil v1.15}}<br />
<br />
These instructions assume you have the ''sedutil-cli'' tool installed (via the [https://aur.archlinux.org/packages/sedutil AUR], or by other means)<br />
<br />
=== Check if your disk supports OPAL ===<br />
<br />
# sedutil-cli --scan<br />
<br />
If you get something like<br />
<br />
Scanning for Opal compliant disks<br />
/dev/sda No LITEONIT LMT-256L9M-11 MSATA 256GB HM8110B<br />
<br />
then your disk doesn't support OPAL. On the contrary, the following output means OPAL is supported:<br />
<br />
/dev/sda 12 Samsung SSD 850 EVO 500GB EMT02B6Q<br />
<br />
Windows version of sedutils output:<br />
<br />
\\.\PhysicalDrive0 12 Samsung SSD 850 PRO 512GB EXM02B6Q<br />
<br />
=== Download (or create) the pre-boot authorization (PBA) image ===<br />
<br />
Download the pre-boot authorization (PBA) image for a [https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions BIOS or UEFI] machine provided by the Drive Trust Alliance.<br />
<br />
{{Note|UEFI support currently requires that Secure Boot be turned off.}}<br />
<br />
<br />
{{Accuracy|These commands do not currently exist in the AUR package. More up-to-date instructions can be found [https://github.com/Drive-Trust-Alliance/sedutil/wiki/Encrypting-your-drive in the official documentation].}}<br />
<br />
Alternatively, you can create your own PBA image using the supplied helpers:<br />
<br />
# mklinuxpba-efi<br />
<br />
to create an EFI image (/boot/linuxpba-efi.diskimg) and<br />
<br />
# mklinuxpba-bios<br />
<br />
to create a BIOS image (/boot/linuxpba.diskimg).<br />
<br />
=== Test the PBA on your machine (optional) ===<br />
<br />
Refer to the [https://github.com/Drive-Trust-Alliance/sedutil/wiki/Test-the-PBA official docs].<br />
<br />
Don't expect to get a list of your OPAL disks. If you try the PBA from a USB stick and your SSD disk is still not activated for OPAL locking (as it is recommended before the PBA has been successfully tested) you will get an error message including "INVALID PARAMETER" (see [https://github.com/Drive-Trust-Alliance/sedutil/issues/73 this issue]). But this shows that the PBA is actually working and finding your disk. The Wiki is outdated in this regard.<br />
<br />
=== Prepare and test the rescue image (optional) ===<br />
<br />
Refer to the [https://github.com/Drive-Trust-Alliance/sedutil/wiki/Test-the-Rescue-system official docs].<br />
<br />
=== Set up the drive ===<br />
<br />
Decompress the PBA (if required):<br />
$ gunzip pba.img.gz<br />
<br />
Use the output of {{ic|lsblk --fs}} to help identify the correct drive.<br />
<br />
# sedutil-cli --initialsetup ''password drive''<br />
# sedutil-cli --loadPBAimage ''password pba_file drive''<br />
# sedutil-cli --setMBREnable on ''password drive''<br />
<br />
=== Enable locking ===<br />
<br />
# sedutil-cli --enableLockingRange 0 ''password'' ''drive''<br />
<br />
Power off the computer to lock the drive.<br />
<br />
When the computer is next powered on, the PBA should ask for your password; then unlock the drive and chain-load the decrypted OS.<br />
<br />
=== Accessing the drive from a live distro ===<br />
<br />
The easiest way is to boot the encrypted SSD first, in order to run the shadow MBR. Then press the key that prompts the boot menu and boot whatever device you prefer. Such a way the SED will be completely transparent.<br />
<br />
Another way is to directly boot into the live distro and use sedutil to unlock the SSD:<br />
<br />
# sedutil-cli --setlockingrange 0 rw ''password drive''<br />
# sedutil-cli --setmbrdone on ''password drive''<br />
# partprobe ''drive''<br />
<br />
{{ic|libata.allow_tpm}} '''must''' be set to {{ic|1}} (true) in order to use sedutil. Either add {{ic|1=libata.allow_tpm=1}} to the [[kernel parameters]], or by setting {{ic|/sys/module/libata/parameters/allow_tpm}} to {{ic|1}} on a running system.<br />
<br />
=== Disable locking ===<br />
<br />
If you want to turn off Locking and the PBA:<br />
<br />
# sedutil-cli --disableLockingRange 0 ''password drive''<br />
# sedutil-cli --setMBREnable off ''password drive''<br />
=== Re-enable locking and the PBA ===<br />
<br />
You can later re-enable locking and the PBA using this command sequence<br />
<br />
# sedutil-cli --enableLockingRange 0 ''password drive''<br />
# sedutil-cli --setMBRDone on ''password drive''<br />
# sedutil-cli --setMBREnable on ''password drive''<br />
<br />
== Encrypting a non-root drive ==<br />
<br />
{{Style|undefined term: non-root drive}}<br />
<br />
A non-root drive does not require loading a PBA. So, activating the encryption is as simple as running:<br />
<br />
# sedutil-cli --initialsetup ''password drive''<br />
<br />
== Changing the passphrase ==<br />
<br />
Changing the passphrase does ''not'' lose existing data on the drive, and does not require re-encryption of data.<br />
<br />
# sedutil-cli --setSIDPassword ''password newpassword device''<br />
# sedutil-cli --setAdmin1Pwd ''password newpassword device''<br />
<br />
Read the [[#Key management technical implementation]] section above to learn about how this is implemented securely within the drive, and why it is possible to change the passphrase ''without'' losing the existing encrypted data on the drive.<br />
<br />
== Waking up from suspend ==<br />
<br />
Suspending the system results in a crash by default, because power is being cut from the drive and that causes it to lock itself. In order to wake up from suspend the kernel should know a hashed password for the disk. This functionality is in-kernel since 4.11, but in it's only available via a fork of sedutil, {{AUR|sedutil-sleep-git}}.<br />
<br />
Generate a hashed password<br />
<br />
# sedutil-sleep --printPasswordHash ''password device''<br />
<br />
Then create a [[systemd]] service, inserting hashes from before<br />
<br />
{{hc|/etc/systemd/system/sedutil-sleep.service|<nowiki><br />
[Unit]<br />
Description=sedutil-sleep<br />
<br />
[Service]<br />
Type=oneshot<br />
#Add as many ExecStart lines as needed here<br />
ExecStart=-+/usr/bin/sedutil-sleep -n -x --prepareForS3Sleep 0 hash /dev/sda<br />
#ExecStart=-+/usr/bin/sedutil-sleep -n -x --prepareForS3Sleep 0 hash2 /dev/sdb<br />
ReainAfterExit=true<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
</nowiki>}}<br />
<br />
== Secure disk erasure ==<br />
<br />
Whole disk erasure is very fast, and remarkably simple for a self-encrypting drive. Simply passing a cryptographic disk erasure (or crypto erase) command (after providing the correct authentication credentials) will have the drive self-generate a new random data encryption key internally. This will permanently discard the old key, thus rendering the encrypted data irrevocably un-decryptable. The drive will now be in a 'new drive' state.<br />
<br />
Read the [[#Key management technical implementation]] section above to learn more about how this is implemented securely within the drive.<br />
<br />
== BIOS based ATA-password ==<br />
<br />
Previous to the industry's TCG OPAL 2.0 standard initiative, the relevant [[w:Parallel_ATA#ATA_standards_versions.2C_transfer_rates.2C_and_features|ATA]] standard defined an "ATA security feature set" for full-disk encryption using self-encrypting drives. This relies on the PC (not SSD/HDD) BIOS to feature an unlocking mechanism utilizing the BIOS to setup the user's encryption password/passphrase. This legacy BIOS-based (ATA) method was considered more unreliable to setup and prone to error with regard to interoperability between PC vendors.[http://www.t13.org/documents/UploadedDocuments/docs2006/e05179r4-ACS-SecurityClarifications.pdf] Typical errors include, for example, inabilities to unlock a device once it is plugged into a system from a different hardware vendor. Hence, the availability of BIOS support for the legacy password mechanism decreases in availability, particularly for consumer hardware. <br />
<br />
See [[Solid state drive#Security]] for more information.<br />
<br />
== See also ==<br />
<br />
* [https://github.com/Drive-Trust-Alliance/sedutil sedutil (github)]<br />
* [https://github.com/Drive-Trust-Alliance/sedutil/wiki sedutil official docs]<br />
* [https://github.com/Drive-Trust-Alliance/sedutil/wiki/Command-Syntax sedutil-cli commands usage]<br />
* [https://github.com/Drive-Trust-Alliance/sedutil/issues/90 sedutil-sleep]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=615298PCI passthrough via OVMF2020-05-23T19:56:44Z<p>NeoTheFox: /* Configuring the host */ updated user service for Scream</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
[[zh-hans:PCI passthrough via OVMF]]<br />
{{Related articles start}}<br />
{{Related|Intel GVT-g}}<br />
{{Related articles end}}<br />
<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of [[QEMU]], it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=873&0_VTD=True List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Note|<br />
* IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
* VT-d stands for ''Intel Virtualization Technology for Directed I/O'' and should not be confused with VT-x ''Intel Virtualization Technology''. VT-x allows one hardware platform to function as multiple “virtual” platforms while VT-d improves security and reliability of the systems and also improves performance of I/O devices in virtualized environments.<br />
}}<br />
<br />
Using IOMMU opens to features like PCI passthrough and memory protection from faulty or malicious devices, see [[Wikipedia:Input-output memory management unit#Advantages]] and [https://www.quora.com/Memory-Management-computer-programming/Could-you-explain-IOMMU-in-plain-English Memory Management (computer programming): Could you explain IOMMU in plain English?].<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is supported by the CPU and enabled in the BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
Enable IOMMU support by setting the correct [[kernel parameter]] depending on the type of CPU in use:<br />
<br />
* For Intel CPUs (VT-d) set {{ic|1=intel_iommu=on}} <br />
* For AMD CPUs (AMD-Vi) set {{ic|1=amd_iommu=on}}<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -i -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for g in /sys/kernel/iommu_groups/*; do<br />
echo "IOMMU Group ${g##*/}:"<br />
for d in $g/devices/*; do<br />
echo -e "\t$(lspci -nns ${d##*/})"<br />
done;<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 1:<br />
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
IOMMU Group 2:<br />
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:0e31] (rev 04)<br />
IOMMU Group 4:<br />
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:0e2d] (rev 04)<br />
IOMMU Group 10:<br />
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:0e26] (rev 04)<br />
IOMMU Group 13:<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1:<br />
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in to your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the preferred method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{Warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this - your motherboard should be set to display using the host GPU.}}<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub did, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use.<br />
<br />
=== Binding vfio-pci via device ID ===<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13:<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|<br />
* You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.<br />
* If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.<br />
}}<br />
<br />
Two methods exist for providing the device IDs. Specifying them via [[kernel parameters]] has the advantage of being able to easily edit, remove, or undo any breaking changes via your boot loader:<br />
<br />
vfio-pci.ids=10de:13c2,10de:0fbb<br />
<br />
Alternatively, the IDs may be added to a modprobe conf file. Since these conf files are embedded in the initramfs image, any changes require regenerating a new image each time:<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
=== Loading vfio-pci early ===<br />
<br />
Since Arch's {{Pkg|linux}} has vfio-pci built as a module, we need to force it to load early before the graphics drivers have a chance to bind to the card. To ensure that, add {{ic|vfio_pci}}, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio_pci vfio vfio_iommu_type1 vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]].<br />
<br />
=== Verifying that the configuration worked ===<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it is possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
Install {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|edk2-ovmf}}, and {{Pkg|virt-manager}}.<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
You may also need to [https://wiki.libvirt.org/page/Networking#NAT_forwarding_.28aka_.22virtual_networks.22.29 activate the default libvirt network].<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self-explanatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the {{ic|libvirt}} [[user group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps:<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that:<br />
** Your hypervisor is running as a system session and not a user session. This can be verified [https://i.ibb.co/N1XZCdp/Deepin-Screenshot-select-area-20190125113216.png by clicking, then hovering] over the session in virt-manager. If you are accidentally running it as a user session, you must open a new connection by clicking "File" > "Add Connection..", then select the option from the drop-down menu station "QEMU/KVM" and not "QEMU/KVM user session".<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, it's easier to setup [[#Virtio disk]] before installing<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine. It is possible you will be dropped into the UEFI menu instead of starting the installation upon powering your VM for the first time. Sometimes the correct ISO file was not automatically detected and you will need to manually specify the drive to boot. By typing exit and navigating to "boot manager" you will enter a menu that allows you to choose between devices.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it is now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Passing keyboard/mouse via Evdev ===<br />
<br />
If you do not have a spare mouse or keyboard to dedicate to your guest, and you do not want to suffer from the video overhead of Spice, you can setup evdev to swap control of your mouse and keyboard between the host and guest on the fly.<br />
<br />
First, modify the libvirt configuration. If you encounter {{ic|failed to get domain [vmname]}} error, add {{ic|-c qemu:///system}} arguments to {{ic|virsh}}.<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Next, find your keyboard and mouse devices in {{ic|/dev/input/by-id/}}. You may find multiple devices associated to your mouse or keyboard, so try {{ic|cat /dev/input/by-id/''device_id''}} and either hit some keys on the keyboard or wiggle your mouse to see if input comes through, if so you have got the right device. Now add those devices to your configuration right before the closing </domain> tag:<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<qemu:commandline><br />
<qemu:arg value='-object'/><br />
<qemu:arg value='input-linux,id=mouse1,evdev=/dev/input/by-id/MOUSE_NAME'/><br />
<qemu:arg value='-object'/><br />
<qemu:arg value='input-linux,id=kbd1,evdev=/dev/input/by-id/KEYBOARD_NAME,grab_all=on,repeat=on'/><br />
</qemu:commandline><br />
...<br />
</nowiki>}}<br />
<br />
Replace {{ic|MOUSE_NAME}} and {{ic|KEYBOARD_NAME}} with your device id. You will also need to include these devices in your qemu config, and setting the user and group to one that has access to your input devices:<br />
<br />
{{hc|/etc/libvirt/qemu.conf|<nowiki><br />
...<br />
user = "<your_user>"<br />
group = "kvm"<br />
...<br />
cgroup_device_acl = [<br />
"/dev/kvm",<br />
"/dev/input/by-id/KEYBOARD_NAME",<br />
"/dev/input/by-id/MOUSE_NAME",<br />
"/dev/null", "/dev/full", "/dev/zero",<br />
"/dev/random", "/dev/urandom",<br />
"/dev/ptmx", "/dev/kvm", "/dev/kqemu",<br />
"/dev/rtc","/dev/hpet", "/dev/sev"<br />
]<br />
...<br />
</nowiki>}}<br />
<br />
Then ensure that the user you provided has access to the {{ic|kvm}} and {{ic|input}} [[user group]]s. [[Restart]] {{ic|libvirtd.service}}. Now you can startup the guest OS and test swapping control of your mouse and keyboard between the host and guest by pressing both the left and right control keys at the same time.<br />
<br />
You may also consider switching from PS/2 to Virtio inputs in your configurations:<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<input type='mouse' bus='virtio'><br />
<address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/><br />
</input><br />
<input type='keyboard' bus='virtio'><br />
<address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/><br />
</input><br />
<input type='mouse' bus='ps2'/><br />
<input type='keyboard' bus='ps2'/><br />
...<br />
</nowiki>}}<br />
<br />
Next startup the guest OS and install the virtIO drivers for those devices.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate Linux/Windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. As such, the local CPU cache benefits (L1/L2) are lost each time the host scheduler reschedules the virtual CPU thread on a different physical CPU. This can noticeably harm performance on the guest. CPU pinning aims to resolve this by limiting which physical CPUs the virtual CPUs are allowed to run on. The ideal setup is a one to one mapping such that the virtual CPU cores match physical CPU cores while taking hyperthreading/SMT into account.<br />
<br />
{{Note|For certain users enabling CPU pinning may introduce stuttering and short hangs, especially with the MuQSS scheduler (present in linux-ck and linux-zen kernels). You might want to try disabling pinning first if you experience similar issues, which effectively trades maximum performance for responsiveness at all times.}}<br />
<br />
==== CPU topology ====<br />
<br />
Most modern CPUs support hardware multitasking, also known as hyper-threading on Intel CPUs or SMT on AMD CPUs. Hyper-threading/SMT is simply a very efficient way of running two threads on one CPU core at any given time. You will want to take into consideration that the CPU pinning you choose will greatly depend on what you do with your host while your VM is running.<br />
<br />
To find the topology for your CPU run {{ic|1=lscpu -e}}:<br />
<br />
{{Note|Pay special attention to the 4th column '''"CORE"''' as this shows the association of the Physical/Logical CPU cores}}<br />
<br />
{{ic|lscpu -e}} on a 6c/12t Ryzen 5 1600:<br />
<br />
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ<br />
0 0 0 0 0:0:0:0 yes 3800.0000 1550.0000<br />
1 0 0 0 0:0:0:0 yes 3800.0000 1550.0000<br />
2 0 0 1 1:1:1:0 yes 3800.0000 1550.0000<br />
3 0 0 1 1:1:1:0 yes 3800.0000 1550.0000<br />
4 0 0 2 2:2:2:0 yes 3800.0000 1550.0000<br />
5 0 0 2 2:2:2:0 yes 3800.0000 1550.0000<br />
6 0 0 3 3:3:3:1 yes 3800.0000 1550.0000<br />
7 0 0 3 3:3:3:1 yes 3800.0000 1550.0000<br />
8 0 0 4 4:4:4:1 yes 3800.0000 1550.0000<br />
9 0 0 4 4:4:4:1 yes 3800.0000 1550.0000<br />
10 0 0 5 5:5:5:1 yes 3800.0000 1550.0000<br />
11 0 0 5 5:5:5:1 yes 3800.0000 1550.0000<br />
<br />
{{Note|Ryzen 3000 ComboPi AGESA changes topology to match Intel example, even on prior generation CPUs. Above valid only on older AGESA. }}<br />
<br />
{{ic|lscpu -e}} on a 6c/12t Intel 8700k:<br />
<br />
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ<br />
0 0 0 0 0:0:0:0 yes 4600.0000 800.0000<br />
1 0 0 1 1:1:1:0 yes 4600.0000 800.0000<br />
2 0 0 2 2:2:2:0 yes 4600.0000 800.0000<br />
3 0 0 3 3:3:3:0 yes 4600.0000 800.0000<br />
4 0 0 4 4:4:4:0 yes 4600.0000 800.0000<br />
5 0 0 5 5:5:5:0 yes 4600.0000 800.0000<br />
6 0 0 0 0:0:0:0 yes 4600.0000 800.0000<br />
7 0 0 1 1:1:1:0 yes 4600.0000 800.0000<br />
8 0 0 2 2:2:2:0 yes 4600.0000 800.0000<br />
9 0 0 3 3:3:3:0 yes 4600.0000 800.0000<br />
10 0 0 4 4:4:4:0 yes 4600.0000 800.0000<br />
11 0 0 5 5:5:5:0 yes 4600.0000 800.0000<br />
<br />
As we see above, with AMD '''Core 0''' is sequential with '''CPU 0 & 1''', whereas Intel places '''Core 0''' on '''CPU 0 & 6'''.<br />
<br />
If you do not need all cores for the guest, it would then be preferable to leave at the very least one core for the host. Choosing which cores one to use for the host or guest should be based on the specific hardware characteristics of your CPU, however '''Core 0''' is a good choice for the host in most cases. If any cores are reserved for the host, it is recommended to pin the emulator and iothreads, if used, to the host cores rather than the VCPUs. This may improve performance and reduce latency for the guest since those threads will not pollute the cache or contend for scheduling with the guest VCPU threads. If all cores are passed to the guest, there is no need or benefit to pinning the emulator or iothreads.<br />
<br />
==== XML examples ====<br />
<br />
{{Note|Do not use the '''iothread''' lines from the XML examples shown below if you have not added an '''iothread''' to your disk controller. '''iothread''''s only work on '''virtio-scsi''' or '''virtio-blk''' devices.}}<br />
<br />
===== 4c/1t CPU w/o Hyperthreading Example =====<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='0'/><br />
<vcpupin vcpu='1' cpuset='1'/><br />
<vcpupin vcpu='2' cpuset='2'/><br />
<vcpupin vcpu='3' cpuset='3'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
===== 4c/2t Intel/AMD CPU example (after ComboPI AGESA bios update) =====<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>8</vcpu><br />
<iothreads>1</iothreads><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='8'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='9'/><br />
<vcpupin vcpu='4' cpuset='4'/><br />
<vcpupin vcpu='5' cpuset='10'/><br />
<vcpupin vcpu='6' cpuset='5'/><br />
<vcpupin vcpu='7' cpuset='11'/><br />
<emulatorpin cpuset='0,6'/><br />
<iothreadpin iothread='1' cpuset='0,6'/><br />
</cputune><br />
...<br />
<topology sockets='1' cores='4' threads='2'/><br />
...<br />
</nowiki>}}<br />
<br />
===== 4c/2t AMD CPU example (Before ComboPi AGESA bios update) =====<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>8</vcpu><br />
<iothreads>1</iothreads><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='3'/><br />
<vcpupin vcpu='2' cpuset='4'/><br />
<vcpupin vcpu='3' cpuset='5'/><br />
<vcpupin vcpu='4' cpuset='6'/><br />
<vcpupin vcpu='5' cpuset='7'/><br />
<vcpupin vcpu='6' cpuset='8'/><br />
<vcpupin vcpu='7' cpuset='9'/><br />
<emulatorpin cpuset='0-1'/><br />
<iothreadpin iothread='1' cpuset='0-1'/><br />
</cputune><br />
...<br />
<topology sockets='1' cores='4' threads='2'/><br />
...<br />
</nowiki>}}<br />
<br />
{{Note|If further CPU isolation is needed, consider using the '''isolcpus''' kernel command-line parameter on the unused physical/logical cores.}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, you may want to pin your VM threads across all of your cores, so that the VM can fully take advantage of the spare CPU time the host has available. Be aware that pinning all physical and logical cores of your CPU could induce latency in the guest VM.<br />
<br />
=== Huge memory pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information across multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4 GiB of memory divided into 4 KiB pages (which is the default size for normal pages) for a total of 1.04 million pages, meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession.<br />
<br />
==== Transparent huge pages ====<br />
<br />
QEMU will use 2MiB sized transparent huge pages automatically without any explicit configuration in QEMU or Libvirt, subject to some important caveats. When using VFIO the pages are locked in at boot time and transparent huge pages are allocated up front when the VM first boots. If the kernel memory is highly fragmented, or the VM is using a majority of the remaining free memory, it is likely that the kernel will not have enough 2MiB pages to fully satisfy the allocation. In such a case, it silently fails by using a mix of 2MiB and 4KiB pages. Since the pages are locked in VFIO mode, the kernel will not be able to convert those 4KiB pages to huge after the VM starts either. The number of available 2MiB huge pages available to THP is the same as via the [[#Dynamic huge pages]] mechanism described in the following sections.<br />
<br />
To check how much memory THP is using globally:<br />
<br />
{{hc|$ grep AnonHugePages /proc/meminfo|<br />
AnonHugePages: 8091648 kB<br />
}}<br />
<br />
To check a specific QEMU instance. QEMU's PID must be substituted in the grep command:<br />
<br />
{{hc|$ grep -P 'AnonHugePages:\s+(?!0)\d+' /proc/[PID]/smaps|<br />
AnonHugePages: 8087552 kB<br />
}}<br />
<br />
In this example, the VM was allocated 8388608KiB of memory, but only 8087552KiB was available via THP. The remaining 301056KiB are allocated as 4KiB pages. Aside from manually checking, there is no indication when partial allocations occur. As such, THP's effectiveness is very much dependent on the host system's memory fragmentation at the time of VM startup. If this trade off is unacceptable or strict guarantees are required, [[#Static huge pages]] is recommended.<br />
<br />
Arch kernels have THP compiled in and enabled by default with {{ic|1=/sys/kernel/mm/transparent_hugepage/enabled}} set to {{ic|1=madvise}} mode.<br />
<br />
==== Static huge pages ====<br />
<br />
While transparent huge pages should work in the vast majority of cases, they can also be allocated statically during boot. This should only be needed to make use 1 GiB hugepages on machines that support it, since transparent huge pages normally only go up to 2 MiB.<br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4 GiBs worth of huge pages on a machine with 8 GiB of memory will only leave you with 4 GiB of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel command line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048 KiB per huge page creates 2 GiB worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GiB huge page support could be verified by {{ic|grep pdpe1gb /proc/cpuinfo}}. Setting 1 GiB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
==== Dynamic huge pages ====<br />
<br />
{{Accuracy|Need futher testing if this variant as effective as static one}}<br />
<br />
Hugepages could be allocated manually via {{ic|vm.nr_overcommit_hugepages}} [[sysctl]] parameter.<br />
<br />
{{hc|/etc/sysctl.d/10-kvm.conf|2=<br />
vm.nr_hugepages = 0<br />
vm.nr_overcommit_hugepages = ''num''<br />
}}<br />
<br />
Where {{ic|''num''}} - is the number of huge pages, which default size if 2 MiB.<br />
Pages will be automatically allocated, and freed after VM stops.<br />
<br />
More manual way:<br />
<br />
# echo ''num'' > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages<br />
# echo ''num'' > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages<br />
<br />
For 2 MiB and 1 GiB page size respectively.<br />
And they should be manually freed in the same way.<br />
<br />
It is hardly recommended to drop caches, compact memory and wait couple of seconds before starting VM, as there could be not enough free contiguous memory for required huge pages blocks. Especially after some uptime of the host system.<br />
<br />
# echo 3 > /proc/sys/vm/drop_caches<br />
# echo 1 > /proc/sys/vm/compact_memory<br />
<br />
Theoretically, 1 GiB pages works as 2 MiB. But practically - no guaranteed way was found to get contiguous 1 GiB memory blocks. Each consequent request of 1 GiB blocks lead to lesser and lesser dynamically allocated count.<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== Isolating pinned CPUs ===<br />
<br />
CPU pinning by itself won't prevent other host processes from running on the pinned CPUs. Properly isolating the pinned CPUs can reduce latency in the guest VM.<br />
<br />
==== With isolcpus kernel parameter ====<br />
<br />
In this example, let us assume you are using CPUs 4-7.<br />
Use the [[kernel parameters]] {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. For example:<br />
<br />
isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.removeddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this Removeddit mirror of a Reddit thread] for more info. ([https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ The original thread] is worthless because of deleted comments.)<br />
<br />
==== Dynamically isolating CPUs ====<br />
<br />
The isolcpus kernel parameter will permanently reserve CPU cores, even when the guest isn't running. A more flexible alternative is to use the cset tool from {{AUR|cpuset-git}} to dynamically isolate CPUs when starting the guest.<br />
<br />
See this [https://www.redhat.com/archives/vfio-users/2016-September/msg00072.html vfio-users post] for more info, as well as this [https://rokups.github.io/#!pages/gaming-vm-performance.md blog post] and this [https://github.com/PassthroughPOST/VFIO-Tools/blob/master/libvirt_hooks/hooks/cset.sh script] for working examples.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you are running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you are on the latest). If you are running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
Starting with QEMU 3.1 the TOPOEXT cpuid flag is disabled by default. In order to use hyperthreading(SMT) on AMD CPUs you need to manually enable it:<br />
<br />
<cpu mode='host-passthrough' check='none'><br />
<topology sockets='1' cores='4' threads='2'/><br />
<feature policy='require' name='topoext'/><br />
</cpu><br />
<br />
commit: https://git.qemu.org/?p=qemu.git;a=commit;h=7210a02c58572b2686a3a8d610c6628f87864aed<br />
<br />
=== Virtio disk ===<br />
<br />
{{Merge|QEMU#Installing virtio drivers|Off-topic.|section=Moving virtio disk section to QEMU}}<br />
<br />
The default disk types are SATA or IDE emulation out of the box. These controllers offer maximum compatibility but are not suited for efficient virtualization. Two accelerated models exist: {{ic|1=virtio-scsi}} for SCSI emulation and passthrough, or {{ic|1=virtio-blk}} for a more basic block device emulation.<br />
<br />
==== Drivers ====<br />
<br />
* Linux guests should support these out of the box on any modern kernel<br />
* macOS has rudimentary {{ic|1=virtio-blk}} support starting in Mojave via {{ic|1=AppleVirtIO.kext}} but performance is poor and TRIM does not appear to work. SATA is still suggested<br />
* Windows needs the [https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/ Windows virtio drivers]. {{ic|1=virtio-scsi}} uses the {{ic|1=vioscsi}} driver. {{ic|1=virtio-blk}} uses the {{ic|1=viostor}} driver<br />
* Windows can be installed directly onto these disks by selecting 'load driver' on the installer disk selection menu. The windows iso and virtio driver iso should both be attached as regular SATA/IDE cdroms during the installation process<br />
* To switch boot disks to virtio on an existing Windows installation:<br />
** {{ic|1=virtio-blk}}: Add a temporary disk with bus {{ic|1=virtio}}, boot windows & load the driver for the disk, then shutdown and switch the boot disk disk bus to {{ic|1=virtio}}<br />
** {{ic|1=virtio-scsi}}: Add a scsi controller with model {{ic|1=virtio}}, boot windows & load the driver for the controller, then shutdown and switch the boot disk bus to {{ic|1=scsi}} (not virtio)<br />
<br />
==== Considerations ====<br />
<br />
* {{ic|1=virtio-scsi}} TRIM support is mature, all versions should support it. Traditionally, {{ic|1=virtio-scsi}} has been the preferred approach for this reason<br />
* {{ic|1=virtio-blk}} TRIM support is new, this requires requires qemu 4.0+, guest linux kernel 5.0+, guest windows drivers 0.1.173+<br />
* Thin provisioning works by enabling TRIM on a sparse image file: {{ic|1=discard='unmap'}}. Unused blocks will be freed and the disk usage will drop (works on both raw and qcow2). Actual on-disk size of a sparse image file may be checked with {{ic|1=du /path/to/disk.img}}<br />
* Thin provisioning can also work with block storage such as zfs zvols or thin lvm<br />
* Virt queue count will influence the number of threads inside the guest kernel used for IO processing, suggest using {{ic|1=queues='4'}} or more<br />
* Native mode ({{ic|1=io='native'}}) uses a single threaded model based on linux AIO, is a bit more CPU efficient but may have lower peak performance and doesn't allow host side caching to be used<br />
* Threaded mode ({{ic|1=io='threads'}}) will spawn dozens of threads on demand as the disk is used. This is less efficient but may perform better if there are enough host cores available to run them, and allows for host side caching to be used<br />
* Modern versions of libvirt will group the dynamic worker threads created when using threaded mode in with the iothread=1 cgroup for pinning purposes. Very old versions of libvirt left these in the emulator cgroup<br />
<br />
==== IO threads ====<br />
<br />
An IO thread is a dedicated thread for processing disk events, rather than using the main qemu emulator loop. This should not be confused with the worker threads spawned on demand with {{ic|1=io='threads'}}.<br />
<br />
* You can only use one iothread per disk controller. The thread must be assigned to a specific controller with {{ic|1=iothread='X'}} in the {{ic|1=<driver>}} tag. Furthermore, extra & unassigned iothreads will not be used and do nothing<br />
* In the case of {{ic|1=virtio-scsi}}, there is one controller for multiple scsi disks. The iothread is assigned on the controller: {{ic|1=<controller><driver iothread='X'>}}<br />
* In the case of {{ic|1=virtio-blk}}, each disk has its own controller. The iothread is assigned in the driver tag under the disk itself: {{ic|1=<disk><driver iothread='X'>}}<br />
* Since emulated disks incur a significant amount of CPU overhead, that can lead to vcpu stuttering under high disk load (especially high random IOPS). In this case it helps to pin the IO to different core(s) than your vcpus with {{ic|1=<iothreadpin>}}<br />
<br />
==== Examples with libvirt ====<br />
<br />
virtio-scsi + iothread + worker threads + host side writeback caching + full disk block device backend:<br />
<domain><br />
<devices><br />
<disk type='block' device='disk'><br />
<driver name='qemu' type='raw' cache='writeback' io='threads' discard='unmap'/><br />
<source dev='/dev/disk/by-id/ata-Samsung_SSD_840_EVO_1TB_S1D9NSAF206396F'/><br />
<target dev='sda' bus='scsi'/><br />
</disk><br />
<controller type='scsi' index='0' model='virtio-scsi'><br />
<driver iothread='1' queues='8'/><br />
</controller><br />
<br />
virtio-blk + iothread + native aio + no host caching + raw sparse image backend:<br />
<domain><br />
<devices><br />
<disk type='file' device='disk'><br />
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap' iothread='1' queues='8'/><br />
<source dev='/var/lib/libvirt/images/pool/win10.img'/><br />
<target dev='vda' bus='virtio'/><br />
</disk><br />
<br />
Creating the iothreads:<br />
<domain><br />
<iothreads>1</iothreads><br />
<br />
Pinning iothreads:<br />
<domain><br />
<cputune><br />
<iothreadpin iothread='1' cpuset='0-1,6-7'/><br />
<br />
=== Virtio network ===<br />
<br />
The default NIC models rtl8139 or e1000 can be a bottleneck for gigabit+ speeds and have a significant amount of CPU overhead compared to {{ic|1=virtio-net}}.<br />
<br />
* Select {{ic|1=virtio}} as the model for the NIC with libvirt or use the {{ic|1=virtio-net-pci}} device in bare qemu<br />
* Windows needs the {{ic|1=NetKVM}} driver from [https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/ Windows virtio drivers]<br />
* Virtio uses vhost-net by default for in-kernel packet processing without exiting to userspace<br />
* Multiqueue can enabled for a further speedup with multiple connections but typically will not boost single stream speeds. For libvirt add {{ic|1=<driver queues='8'/>}} under the interface tag<br />
* Zero copy transmit may also be enabled on macvtap by setting the module parameter {{ic|1=vhost_net.experimental_zcopytx=1}} but this may actually have worse performance, see [https://github.com/torvalds/linux/commit/098eadce3c622c07b328d0a43dda379b38cf7c5e commit]<br />
<br />
Libvirt example with a bridge:<br />
<br />
<interface type='bridge'><br />
<mac address="52:54:00:6d:6e:2e"/><br />
<source bridge='br0'/><br />
<model type='virtio'/><br />
<driver queues='8'/><br />
</interface><br />
<br />
=== Further tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide].<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how vfio-pci uses your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/local/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/bus/pci/devices/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
DEVS="0000:03:00.0 0000:03:00.1"<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}:<br />
<br />
# Add {{ic|modconf}} to the [[mkinitcpio#HOOKS|HOOKS]] array and {{ic|/usr/local/bin/vfio-pci-override.sh}} to the [[mkinitcpio#BINARIES and FILES|FILES]] array.<br />
<br />
Edit {{ic|/etc/modprobe.d/vfio.conf}}:<br />
<br />
# Add the following line: {{ic|install vfio-pci /usr/local/bin/vfio-pci-override.sh}}<br />
# [[Regenerate the initramfs]] and reboot.<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|This is related to VBIOS issues and should be moved into a separate section regarding VBIOS compatibility.|section=UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that is not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.hostfission.com/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total mebibytes + 2<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MiB + 2 = 17.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 17.82 is bigger than 16 we should choose 32.<br />
<br />
Next create a configuration file to create the shared memory file on boot<br />
<br />
{{hc|/etc/tmpfiles.d/10-looking-glass.conf|2=<br />
f /dev/shm/looking-glass 0660 '''user''' kvm -<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Ask systemd-tmpfiles to create the shared memory file now without waiting to next boot<br />
<br />
# systemd-tmpfiles --create /etc/tmpfiles.d/10-looking-glass.conf<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to go into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download the signed driver [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/upstream-virtio/ from Red Hat].<br />
<br />
Once the driver is installed you must download a matching [https://github.com/gnif/LookingGlass/releases looking-glass-host] that matches the client you installed from AUR and start it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]<br />
It is also possible to make it start automatically on VM boot by editing the {{ic|HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run}} registry and adding a path to the downloaded executable.<br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you do not want to use Spice to control the guest mouse and keyboard you can disable the Spice server.<br />
<br />
$ looking-glass-client -s<br />
<br />
Additionally you may want to start Looking Glass Client as full screen, otherwise the image may be scaled down resulting in poor image fidelity.<br />
<br />
$ looking-glass-client -F<br />
<br />
Launch with the {{ic|--help}} option for further information.<br />
<br />
=== Swap peripherals to and from the Host ===<br />
<br />
Looking Glass includes a Spice client in order to control mouse movement on the Windows guest. However this may have too much latency for certain applications, such as gaming. An alternative method is passing through specific USB devices for minimal latency. This allows for switching the devices between host and guest.<br />
<br />
First create a .xml file for the device(s) you wish to pass-through, which libvirt will use to identify the device.<br />
<br />
{{hc|~/.VFIOinput/input_1.xml|2=<br />
<hostdev mode='subsystem' type='usb' managed='no'><br />
<source><br />
<vendor id='0x[Before Colon]'/><br />
<product id='0x[After Colon]'/><br />
</source><br />
</hostdev><br />
}}<br />
<br />
Replace [Before/After Colon] with the contents of the 'lsusb' command, specific to the device you want to pass-through.<br />
<br />
For instance my mouse is {{ic|Bus 005 Device 002: ID 1532:0037 Razer USA, Ltd}} so I would replace {{ic|vendor id}} with 1532, and {{ic|product id}} with 1037.<br />
<br />
Repeat this process for any additional USB devices you want to pass-through. If your mouse / keyboard has multiple entries in {{ic|lsusb}}, perhaps if it is wireless, then create additional xml files for each.<br />
<br />
{{Note|Do not forget to change the path & name of the script(s) above and below to match your user and specific system.}}<br />
<br />
Next a bash script file is needed to tell libvirt what to attach/detach the USB devices to the guest.<br />
<br />
{{hc|~/.VFIOinput/input_attach.sh|2=<br />
#!/bin/bash<br />
<br />
virsh attach-device [VM-Name] [USBdevice]<br />
}}<br />
<br />
Replace [VM-Name] with the name of your virtual machine, which can be seen under virt-manager. Additionally replace [USBdevice] with the '''full''' path to the .xml file for the device you wish to pass-through. Add additional lines for more than 1 device. For example here is my script:<br />
<br />
{{hc|~/.VFIOinput/input_attach.sh|2=<br />
#!/bin/bash<br />
<br />
virsh attach-device win10 /home/$USER/.VFIOinput/input_mouse.xml<br />
virsh attach-device win10 /home/$USER/.VFIOinput/input_keyboard.xml<br />
}}<br />
<br />
Next duplicate the script file and replace {{ic|attach-device}} with {{ic|detach-device}}. Ensure both scripts are executable with {{ic|chmod +x $script.sh}}<br />
<br />
This 2 script files can now be executed to attach or detach your USB devices from the host to the guest VM. It is important to note that they may need to be executed as root. To run the script from the Windows VM, one possibility is using [[PuTTY]] to [[SSH]] into the host, and execute the script. On Windows PuTTY comes with plink.exe which can execute singular commands over SSH before then logging out, instead of opening a SSH terminal, all in the background.<br />
<br />
{{hc|detach_devices.bat|2=<br />
"C:\Program Files\PuTTY\plink.exe" root@$HOST_IP -pw $ROOTPASSWORD /home/$USER/.VFIOinput/input_detach.sh<br />
}}<br />
<br />
Replace {{ic|$HOST_IP}} with the Host [[Network configuration#IP addresses|IP Address]] and $ROOTPASSWORD with the root password.<br />
<br />
{{warning|This method is insecure if somebody has access to your VM, since they could open the file and read your password. It is advisable to use [[SSH keys]] instead!}}<br />
<br />
You may also want to execute the script files using key binds. On Windows one option is [https://autohotkey.com/ Autohotkey], and on the Host [[Xbindkeys]]. Because of the need to run the scripts as root, you may also need to use [[Polkit]] or [[Sudo]] which can both be used to authenticate specific executables as able to run as root without needing a password.<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream,multifunction}} should break up as many devices as possible.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|edk2-ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/edk2-ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/edk2-ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it is not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It is recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing through other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting: [[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
{{Note|If your USB controller does not support resetting, is not in an isolated group, or is otherwise unable to be passed through then it may still be possible to accomplish similar results through [[udev]] rules. See [https://github.com/olavmrk/usb-libvirt-hotplug] which allows any device connected to specified USB ports to be automatically attached to a virtual machine.}}<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<nowiki> </devices><br />
<qemu:commandline><br />
<qemu:arg value="-audiodev"/><br />
<qemu:arg value="pa,id=snd0,server=/run/user/1000/pulse/native"/><br />
</qemu:commandline><br />
</domain></nowiki><br />
}}<br />
<br />
You can also use the /tmp directory if you have multiple users accessing PulseAudio<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<nowiki> </devices><br />
<qemu:commandline><br />
<qemu:arg value="-audiodev"/><br />
<qemu:arg value="pa,id=snd0,server=unix:/tmp/pulse-socket"/><br />
</qemu:commandline><br />
</domain></nowiki><br />
}}<br />
<br />
You may want to change 1000 under the user directory to your current user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied. Virtual Machine audio will now be routed through the host as an application. A application such as {{Pkg|pavucontrol}} can be used to control the output device.<br />
<br />
==== QEMU 4.0/4.2+ audio changes ====<br />
<br />
As of QEMU 4.0 and made official with [https://www.kraxel.org/blog/2020/01/qemu-sound-audiodev/ QEMU 4.2] the {{ic|1=-audiodev}} parameter is now to be used with PulseAudio sound passthrough, any previous environmental variables you have defined in your XML '''need''' to be removed e.g. {{ic|qemu:env}} to properly use this.<br />
<br />
You will also need to change the chipset accordingly with each QEMU update to how your VM is set up, i.e. {{ic|pc-q35-4.2}} or {{ic|pc-i440fx-4.2}} to advantage of the changes in QEMU:<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<os><br />
<type arch<nowiki>=</nowiki>'x86_64' machine<nowiki>=</nowiki>'pc-q35-4.2'>hvm</type><br />
...<br />
</os><br />
...<br />
<os><br />
<type arch<nowiki>=</nowiki>'x86_64' machine<nowiki>=</nowiki>'pc-i440fx-4.2'>hvm</type><br />
...<br />
</os>}}<br />
<br />
The latest audiodev changes in QEMU also suggest using the ICH9 instead of ICH6 or AC97 for audio emulation to take advantage of this newer code, it is recommended to use the parameters as shown below because Libvirt currently does not define the {{ic|1=id=}} parameter correctly:<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<nowiki><qemu:arg value="-device"/><br />
<qemu:arg value="ich9-intel-hda,bus=pcie.0,addr=0x1b"/><br />
<qemu:arg value="-device"/><br />
<qemu:arg value="hda-micro,audiodev=hda"/><br />
<qemu:arg value="-audiodev"/><br />
<qemu:arg value="pa,id=hda,server=unix:/tmp/pulse-socket"/></nowiki><br />
}}<br />
<br />
QEMU 4.2 also has 5.1 channel audio support via {{ic|usb-audio}} emulation, this can be enabled as shown below:<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<nowiki><qemu:arg value="-device"/><br />
<qemu:arg value="usb-audio,audiodev=usb,multi=on"/><br />
<qemu:arg value="-audiodev"/><br />
<qemu:arg value="pa,id=usb,server=unix:/tmp/pulse-socket,out.mixing-engine=off"/></nowiki><br />
}}<br />
<br />
{{Note|1=<nowiki></nowiki><br />
* You can have multiple audio backends, by simply specifying {{ic|1=-audiodev}} multiple times in your XML and by assigning them different ids. This can be useful for a use case of having two identical backends. With PulseAudio each backend is a separate stream and can be routed to different output devices on the host (using a pulse mixer app like {{Pkg|pavucontrol}} or {{Pkg|pulsemixer}}).<br />
* USB 3 emulation is needed in Libvirt/QEMU to enable the {{ic|usb-audio}} parameter<br />
* It is recommended to enable MSI interrupts with a tool such as [https://github.com/CHEF-KOCH/MSI-utility this] on the ICH9 audio device to mitigate any crackling, stuttering, speedup, or no audio at all after VM restart<br />
* The {{ic|1=-audiodev=hda}} parameter can be changed to your liking, but as shown above, the names must match each other or the VM will not start<br />
* If audio is crackling/stuttering/speedup etc. is still present you may want to adjust parameters such as {{ic|buffer-length}} and {{ic|timer-period}}, more information on these parameters and more can be found in the {{man|1|qemu}} manual<br />
* Some audio chipsets such as [https://bugzilla.kernel.org/show_bug.cgi?id=195303 Realtek alc1220] may also have issues out of the box so do consider this when using any audio emulation with QEMU<br />
* Improper pinning or heavy host usage without using [[#Isolating pinned CPUs|isolcpus]] can also influence sound bugs, especially while gaming in a VM<br />
}}<br />
<br />
=== Passing VM audio to host via Scream ===<br />
<br />
It is possible to pass VM audio through a bridged network such as the one provided by Libvirt or by adding a IVSHMEM device to the host by using a application called [https://github.com/duncanthrax/scream Scream]. This section will only cover using PulseAudio as a receiver on the host.<br />
See the project page for more details and instructions on other methods.<br />
<br />
==== Using Scream with a bridged network ====<br />
<br />
{{Note|<br />
* This is the ''preferred'' way to use this, although results may vary per user<br />
* It is recommend to use the [[#Virtio network]] adapter while using Scream, other virtual adapters provided by QEMU such as '''e1000e''' may lead to poor performance<br />
}}<br />
<br />
To use scream via your network you will want to find your bridge name via {{ic|1=ip -a}}, in most cases it will be called '''br0''' or '''virbr0'''. Below is a example of the command needed to start the Scream application:<br />
<br />
{{bc|<nowiki> $ scream-pulse -i virbr0 &</nowiki>}}<br />
<br />
{{Warning| This will not work with a '''macvtap bridge''' as that doesn't allow host to guest communication, also make sure you have the proper firewall ports open for it to communicate with the VM}}<br />
<br />
==== Adding the IVSHMEM device to use Scream with IVSHMEM ====<br />
<br />
With the VM turned off, edit the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='scream-ivshmem'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>2</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
In the above configuration, the size of the IVSHMEM device is 2MB (the recommended amount). Change this as needed.<br />
<br />
Now refer to [[#Adding IVSHMEM Device to VM]] to configure the host to create the shared memory file on boot, replacing {{ic|looking-glass}} with {{ic|scream-ivshmem}}.<br />
<br />
===== Configuring the Windows guest for IVSHMEM =====<br />
<br />
The correct driver must be installed for the IVSHMEM device on the guest. <br />
See [[#Installing the IVSHMEM Host to Windows guest]]. Ignore the part about {{ic|looking-glass-host}}.<br />
<br />
Install the [https://github.com/duncanthrax/scream/releases Scream] virtual audio driver on the guest. <br />
If you have secure boot enabled for your VM, you may need to disable it. <br />
<br />
Using the registry editor, set the DWORD {{ic|HKLM\SYSTEM\CurrentControlSet\Services\Scream\Options\UseIVSHMEM}} to the size of the IVSHMEM device in MB.<br />
Note that scream identifies its IVSHMEM device using its size, so make sure there is only one device of that size.<br />
<br />
====== Configuring the host ======<br />
<br />
Install {{AUR|scream}}.<br />
<br />
Create the systemd user service file to control the reciever<br />
<br />
{{hc|~/.config/systemd/user/scream-ivshmem-pulse.service|<nowiki><br />
[Unit]<br />
Description=Scream IVSHMEM pulse reciever<br />
After=pulseaudio.service<br />
Wants=pulseaudio.service<br />
<br />
[Service]<br />
Type=simple<br />
ExecStartPre=/usr/bin/truncate -s 0 /dev/shm/scream-ivshmem<br />
ExecStartPre=/usr/bin/dd if=/dev/zero of=/dev/shm/scream-ivshmem bs=1M count=2<br />
ExecStart=/usr/bin/scream -m /dev/shm/scream-ivshmem<br />
<br />
[Install]<br />
WantedBy=default.target<br />
<br />
</nowiki>}}<br />
<br />
Edit {{ic|1=count=2}} with the size of the IVSHMEM device in MB.<br />
<br />
Now start the service with <br />
$ systemctl start --user scream-ivshmem-pulse<br />
<br />
To have it automatically start on next login, enable the service<br />
$ systemctl enable --user scream-ivshmem-pulse<br />
<br />
=== Physical disk/partition ===<br />
<br />
Raw and qcow2 especially can have noticeable overhead for heavy IO. A whole disk or a partition may be used directly to bypass the filesystem and improve I/O performance. If you wish to dual boot the guest OS natively you would need to pass the entire disk without any partitioning. It is suggested to use /dev/disk/by- paths to refer to the disk since /dev/sdX entries can change between boots. To find out which disk/partition is associated with the one you would like to pass:<br />
<br />
{{hc|$ ls -l /dev/disk/by-id|<br />
ata-ST1000LM002-9VQ14L_Z0501SZ9 -> ../../sdd<br />
}}<br />
<br />
See [[#Virtio disk]] on how to add these with libvirt XML. You can also add the disk with Virt-Manager's '''Add Hardware''' menu and then type the disk you want in the '''Select or create custom storage''' box, e.g. '''/dev/disk/by-id/ata-ST1000LM002-9VQ14L_Z0501SZ9'''<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functional and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
For many reasons users may seek to see [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]].<br />
<br />
These examples offer a supplement to existing hardware compatibility lists. Additionally, if you have trouble configuring a certain mechanism in your setup, you might find these examples very valuable. Users there have described their setups in detail, and some have provided examples of their configuration files as well. <br />
<br />
We encourage those who successfully build their system from this resource to help improve it by contributing their builds. Due to the many different hardware manufacturers involved, the seemingly significant lack of sufficient documentation, as well as other issues due to the nature of this process, community contributions are necessary.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== QEMU 4.0: Unable to load graphics drivers/BSOD/Graphics stutter after driver install using Q35 ===<br />
<br />
Starting with QEMU 4.0, the Q35 machine type changes the default {{ic|<nowiki>kernel_irqchip</nowiki>}} from {{ic|off}} to {{ic|split}} which breaks some guest devices, such as nVidia graphics (the driver fails to load / black screen / code 43 / graphics stutters, usually when mouse moving). Switch to full KVM mode instead by adding {{ic|<nowiki><ioapic driver='kvm'/></nowiki>}} under libvirt's {{ic|<nowiki><features></nowiki>}} tag in your VM configuration or by adding {{ic|<nowiki>kernel_irqchip=on</nowiki>}} in the {{ic|-machine}} QEMU arg.<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the hypervisor parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. The vendor_id can be [https://libvirt.org/formatdomain.html#elementsFeatures any string value up to 12 characters].<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
=== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ===<br />
<br />
{{Expansion|This error is actually related to the boot_vgs issue and should be merged together with everything else concerning GPU ROMs.|section=UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
In addition try adding kernel parameter {{ic|1=pci=realloc}} which also [https://github.com/Dunedan/mbp-2016-linux/issues/60#issuecomment-396311301 helps with hotplugging issues].<br />
<br />
=== UEFI (OVMF) compatibility in VBIOS ===<br />
<br />
{{Remove|Flashing you guest GPU for the purpose of a GPU passthrough is '''never''' good advice. A full section should be dedicated to VBIOS compatibility.|section= UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it is not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it is stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it is pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
On a linux guest you can use modinfo to see if there is option to enable MSI (for example: "modinfo snd_hda_intel |grep msi"). If there is, one can enable it by adding the relevant option to a custom omdprobe file - in "/etc/modprobe.d/snd-hda-intel.conf" inserting "options snd-hda-intel enable_msi=1"<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
A UI tool called [https://github.com/CHEF-KOCH/MSI-utility MSI Utility (FOSS Version 2)] works with Windows 10 64-bit and simplifies the process.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read [https://www.kernel.org/doc/html/latest/x86/intel-iommu.html#graphics-problems intel-iommu.html] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X does not start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user is added to the {{ic|kvm}} and {{ic|libvirt}} [[user group]]s.<br />
<br />
=== Host lockup after VM shutdown ===<br />
<br />
This issue seems to primarily affect users running a Windows 10 guest and usually after the VM has been run for a prolonged period of time: the host will experience multiple CPU core lockups (see [https://bbs.archlinux.org/viewtopic.php?id=206050&p=2]). To fix this try enabling Message Signal Interrupts on the GPU passed through to the guest. A good guide for how to do this can be found in [https://forums.guru3d.com/threads/windows-line-based-vs-message-signaled-based-interrupts.378044/]. You can also download this application for windows here [https://github.com/TechtonicSoftware/MSIInturruptEnabler] that should make the process easier.<br />
<br />
=== Host lockup if guest is left running during sleep ===<br />
<br />
VFIO-enabled virtual machines tend to become unstable if left running through a sleep/wakeup cycle and have been known to cause the host machine to lockup when an attempt is then made to shut them down. In order to avoid this, one can simply prevent the host from going into sleep while the guest is running using the following libvirt hook script and systemd unit. The hook file needs executable permissions to work.<br />
<br />
{{hc|/etc/libvirt/hooks/qemu|<nowiki><br />
#!/bin/bash<br />
<br />
OBJECT="$1"<br />
OPERATION="$2"<br />
SUBOPERATION="$3"<br />
EXTRA_ARG="$4"<br />
<br />
case "$OPERATION" in<br />
"prepare")<br />
systemctl start libvirt-nosleep@"$OBJECT"<br />
;;<br />
"release")<br />
systemctl stop libvirt-nosleep@"$OBJECT"<br />
;;<br />
esac<br />
</nowiki>}}<br />
<br />
{{hc|/etc/systemd/system/libvirt-nosleep@.service|<nowiki><br />
[Unit]<br />
Description=Preventing sleep while libvirt domain "%i" is running<br />
<br />
[Service]<br />
Type=simple<br />
ExecStart=/usr/bin/systemd-inhibit --what=sleep --why="Libvirt domain \"%i\" is running" --who=%U --mode=block sleep infinity<br />
</nowiki>}}<br />
<br />
=== Cannot boot after upgrading ovmf ===<br />
<br />
If you cannot boot after upgrading from {{Pkg|ovmf}}{{Broken package link|replaced by {{Pkg|edk2-ovmf}}}} version 1:r23112.018432f0ce-1 then you need to remove the old {{ic|*VARS.fd}} file in {{ic|/var/lib/libvirt/qemu/nvram/}}:<br />
<br />
# mv /var/lib/libvirt/qemu/nvram/vmname_VARS.fd /var/lib/libvirt/qemu/nvram/vmname_VARS.fd.old<br />
<br />
See {{Bug|57825}} for further details.<br />
<br />
=== QEMU via cli pulseaudio stuttering/delay ===<br />
<br />
Using following flags for the audio device and chipset might help if you are running into the stuttering/delay audio issues when running QEMU via cli:<br />
<br />
qemu-system-x86_64 \<br />
-machine pc-i440fx-3.0 \<br />
-device hda-micro \<br />
-soundhw hda \<br />
-...<br />
<br />
As noted in [[#QEMU 3.0 audio changes|QEMU 3.0 audio changes]]{{Broken section link}} the specified chipset will include a series of audio patches.<br />
<br />
Setting {{ic|QEMU_AUDIO_TIMER_PERIOD}} to values higher than 100 might also help (did not test value lower than 100).<br />
<br />
=== Bluescreen at boot since Windows 10 1803 ===<br />
<br />
Since Windows 10 1803 there is a problem when you are using "host-passthrough" as cpu model. The machine cannot boot and is either boot looping or you get a bluescreen.<br />
You can workaround this by:<br />
<br />
# echo 1 > /sys/module/kvm/parameters/ignore_msrs<br />
<br />
To make it permanently you can create a modprobe file {{ic|kvm.conf}}:<br />
<br />
options kvm ignore_msrs=1<br />
=== AMD Ryzen / BIOS updates (AGESA) yields "Error: internal error: Unknown PCI header type ‘127’" ===<br />
<br />
AMD users have been experiencing breakage of their KVM setups after updating the BIOS on their motherboard. There is a kernel [https://clbin.com/VCiYJ patch], (see [[Kernel/Arch Build System]] for instruction on compiling kernels with custom patches) that can resolve the issue as of now (7/28/19), but this is not the first time AMD has made an error of this very nature, so take this into account if you are considering updating your BIOS in the future as a VFIO user.<br />
=== Host crashes when hotplugging Nvidia card with USB ===<br />
<br />
If attempting to hotplug an Nvidia card with a USB port, you may have to blacklist the {{ic|i2c_nvidia_gpu}} driver. Do this by adding the line {{ic|blacklist i2c_nvidia_gpu}} to {{ic|/etc/modprobe.d/blacklist.conf}}.<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script] from https://www.youtube.com/watch?v=37D2bRsthfI<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [ircs://chat.freenode.net/vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]<br />
* [https://www.reddit.com/r/VFIO /r/VFIO: A subreddit focused on vfio]<br />
* [https://github.com/intel/gvt-linux/wiki/GVTd_Setup_Guide GVT-d: passthrough of an entire integrated GPU]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=614376PCI passthrough via OVMF2020-05-19T11:01:42Z<p>NeoTheFox: /* Configuring the host */ replaced scream-pulse with scream for a proper AUR link</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
[[zh-hans:PCI passthrough via OVMF]]<br />
{{Related articles start}}<br />
{{Related|Intel GVT-g}}<br />
{{Related articles end}}<br />
<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of [[QEMU]], it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=873&0_VTD=True List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Note|<br />
* IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
* VT-d stands for ''Intel Virtualization Technology for Directed I/O'' and should not be confused with VT-x ''Intel Virtualization Technology''. VT-x allows one hardware platform to function as multiple “virtual” platforms while VT-d improves security and reliability of the systems and also improves performance of I/O devices in virtualized environments.<br />
}}<br />
<br />
Using IOMMU opens to features like PCI passthrough and memory protection from faulty or malicious devices, see [[Wikipedia:Input-output memory management unit#Advantages]] and [https://www.quora.com/Memory-Management-computer-programming/Could-you-explain-IOMMU-in-plain-English Memory Management (computer programming): Could you explain IOMMU in plain English?].<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is supported by the CPU and enabled in the BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
Enable IOMMU support by setting the correct [[kernel parameter]] depending on the type of CPU in use:<br />
<br />
* For Intel CPUs (VT-d) set {{ic|1=intel_iommu=on}} <br />
* For AMD CPUs (AMD-Vi) set {{ic|1=amd_iommu=on}}<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -i -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for g in /sys/kernel/iommu_groups/*; do<br />
echo "IOMMU Group ${g##*/}:"<br />
for d in $g/devices/*; do<br />
echo -e "\t$(lspci -nns ${d##*/})"<br />
done;<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 1:<br />
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
IOMMU Group 2:<br />
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:0e31] (rev 04)<br />
IOMMU Group 4:<br />
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:0e2d] (rev 04)<br />
IOMMU Group 10:<br />
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:0e26] (rev 04)<br />
IOMMU Group 13:<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1:<br />
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in to your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the preferred method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{Warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this - your motherboard should be set to display using the host GPU.}}<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub did, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use.<br />
<br />
=== Binding vfio-pci via device ID ===<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13:<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|<br />
* You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.<br />
* If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.<br />
}}<br />
<br />
Two methods exist for providing the device IDs. Specifying them via [[kernel parameters]] has the advantage of being able to easily edit, remove, or undo any breaking changes via your boot loader:<br />
<br />
vfio-pci.ids=10de:13c2,10de:0fbb<br />
<br />
Alternatively, the IDs may be added to a modprobe conf file. Since these conf files are embedded in the initramfs image, any changes require regenerating a new image each time:<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
=== Loading vfio-pci early ===<br />
<br />
Since Arch's {{Pkg|linux}} has vfio-pci built as a module, we need to force it to load early before the graphics drivers have a chance to bind to the card. To ensure that, add {{ic|vfio_pci}}, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio_pci vfio vfio_iommu_type1 vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]].<br />
<br />
=== Verifying that the configuration worked ===<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it is possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
Install {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|edk2-ovmf}}, and {{Pkg|virt-manager}}.<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
You may also need to [https://wiki.libvirt.org/page/Networking#NAT_forwarding_.28aka_.22virtual_networks.22.29 activate the default libvirt network].<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self-explanatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the {{ic|libvirt}} [[user group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps:<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that:<br />
** Your hypervisor is running as a system session and not a user session. This can be verified [https://i.ibb.co/N1XZCdp/Deepin-Screenshot-select-area-20190125113216.png by clicking, then hovering] over the session in virt-manager. If you are accidentally running it as a user session, you must open a new connection by clicking "File" > "Add Connection..", then select the option from the drop-down menu station "QEMU/KVM" and not "QEMU/KVM user session".<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, it's easier to setup [[#Virtio disk]] before installing<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine. It is possible you will be dropped into the UEFI menu instead of starting the installation upon powering your VM for the first time. Sometimes the correct ISO file was not automatically detected and you will need to manually specify the drive to boot. By typing exit and navigating to "boot manager" you will enter a menu that allows you to choose between devices.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it is now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Passing keyboard/mouse via Evdev ===<br />
<br />
If you do not have a spare mouse or keyboard to dedicate to your guest, and you do not want to suffer from the video overhead of Spice, you can setup evdev to swap control of your mouse and keyboard between the host and guest on the fly.<br />
<br />
First, modify the libvirt configuration. If you encounter {{ic|failed to get domain [vmname]}} error, add {{ic|-c qemu:///system}} arguments to {{ic|virsh}}.<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Next, find your keyboard and mouse devices in {{ic|/dev/input/by-id/}}. You may find multiple devices associated to your mouse or keyboard, so try {{ic|cat /dev/input/by-id/''device_id''}} and either hit some keys on the keyboard or wiggle your mouse to see if input comes through, if so you have got the right device. Now add those devices to your configuration right before the closing </domain> tag:<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<qemu:commandline><br />
<qemu:arg value='-object'/><br />
<qemu:arg value='input-linux,id=mouse1,evdev=/dev/input/by-id/MOUSE_NAME'/><br />
<qemu:arg value='-object'/><br />
<qemu:arg value='input-linux,id=kbd1,evdev=/dev/input/by-id/KEYBOARD_NAME,grab_all=on,repeat=on'/><br />
</qemu:commandline><br />
...<br />
</nowiki>}}<br />
<br />
Replace {{ic|MOUSE_NAME}} and {{ic|KEYBOARD_NAME}} with your device id. You will also need to include these devices in your qemu config, and setting the user and group to one that has access to your input devices:<br />
<br />
{{hc|/etc/libvirt/qemu.conf|<nowiki><br />
...<br />
user = "<your_user>"<br />
group = "kvm"<br />
...<br />
cgroup_device_acl = [<br />
"/dev/kvm",<br />
"/dev/input/by-id/KEYBOARD_NAME",<br />
"/dev/input/by-id/MOUSE_NAME",<br />
"/dev/null", "/dev/full", "/dev/zero",<br />
"/dev/random", "/dev/urandom",<br />
"/dev/ptmx", "/dev/kvm", "/dev/kqemu",<br />
"/dev/rtc","/dev/hpet", "/dev/sev"<br />
]<br />
...<br />
</nowiki>}}<br />
<br />
Then ensure that the user you provided has access to the {{ic|kvm}} and {{ic|input}} [[user group]]s. [[Restart]] {{ic|libvirtd.service}}. Now you can startup the guest OS and test swapping control of your mouse and keyboard between the host and guest by pressing both the left and right control keys at the same time.<br />
<br />
You may also consider switching from PS/2 to Virtio inputs in your configurations:<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<input type='mouse' bus='virtio'><br />
<address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/><br />
</input><br />
<input type='keyboard' bus='virtio'><br />
<address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/><br />
</input><br />
<input type='mouse' bus='ps2'/><br />
<input type='keyboard' bus='ps2'/><br />
...<br />
</nowiki>}}<br />
<br />
Next startup the guest OS and install the virtIO drivers for those devices.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate Linux/Windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. As such, the local CPU cache benefits (L1/L2) are lost each time the host scheduler reschedules the virtual CPU thread on a different physical CPU. This can noticeably harm performance on the guest. CPU pinning aims to resolve this by limiting which physical CPUs the virtual CPUs are allowed to run on. The ideal setup is a one to one mapping such that the virtual CPU cores match physical CPU cores while taking hyperthreading/SMT into account.<br />
<br />
{{Note|For certain users enabling CPU pinning may introduce stuttering and short hangs, especially with the MuQSS scheduler (present in linux-ck and linux-zen kernels). You might want to try disabling pinning first if you experience similar issues, which effectively trades maximum performance for responsiveness at all times.}}<br />
<br />
==== CPU topology ====<br />
<br />
Most modern CPUs support hardware multitasking, also known as hyper-threading on Intel CPUs or SMT on AMD CPUs. Hyper-threading/SMT is simply a very efficient way of running two threads on one CPU core at any given time. You will want to take into consideration that the CPU pinning you choose will greatly depend on what you do with your host while your VM is running.<br />
<br />
To find the topology for your CPU run {{ic|1=lscpu -e}}:<br />
<br />
{{Note|Pay special attention to the 4th column '''"CORE"''' as this shows the association of the Physical/Logical CPU cores}}<br />
<br />
{{ic|lscpu -e}} on a 6c/12t Ryzen 5 1600:<br />
<br />
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ<br />
0 0 0 0 0:0:0:0 yes 3800.0000 1550.0000<br />
1 0 0 0 0:0:0:0 yes 3800.0000 1550.0000<br />
2 0 0 1 1:1:1:0 yes 3800.0000 1550.0000<br />
3 0 0 1 1:1:1:0 yes 3800.0000 1550.0000<br />
4 0 0 2 2:2:2:0 yes 3800.0000 1550.0000<br />
5 0 0 2 2:2:2:0 yes 3800.0000 1550.0000<br />
6 0 0 3 3:3:3:1 yes 3800.0000 1550.0000<br />
7 0 0 3 3:3:3:1 yes 3800.0000 1550.0000<br />
8 0 0 4 4:4:4:1 yes 3800.0000 1550.0000<br />
9 0 0 4 4:4:4:1 yes 3800.0000 1550.0000<br />
10 0 0 5 5:5:5:1 yes 3800.0000 1550.0000<br />
11 0 0 5 5:5:5:1 yes 3800.0000 1550.0000<br />
<br />
{{Note|Ryzen 3000 ComboPi AGESA changes topology to match Intel example, even on prior generation CPUs. Above valid only on older AGESA. }}<br />
<br />
{{ic|lscpu -e}} on a 6c/12t Intel 8700k:<br />
<br />
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ<br />
0 0 0 0 0:0:0:0 yes 4600.0000 800.0000<br />
1 0 0 1 1:1:1:0 yes 4600.0000 800.0000<br />
2 0 0 2 2:2:2:0 yes 4600.0000 800.0000<br />
3 0 0 3 3:3:3:0 yes 4600.0000 800.0000<br />
4 0 0 4 4:4:4:0 yes 4600.0000 800.0000<br />
5 0 0 5 5:5:5:0 yes 4600.0000 800.0000<br />
6 0 0 0 0:0:0:0 yes 4600.0000 800.0000<br />
7 0 0 1 1:1:1:0 yes 4600.0000 800.0000<br />
8 0 0 2 2:2:2:0 yes 4600.0000 800.0000<br />
9 0 0 3 3:3:3:0 yes 4600.0000 800.0000<br />
10 0 0 4 4:4:4:0 yes 4600.0000 800.0000<br />
11 0 0 5 5:5:5:0 yes 4600.0000 800.0000<br />
<br />
As we see above, with AMD '''Core 0''' is sequential with '''CPU 0 & 1''', whereas Intel places '''Core 0''' on '''CPU 0 & 6'''.<br />
<br />
If you do not need all cores for the guest, it would then be preferable to leave at the very least one core for the host. Choosing which cores one to use for the host or guest should be based on the specific hardware characteristics of your CPU, however '''Core 0''' is a good choice for the host in most cases. If any cores are reserved for the host, it is recommended to pin the emulator and iothreads, if used, to the host cores rather than the VCPUs. This may improve performance and reduce latency for the guest since those threads will not pollute the cache or contend for scheduling with the guest VCPU threads. If all cores are passed to the guest, there is no need or benefit to pinning the emulator or iothreads.<br />
<br />
==== XML examples ====<br />
<br />
{{Note|Do not use the '''iothread''' lines from the XML examples shown below if you have not added an '''iothread''' to your disk controller. '''iothread''''s only work on '''virtio-scsi''' or '''virtio-blk''' devices.}}<br />
<br />
===== 4c/1t CPU w/o Hyperthreading Example =====<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='0'/><br />
<vcpupin vcpu='1' cpuset='1'/><br />
<vcpupin vcpu='2' cpuset='2'/><br />
<vcpupin vcpu='3' cpuset='3'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
===== 4c/2t Intel/AMD CPU example (after ComboPI AGESA bios update) =====<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>8</vcpu><br />
<iothreads>1</iothreads><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='8'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='9'/><br />
<vcpupin vcpu='4' cpuset='4'/><br />
<vcpupin vcpu='5' cpuset='10'/><br />
<vcpupin vcpu='6' cpuset='5'/><br />
<vcpupin vcpu='7' cpuset='11'/><br />
<emulatorpin cpuset='0,6'/><br />
<iothreadpin iothread='1' cpuset='0,6'/><br />
</cputune><br />
...<br />
<topology sockets='1' cores='4' threads='2'/><br />
...<br />
</nowiki>}}<br />
<br />
===== 4c/2t AMD CPU example (Before ComboPi AGESA bios update) =====<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>8</vcpu><br />
<iothreads>1</iothreads><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='3'/><br />
<vcpupin vcpu='2' cpuset='4'/><br />
<vcpupin vcpu='3' cpuset='5'/><br />
<vcpupin vcpu='4' cpuset='6'/><br />
<vcpupin vcpu='5' cpuset='7'/><br />
<vcpupin vcpu='6' cpuset='8'/><br />
<vcpupin vcpu='7' cpuset='9'/><br />
<emulatorpin cpuset='0-1'/><br />
<iothreadpin iothread='1' cpuset='0-1'/><br />
</cputune><br />
...<br />
<topology sockets='1' cores='4' threads='2'/><br />
...<br />
</nowiki>}}<br />
<br />
{{Note|If further CPU isolation is needed, consider using the '''isolcpus''' kernel command-line parameter on the unused physical/logical cores.}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, you may want to pin your VM threads across all of your cores, so that the VM can fully take advantage of the spare CPU time the host has available. Be aware that pinning all physical and logical cores of your CPU could induce latency in the guest VM.<br />
<br />
=== Huge memory pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information across multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4 GiB of memory divided into 4 KiB pages (which is the default size for normal pages) for a total of 1.04 million pages, meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession.<br />
<br />
==== Transparent huge pages ====<br />
<br />
QEMU will use 2MiB sized transparent huge pages automatically without any explicit configuration in QEMU or Libvirt, subject to some important caveats. When using VFIO the pages are locked in at boot time and transparent huge pages are allocated up front when the VM first boots. If the kernel memory is highly fragmented, or the VM is using a majority of the remaining free memory, it is likely that the kernel will not have enough 2MiB pages to fully satisfy the allocation. In such a case, it silently fails by using a mix of 2MiB and 4KiB pages. Since the pages are locked in VFIO mode, the kernel will not be able to convert those 4KiB pages to huge after the VM starts either. The number of available 2MiB huge pages available to THP is the same as via the [[#Dynamic huge pages]] mechanism described in the following sections.<br />
<br />
To check how much memory THP is using globally:<br />
<br />
{{hc|$ grep AnonHugePages /proc/meminfo|<br />
AnonHugePages: 8091648 kB<br />
}}<br />
<br />
To check a specific QEMU instance. QEMU's PID must be substituted in the grep command:<br />
<br />
{{hc|$ grep -P 'AnonHugePages:\s+(?!0)\d+' /proc/[PID]/smaps|<br />
AnonHugePages: 8087552 kB<br />
}}<br />
<br />
In this example, the VM was allocated 8388608KiB of memory, but only 8087552KiB was available via THP. The remaining 301056KiB are allocated as 4KiB pages. Aside from manually checking, there is no indication when partial allocations occur. As such, THP's effectiveness is very much dependent on the host system's memory fragmentation at the time of VM startup. If this trade off is unacceptable or strict guarantees are required, [[#Static huge pages]] is recommended.<br />
<br />
Arch kernels have THP compiled in and enabled by default with {{ic|1=/sys/kernel/mm/transparent_hugepage/enabled}} set to {{ic|1=madvise}} mode.<br />
<br />
==== Static huge pages ====<br />
<br />
While transparent huge pages should work in the vast majority of cases, they can also be allocated statically during boot. This should only be needed to make use 1 GiB hugepages on machines that support it, since transparent huge pages normally only go up to 2 MiB.<br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4 GiBs worth of huge pages on a machine with 8 GiB of memory will only leave you with 4 GiB of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel command line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048 KiB per huge page creates 2 GiB worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GiB huge page support could be verified by {{ic|grep pdpe1gb /proc/cpuinfo}}. Setting 1 GiB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
==== Dynamic huge pages ====<br />
<br />
{{Accuracy|Need futher testing if this variant as effective as static one}}<br />
<br />
Hugepages could be allocated manually via {{ic|vm.nr_overcommit_hugepages}} [[sysctl]] parameter.<br />
<br />
{{hc|/etc/sysctl.d/10-kvm.conf|2=<br />
vm.nr_hugepages = 0<br />
vm.nr_overcommit_hugepages = ''num''<br />
}}<br />
<br />
Where {{ic|''num''}} - is the number of huge pages, which default size if 2 MiB.<br />
Pages will be automatically allocated, and freed after VM stops.<br />
<br />
More manual way:<br />
<br />
# echo ''num'' > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages<br />
# echo ''num'' > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages<br />
<br />
For 2 MiB and 1 GiB page size respectively.<br />
And they should be manually freed in the same way.<br />
<br />
It is hardly recommended to drop caches, compact memory and wait couple of seconds before starting VM, as there could be not enough free contiguous memory for required huge pages blocks. Especially after some uptime of the host system.<br />
<br />
# echo 3 > /proc/sys/vm/drop_caches<br />
# echo 1 > /proc/sys/vm/compact_memory<br />
<br />
Theoretically, 1 GiB pages works as 2 MiB. But practically - no guaranteed way was found to get contiguous 1 GiB memory blocks. Each consequent request of 1 GiB blocks lead to lesser and lesser dynamically allocated count.<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== Isolating pinned CPUs ===<br />
<br />
CPU pinning by itself won't prevent other host processes from running on the pinned CPUs. Properly isolating the pinned CPUs can reduce latency in the guest VM.<br />
<br />
==== With isolcpus kernel parameter ====<br />
<br />
In this example, let us assume you are using CPUs 4-7.<br />
Use the [[kernel parameters]] {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. For example:<br />
<br />
isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.removeddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this Removeddit mirror of a Reddit thread] for more info. ([https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ The original thread] is worthless because of deleted comments.)<br />
<br />
==== Dynamically isolating CPUs ====<br />
<br />
The isolcpus kernel parameter will permanently reserve CPU cores, even when the guest isn't running. A more flexible alternative is to use the cset tool from {{AUR|cpuset-git}} to dynamically isolate CPUs when starting the guest.<br />
<br />
See this [https://www.redhat.com/archives/vfio-users/2016-September/msg00072.html vfio-users post] for more info, as well as this [https://rokups.github.io/#!pages/gaming-vm-performance.md blog post] and this [https://github.com/PassthroughPOST/VFIO-Tools/blob/master/libvirt_hooks/hooks/cset.sh script] for working examples.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you are running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you are on the latest). If you are running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
Starting with QEMU 3.1 the TOPOEXT cpuid flag is disabled by default. In order to use hyperthreading(SMT) on AMD CPUs you need to manually enable it:<br />
<br />
<cpu mode='host-passthrough' check='none'><br />
<topology sockets='1' cores='4' threads='2'/><br />
<feature policy='require' name='topoext'/><br />
</cpu><br />
<br />
commit: https://git.qemu.org/?p=qemu.git;a=commit;h=7210a02c58572b2686a3a8d610c6628f87864aed<br />
<br />
=== Virtio disk ===<br />
<br />
{{Merge|QEMU#Installing virtio drivers|Off-topic.|section=Moving virtio disk section to QEMU}}<br />
<br />
The default disk types are SATA or IDE emulation out of the box. These controllers offer maximum compatibility but are not suited for efficient virtualization. Two accelerated models exist: {{ic|1=virtio-scsi}} for SCSI emulation and passthrough, or {{ic|1=virtio-blk}} for a more basic block device emulation.<br />
<br />
==== Drivers ====<br />
<br />
* Linux guests should support these out of the box on any modern kernel<br />
* macOS has rudimentary {{ic|1=virtio-blk}} support starting in Mojave via {{ic|1=AppleVirtIO.kext}} but performance is poor and TRIM does not appear to work. SATA is still suggested<br />
* Windows needs the [https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/ Windows virtio drivers]. {{ic|1=virtio-scsi}} uses the {{ic|1=vioscsi}} driver. {{ic|1=virtio-blk}} uses the {{ic|1=viostor}} driver<br />
* Windows can be installed directly onto these disks by selecting 'load driver' on the installer disk selection menu. The windows iso and virtio driver iso should both be attached as regular SATA/IDE cdroms during the installation process<br />
* To switch boot disks to virtio on an existing Windows installation:<br />
** {{ic|1=virtio-blk}}: Add a temporary disk with bus {{ic|1=virtio}}, boot windows & load the driver for the disk, then shutdown and switch the boot disk disk bus to {{ic|1=virtio}}<br />
** {{ic|1=virtio-scsi}}: Add a scsi controller with model {{ic|1=virtio}}, boot windows & load the driver for the controller, then shutdown and switch the boot disk bus to {{ic|1=scsi}} (not virtio)<br />
<br />
==== Considerations ====<br />
<br />
* {{ic|1=virtio-scsi}} TRIM support is mature, all versions should support it. Traditionally, {{ic|1=virtio-scsi}} has been the preferred approach for this reason<br />
* {{ic|1=virtio-blk}} TRIM support is new, this requires requires qemu 4.0+, guest linux kernel 5.0+, guest windows drivers 0.1.173+<br />
* Thin provisioning works by enabling TRIM on a sparse image file: {{ic|1=discard='unmap'}}. Unused blocks will be freed and the disk usage will drop (works on both raw and qcow2). Actual on-disk size of a sparse image file may be checked with {{ic|1=du /path/to/disk.img}}<br />
* Thin provisioning can also work with block storage such as zfs zvols or thin lvm<br />
* Virt queue count will influence the number of threads inside the guest kernel used for IO processing, suggest using {{ic|1=queues='4'}} or more<br />
* Native mode ({{ic|1=io='native'}}) uses a single threaded model based on linux AIO, is a bit more CPU efficient but may have lower peak performance and doesn't allow host side caching to be used<br />
* Threaded mode ({{ic|1=io='threads'}}) will spawn dozens of threads on demand as the disk is used. This is less efficient but may perform better if there are enough host cores available to run them, and allows for host side caching to be used<br />
* Modern versions of libvirt will group the dynamic worker threads created when using threaded mode in with the iothread=1 cgroup for pinning purposes. Very old versions of libvirt left these in the emulator cgroup<br />
<br />
==== IO threads ====<br />
<br />
An IO thread is a dedicated thread for processing disk events, rather than using the main qemu emulator loop. This should not be confused with the worker threads spawned on demand with {{ic|1=io='threads'}}.<br />
<br />
* You can only use one iothread per disk controller. The thread must be assigned to a specific controller with {{ic|1=iothread='X'}} in the {{ic|1=<driver>}} tag. Furthermore, extra & unassigned iothreads will not be used and do nothing<br />
* In the case of {{ic|1=virtio-scsi}}, there is one controller for multiple scsi disks. The iothread is assigned on the controller: {{ic|1=<controller><driver iothread='X'>}}<br />
* In the case of {{ic|1=virtio-blk}}, each disk has its own controller. The iothread is assigned in the driver tag under the disk itself: {{ic|1=<disk><driver iothread='X'>}}<br />
* Since emulated disks incur a significant amount of CPU overhead, that can lead to vcpu stuttering under high disk load (especially high random IOPS). In this case it helps to pin the IO to different core(s) than your vcpus with {{ic|1=<iothreadpin>}}<br />
<br />
==== Examples with libvirt ====<br />
<br />
virtio-scsi + iothread + worker threads + host side writeback caching + full disk block device backend:<br />
<domain><br />
<devices><br />
<disk type='block' device='disk'><br />
<driver name='qemu' type='raw' cache='writeback' io='threads' discard='unmap'/><br />
<source dev='/dev/disk/by-id/ata-Samsung_SSD_840_EVO_1TB_S1D9NSAF206396F'/><br />
<target dev='sda' bus='scsi'/><br />
</disk><br />
<controller type='scsi' index='0' model='virtio-scsi'><br />
<driver iothread='1' queues='8'/><br />
</controller><br />
<br />
virtio-blk + iothread + native aio + no host caching + raw sparse image backend:<br />
<domain><br />
<devices><br />
<disk type='file' device='disk'><br />
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap' iothread='1' queues='8'/><br />
<source dev='/var/lib/libvirt/images/pool/win10.img'/><br />
<target dev='vda' bus='virtio'/><br />
</disk><br />
<br />
Creating the iothreads:<br />
<domain><br />
<iothreads>1</iothreads><br />
<br />
Pinning iothreads:<br />
<domain><br />
<cputune><br />
<iothreadpin iothread='1' cpuset='0-1,6-7'/><br />
<br />
=== Virtio network ===<br />
<br />
The default NIC models rtl8139 or e1000 can be a bottleneck for gigabit+ speeds and have a significant amount of CPU overhead compared to {{ic|1=virtio-net}}.<br />
<br />
* Select {{ic|1=virtio}} as the model for the NIC with libvirt or use the {{ic|1=virtio-net-pci}} device in bare qemu<br />
* Windows needs the {{ic|1=NetKVM}} driver from [https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/ Windows virtio drivers]<br />
* Virtio uses vhost-net by default for in-kernel packet processing without exiting to userspace<br />
* Multiqueue can enabled for a further speedup with multiple connections but typically will not boost single stream speeds. For libvirt add {{ic|1=<driver queues='8'/>}} under the interface tag<br />
* Zero copy transmit may also be enabled on macvtap by setting the module parameter {{ic|1=vhost_net.experimental_zcopytx=1}} but this may actually have worse performance, see [https://github.com/torvalds/linux/commit/098eadce3c622c07b328d0a43dda379b38cf7c5e commit]<br />
<br />
Libvirt example with a bridge:<br />
<br />
<interface type='bridge'><br />
<mac address="52:54:00:6d:6e:2e"/><br />
<source bridge='br0'/><br />
<model type='virtio'/><br />
<driver queues='8'/><br />
</interface><br />
<br />
=== Further tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide].<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how vfio-pci uses your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/local/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/bus/pci/devices/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
DEVS="0000:03:00.0 0000:03:00.1"<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}:<br />
<br />
# Add {{ic|modconf}} to the [[mkinitcpio#HOOKS|HOOKS]] array and {{ic|/usr/local/bin/vfio-pci-override.sh}} to the [[mkinitcpio#BINARIES and FILES|FILES]] array.<br />
<br />
Edit {{ic|/etc/modprobe.d/vfio.conf}}:<br />
<br />
# Add the following line: {{ic|install vfio-pci /usr/local/bin/vfio-pci-override.sh}}<br />
# [[Regenerate the initramfs]] and reboot.<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|This is related to VBIOS issues and should be moved into a separate section regarding VBIOS compatibility.|section=UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that is not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.hostfission.com/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total mebibytes + 2<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MiB + 2 = 17.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 17.82 is bigger than 16 we should choose 32.<br />
<br />
Next create a configuration file to create the shared memory file on boot<br />
<br />
{{hc|/etc/tmpfiles.d/10-looking-glass.conf|2=<br />
f /dev/shm/looking-glass 0660 '''user''' kvm -<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Ask systemd-tmpfiles to create the shared memory file now without waiting to next boot<br />
<br />
# systemd-tmpfiles --create /etc/tmpfiles.d/10-looking-glass.conf<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to go into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download the signed driver [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/upstream-virtio/ from Red Hat].<br />
<br />
Once the driver is installed you must download a matching [https://github.com/gnif/LookingGlass/releases looking-glass-host] that matches the client you installed from AUR and start it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]<br />
It is also possible to make it start automatically on VM boot by editing the {{ic|HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run}} registry and adding a path to the downloaded executable.<br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you do not want to use Spice to control the guest mouse and keyboard you can disable the Spice server.<br />
<br />
$ looking-glass-client -s<br />
<br />
Additionally you may want to start Looking Glass Client as full screen, otherwise the image may be scaled down resulting in poor image fidelity.<br />
<br />
$ looking-glass-client -F<br />
<br />
Launch with the {{ic|--help}} option for further information.<br />
<br />
=== Swap peripherals to and from the Host ===<br />
<br />
Looking Glass includes a Spice client in order to control mouse movement on the Windows guest. However this may have too much latency for certain applications, such as gaming. An alternative method is passing through specific USB devices for minimal latency. This allows for switching the devices between host and guest.<br />
<br />
First create a .xml file for the device(s) you wish to pass-through, which libvirt will use to identify the device.<br />
<br />
{{hc|~/.VFIOinput/input_1.xml|2=<br />
<hostdev mode='subsystem' type='usb' managed='no'><br />
<source><br />
<vendor id='0x[Before Colon]'/><br />
<product id='0x[After Colon]'/><br />
</source><br />
</hostdev><br />
}}<br />
<br />
Replace [Before/After Colon] with the contents of the 'lsusb' command, specific to the device you want to pass-through.<br />
<br />
For instance my mouse is {{ic|Bus 005 Device 002: ID 1532:0037 Razer USA, Ltd}} so I would replace {{ic|vendor id}} with 1532, and {{ic|product id}} with 1037.<br />
<br />
Repeat this process for any additional USB devices you want to pass-through. If your mouse / keyboard has multiple entries in {{ic|lsusb}}, perhaps if it is wireless, then create additional xml files for each.<br />
<br />
{{Note|Do not forget to change the path & name of the script(s) above and below to match your user and specific system.}}<br />
<br />
Next a bash script file is needed to tell libvirt what to attach/detach the USB devices to the guest.<br />
<br />
{{hc|~/.VFIOinput/input_attach.sh|2=<br />
#!/bin/bash<br />
<br />
virsh attach-device [VM-Name] [USBdevice]<br />
}}<br />
<br />
Replace [VM-Name] with the name of your virtual machine, which can be seen under virt-manager. Additionally replace [USBdevice] with the '''full''' path to the .xml file for the device you wish to pass-through. Add additional lines for more than 1 device. For example here is my script:<br />
<br />
{{hc|~/.VFIOinput/input_attach.sh|2=<br />
#!/bin/bash<br />
<br />
virsh attach-device win10 /home/$USER/.VFIOinput/input_mouse.xml<br />
virsh attach-device win10 /home/$USER/.VFIOinput/input_keyboard.xml<br />
}}<br />
<br />
Next duplicate the script file and replace {{ic|attach-device}} with {{ic|detach-device}}. Ensure both scripts are executable with {{ic|chmod +x $script.sh}}<br />
<br />
This 2 script files can now be executed to attach or detach your USB devices from the host to the guest VM. It is important to note that they may need to be executed as root. To run the script from the Windows VM, one possibility is using [[PuTTY]] to [[SSH]] into the host, and execute the script. On Windows PuTTY comes with plink.exe which can execute singular commands over SSH before then logging out, instead of opening a SSH terminal, all in the background.<br />
<br />
{{hc|detach_devices.bat|2=<br />
"C:\Program Files\PuTTY\plink.exe" root@$HOST_IP -pw $ROOTPASSWORD /home/$USER/.VFIOinput/input_detach.sh<br />
}}<br />
<br />
Replace {{ic|$HOST_IP}} with the Host [[Network configuration#IP addresses|IP Address]] and $ROOTPASSWORD with the root password.<br />
<br />
{{warning|This method is insecure if somebody has access to your VM, since they could open the file and read your password. It is advisable to use [[SSH keys]] instead!}}<br />
<br />
You may also want to execute the script files using key binds. On Windows one option is [https://autohotkey.com/ Autohotkey], and on the Host [[Xbindkeys]]. Because of the need to run the scripts as root, you may also need to use [[Polkit]] or [[Sudo]] which can both be used to authenticate specific executables as able to run as root without needing a password.<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream,multifunction}} should break up as many devices as possible.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|edk2-ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/edk2-ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/edk2-ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it is not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It is recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing through other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting: [[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
{{Note|If your USB controller does not support resetting, is not in an isolated group, or is otherwise unable to be passed through then it may still be possible to accomplish similar results through [[udev]] rules. See [https://github.com/olavmrk/usb-libvirt-hotplug] which allows any device connected to specified USB ports to be automatically attached to a virtual machine.}}<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<nowiki> </devices><br />
<qemu:commandline><br />
<qemu:arg value="-audiodev"/><br />
<qemu:arg value="pa,id=snd0,server=/run/user/1000/pulse/native"/><br />
</qemu:commandline><br />
</domain></nowiki><br />
}}<br />
<br />
You can also use the /tmp directory if you have multiple users accessing PulseAudio<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<nowiki> </devices><br />
<qemu:commandline><br />
<qemu:arg value="-audiodev"/><br />
<qemu:arg value="pa,id=snd0,server=unix:/tmp/pulse-socket"/><br />
</qemu:commandline><br />
</domain></nowiki><br />
}}<br />
<br />
You may want to change 1000 under the user directory to your current user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied. Virtual Machine audio will now be routed through the host as an application. A application such as {{Pkg|pavucontrol}} can be used to control the output device.<br />
<br />
==== QEMU 4.0/4.2+ audio changes ====<br />
<br />
As of QEMU 4.0 and made official with [https://www.kraxel.org/blog/2020/01/qemu-sound-audiodev/ QEMU 4.2] the {{ic|1=-audiodev}} parameter is now to be used with PulseAudio sound passthrough, any previous environmental variables you have defined in your XML '''need''' to be removed e.g. {{ic|qemu:env}} to properly use this.<br />
<br />
You will also need to change the chipset accordingly with each QEMU update to how your VM is set up, i.e. {{ic|pc-q35-4.2}} or {{ic|pc-i440fx-4.2}} to advantage of the changes in QEMU:<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<os><br />
<type arch<nowiki>=</nowiki>'x86_64' machine<nowiki>=</nowiki>'pc-q35-4.2'>hvm</type><br />
...<br />
</os><br />
...<br />
<os><br />
<type arch<nowiki>=</nowiki>'x86_64' machine<nowiki>=</nowiki>'pc-i440fx-4.2'>hvm</type><br />
...<br />
</os>}}<br />
<br />
The latest audiodev changes in QEMU also suggest using the ICH9 instead of ICH6 or AC97 for audio emulation to take advantage of this newer code, it is recommended to use the parameters as shown below because Libvirt currently does not define the {{ic|1=id=}} parameter correctly:<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<nowiki><qemu:arg value="-device"/><br />
<qemu:arg value="ich9-intel-hda,bus=pcie.0,addr=0x1b"/><br />
<qemu:arg value="-device"/><br />
<qemu:arg value="hda-micro,audiodev=hda"/><br />
<qemu:arg value="-audiodev"/><br />
<qemu:arg value="pa,id=hda,server=unix:/tmp/pulse-socket"/></nowiki><br />
}}<br />
<br />
QEMU 4.2 also has 5.1 channel audio support via {{ic|usb-audio}} emulation, this can be enabled as shown below:<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<nowiki><qemu:arg value="-device"/><br />
<qemu:arg value="usb-audio,audiodev=usb,multi=on"/><br />
<qemu:arg value="-audiodev"/><br />
<qemu:arg value="pa,id=usb,server=unix:/tmp/pulse-socket,out.mixing-engine=off"/></nowiki><br />
}}<br />
<br />
{{Note|1=<nowiki></nowiki><br />
* You can have multiple audio backends, by simply specifying {{ic|1=-audiodev}} multiple times in your XML and by assigning them different ids. This can be useful for a use case of having two identical backends. With PulseAudio each backend is a separate stream and can be routed to different output devices on the host (using a pulse mixer app like {{Pkg|pavucontrol}} or {{Pkg|pulsemixer}}).<br />
* USB 3 emulation is needed in Libvirt/QEMU to enable the {{ic|usb-audio}} parameter<br />
* It is recommended to enable MSI interrupts with a tool such as [https://github.com/CHEF-KOCH/MSI-utility this] on the ICH9 audio device to mitigate any crackling, stuttering, speedup, or no audio at all after VM restart<br />
* The {{ic|1=-audiodev=hda}} parameter can be changed to your liking, but as shown above, the names must match each other or the VM will not start<br />
* If audio is crackling/stuttering/speedup etc. is still present you may want to adjust parameters such as {{ic|buffer-length}} and {{ic|timer-period}}, more information on these parameters and more can be found in the {{man|1|qemu}} manual<br />
* Some audio chipsets such as [https://bugzilla.kernel.org/show_bug.cgi?id=195303 Realtek alc1220] may also have issues out of the box so do consider this when using any audio emulation with QEMU<br />
* Improper pinning or heavy host usage without using [[#Isolating pinned CPUs|isolcpus]] can also influence sound bugs, especially while gaming in a VM<br />
}}<br />
<br />
=== Passing VM audio to host via Scream ===<br />
<br />
It is possible to pass VM audio through a bridged network such as the one provided by Libvirt or by adding a IVSHMEM device to the host by using a application called [https://github.com/duncanthrax/scream Scream]. This section will only cover using PulseAudio as a receiver on the host.<br />
See the project page for more details and instructions on other methods.<br />
<br />
==== Using Scream with a bridged network ====<br />
<br />
{{Note|<br />
* This is the ''preferred'' way to use this, although results may vary per user<br />
* It is recommend to use the [[#Virtio network]] adapter while using Scream, other virtual adapters provided by QEMU such as '''e1000e''' may lead to poor performance<br />
}}<br />
<br />
To use scream via your network you will want to find your bridge name via {{ic|1=ip -a}}, in most cases it will be called '''br0''' or '''virbr0'''. Below is a example of the command needed to start the Scream application:<br />
<br />
{{bc|<nowiki> $ scream-pulse -i virbr0 &</nowiki>}}<br />
<br />
{{Warning| This will not work with a '''macvtap bridge''' as that doesn't allow host to guest communication, also make sure you have the proper firewall ports open for it to communicate with the VM}}<br />
<br />
==== Adding the IVSHMEM device to use Scream with IVSHMEM ====<br />
<br />
With the VM turned off, edit the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='scream-ivshmem'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>2</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
In the above configuration, the size of the IVSHMEM device is 2MB (the recommended amount). Change this as needed.<br />
<br />
Now refer to [[#Adding IVSHMEM Device to VM]] to configure the host to create the shared memory file on boot, replacing {{ic|looking-glass}} with {{ic|scream-ivshmem}}.<br />
<br />
===== Configuring the Windows guest for IVSHMEM =====<br />
<br />
The correct driver must be installed for the IVSHMEM device on the guest. <br />
See [[#Installing the IVSHMEM Host to Windows guest]]. Ignore the part about {{ic|looking-glass-host}}.<br />
<br />
Install the [https://github.com/duncanthrax/scream/releases Scream] virtual audio driver on the guest. <br />
If you have secure boot enabled for your VM, you may need to disable it. <br />
<br />
Using the registry editor, set the DWORD {{ic|HKLM\SYSTEM\CurrentControlSet\Services\Scream\Options\UseIVSHMEM}} to the size of the IVSHMEM device in MB.<br />
Note that scream identifies its IVSHMEM device using its size, so make sure there is only one device of that size.<br />
<br />
====== Configuring the host ======<br />
<br />
Install {{AUR|scream}}.<br />
<br />
Create the systemd user service file to control the reciever<br />
<br />
{{hc|~/.config/systemd/user/scream-ivshmem-pulse.service|<nowiki><br />
[Unit]<br />
Description=Scream IVSHMEM pulse reciever<br />
After=pulseaudio.service<br />
Wants=pulseaudio.service<br />
<br />
[Service]<br />
Type=simple<br />
ExecStartPre=/usr/bin/truncate -s 0 /dev/shm/scream-ivshmem<br />
ExecStartPre=/usr/bin/dd if=/dev/zero of=/dev/shm/scream-ivshmem bs=1M count=2<br />
ExecStart=/usr/bin/scream-ivshmem-pulse /dev/shm/scream-ivshmem<br />
<br />
[Install]<br />
WantedBy=default.target<br />
<br />
</nowiki>}}<br />
<br />
Edit {{ic|1=count=2}} with the size of the IVSHMEM device in MB.<br />
<br />
Now start the service with <br />
$ systemctl start --user scream-ivshmem-pulse<br />
<br />
To have it automatically start on next login, enable the service<br />
$ systemctl enable --user scream-ivshmem-pulse<br />
<br />
=== Physical disk/partition ===<br />
<br />
Raw and qcow2 especially can have noticeable overhead for heavy IO. A whole disk or a partition may be used directly to bypass the filesystem and improve I/O performance. If you wish to dual boot the guest OS natively you would need to pass the entire disk without any partitioning. It is suggested to use /dev/disk/by- paths to refer to the disk since /dev/sdX entries can change between boots. To find out which disk/partition is associated with the one you would like to pass:<br />
<br />
{{hc|$ ls -l /dev/disk/by-id|<br />
ata-ST1000LM002-9VQ14L_Z0501SZ9 -> ../../sdd<br />
}}<br />
<br />
See [[#Virtio disk]] on how to add these with libvirt XML. You can also add the disk with Virt-Manager's '''Add Hardware''' menu and then type the disk you want in the '''Select or create custom storage''' box, e.g. '''/dev/disk/by-id/ata-ST1000LM002-9VQ14L_Z0501SZ9'''<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functional and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
For many reasons users may seek to see [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]].<br />
<br />
These examples offer a supplement to existing hardware compatibility lists. Additionally, if you have trouble configuring a certain mechanism in your setup, you might find these examples very valuable. Users there have described their setups in detail, and some have provided examples of their configuration files as well. <br />
<br />
We encourage those who successfully build their system from this resource to help improve it by contributing their builds. Due to the many different hardware manufacturers involved, the seemingly significant lack of sufficient documentation, as well as other issues due to the nature of this process, community contributions are necessary.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== QEMU 4.0: Unable to load graphics drivers/BSOD/Graphics stutter after driver install using Q35 ===<br />
<br />
Starting with QEMU 4.0, the Q35 machine type changes the default {{ic|<nowiki>kernel_irqchip</nowiki>}} from {{ic|off}} to {{ic|split}} which breaks some guest devices, such as nVidia graphics (the driver fails to load / black screen / code 43 / graphics stutters, usually when mouse moving). Switch to full KVM mode instead by adding {{ic|<nowiki><ioapic driver='kvm'/></nowiki>}} under libvirt's {{ic|<nowiki><features></nowiki>}} tag in your VM configuration or by adding {{ic|<nowiki>kernel_irqchip=on</nowiki>}} in the {{ic|-machine}} QEMU arg.<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the hypervisor parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. The vendor_id can be [https://libvirt.org/formatdomain.html#elementsFeatures any string value up to 12 characters].<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
=== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ===<br />
<br />
{{Expansion|This error is actually related to the boot_vgs issue and should be merged together with everything else concerning GPU ROMs.|section=UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
In addition try adding kernel parameter {{ic|1=pci=realloc}} which also [https://github.com/Dunedan/mbp-2016-linux/issues/60#issuecomment-396311301 helps with hotplugging issues].<br />
<br />
=== UEFI (OVMF) compatibility in VBIOS ===<br />
<br />
{{Remove|Flashing you guest GPU for the purpose of a GPU passthrough is '''never''' good advice. A full section should be dedicated to VBIOS compatibility.|section= UEFI (OVMF) Compatibility in VBIOS}}<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it is not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it is stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it is pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
On a linux guest you can use modinfo to see if there is option to enable MSI (for example: "modinfo snd_hda_intel |grep msi"). If there is, one can enable it by adding the relevant option to a custom omdprobe file - in "/etc/modprobe.d/snd-hda-intel.conf" inserting "options snd-hda-intel enable_msi=1"<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
A UI tool called [https://github.com/CHEF-KOCH/MSI-utility MSI Utility (FOSS Version 2)] works with Windows 10 64-bit and simplifies the process.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read [https://www.kernel.org/doc/html/latest/x86/intel-iommu.html#graphics-problems intel-iommu.html] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X does not start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user is added to the {{ic|kvm}} and {{ic|libvirt}} [[user group]]s.<br />
<br />
=== Host lockup after VM shutdown ===<br />
<br />
This issue seems to primarily affect users running a Windows 10 guest and usually after the VM has been run for a prolonged period of time: the host will experience multiple CPU core lockups (see [https://bbs.archlinux.org/viewtopic.php?id=206050&p=2]). To fix this try enabling Message Signal Interrupts on the GPU passed through to the guest. A good guide for how to do this can be found in [https://forums.guru3d.com/threads/windows-line-based-vs-message-signaled-based-interrupts.378044/]. You can also download this application for windows here [https://github.com/TechtonicSoftware/MSIInturruptEnabler] that should make the process easier.<br />
<br />
=== Host lockup if guest is left running during sleep ===<br />
<br />
VFIO-enabled virtual machines tend to become unstable if left running through a sleep/wakeup cycle and have been known to cause the host machine to lockup when an attempt is then made to shut them down. In order to avoid this, one can simply prevent the host from going into sleep while the guest is running using the following libvirt hook script and systemd unit. The hook file needs executable permissions to work.<br />
<br />
{{hc|/etc/libvirt/hooks/qemu|<nowiki><br />
#!/bin/bash<br />
<br />
OBJECT="$1"<br />
OPERATION="$2"<br />
SUBOPERATION="$3"<br />
EXTRA_ARG="$4"<br />
<br />
case "$OPERATION" in<br />
"prepare")<br />
systemctl start libvirt-nosleep@"$OBJECT"<br />
;;<br />
"release")<br />
systemctl stop libvirt-nosleep@"$OBJECT"<br />
;;<br />
esac<br />
</nowiki>}}<br />
<br />
{{hc|/etc/systemd/system/libvirt-nosleep@.service|<nowiki><br />
[Unit]<br />
Description=Preventing sleep while libvirt domain "%i" is running<br />
<br />
[Service]<br />
Type=simple<br />
ExecStart=/usr/bin/systemd-inhibit --what=sleep --why="Libvirt domain \"%i\" is running" --who=%U --mode=block sleep infinity<br />
</nowiki>}}<br />
<br />
=== Cannot boot after upgrading ovmf ===<br />
<br />
If you cannot boot after upgrading from {{Pkg|ovmf}}{{Broken package link|replaced by {{Pkg|edk2-ovmf}}}} version 1:r23112.018432f0ce-1 then you need to remove the old {{ic|*VARS.fd}} file in {{ic|/var/lib/libvirt/qemu/nvram/}}:<br />
<br />
# mv /var/lib/libvirt/qemu/nvram/vmname_VARS.fd /var/lib/libvirt/qemu/nvram/vmname_VARS.fd.old<br />
<br />
See {{Bug|57825}} for further details.<br />
<br />
=== QEMU via cli pulseaudio stuttering/delay ===<br />
<br />
Using following flags for the audio device and chipset might help if you are running into the stuttering/delay audio issues when running QEMU via cli:<br />
<br />
qemu-system-x86_64 \<br />
-machine pc-i440fx-3.0 \<br />
-device hda-micro \<br />
-soundhw hda \<br />
-...<br />
<br />
As noted in [[#QEMU 3.0 audio changes|QEMU 3.0 audio changes]]{{Broken section link}} the specified chipset will include a series of audio patches.<br />
<br />
Setting {{ic|QEMU_AUDIO_TIMER_PERIOD}} to values higher than 100 might also help (did not test value lower than 100).<br />
<br />
=== Bluescreen at boot since Windows 10 1803 ===<br />
<br />
Since Windows 10 1803 there is a problem when you are using "host-passthrough" as cpu model. The machine cannot boot and is either boot looping or you get a bluescreen.<br />
You can workaround this by:<br />
<br />
# echo 1 > /sys/module/kvm/parameters/ignore_msrs<br />
<br />
To make it permanently you can create a modprobe file {{ic|kvm.conf}}:<br />
<br />
options kvm ignore_msrs=1<br />
=== AMD Ryzen / BIOS updates (AGESA) yields "Error: internal error: Unknown PCI header type ‘127’" ===<br />
<br />
AMD users have been experiencing breakage of their KVM setups after updating the BIOS on their motherboard. There is a kernel [https://clbin.com/VCiYJ patch], (see [[Kernel/Arch Build System]] for instruction on compiling kernels with custom patches) that can resolve the issue as of now (7/28/19), but this is not the first time AMD has made an error of this very nature, so take this into account if you are considering updating your BIOS in the future as a VFIO user.<br />
=== Host crashes when hotplugging Nvidia card with USB ===<br />
<br />
If attempting to hotplug an Nvidia card with a USB port, you may have to blacklist the {{ic|i2c_nvidia_gpu}} driver. Do this by adding the line {{ic|blacklist i2c_nvidia_gpu}} to {{ic|/etc/modprobe.d/blacklist.conf}}.<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script] from https://www.youtube.com/watch?v=37D2bRsthfI<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [ircs://chat.freenode.net/vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]<br />
* [https://www.reddit.com/r/VFIO /r/VFIO: A subreddit focused on vfio]<br />
* [https://github.com/intel/gvt-linux/wiki/GVTd_Setup_Guide GVT-d: passthrough of an entire integrated GPU]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Users_and_groups&diff=601570Users and groups2020-03-15T18:45:38Z<p>NeoTheFox: Added systemd-homed to related articles</p>
<hr />
<div>[[Category:Security]]<br />
[[fa:کاربران و گروه ها]]<br />
[[de:Benutzer und Gruppen]]<br />
[[es:Users and groups]]<br />
[[fr:Utilisateurs et Groupes]]<br />
[[it:Users and groups]]<br />
[[ja:ユーザーとグループ]]<br />
[[pt:Users and groups]]<br />
[[ru:Users and groups]]<br />
[[sr:Users and groups]]<br />
[[zh-hans:Users and groups]]<br />
[[zh-hant:Users and groups]]<br />
{{Related articles start}}<br />
{{Related|DeveloperWiki:UID / GID Database}}<br />
{{Related|Sudo}}<br />
{{Related|Polkit}}<br />
{{Related|File permissions and attributes}}<br />
{{Related|Systemd-homed}}<br />
{{Related articles end}}<br />
<br />
Users and groups are used on GNU/Linux for [[Wikipedia:access control#Computer security|access control]]—that is, to control access to the system's files, directories, and peripherals. Linux offers relatively simple/coarse access control mechanisms by default. For more advanced options, see [[ACL]] and [[PAM#Configuration How-Tos]].<br />
<br />
== Overview ==<br />
<br />
A ''user'' is anyone who uses a computer. In this case, we are describing the names which represent those users. It may be Mary or Bill, and they may use the names Dragonlady or Pirate in place of their real name. All that matters is that the computer has a name for each account it creates, and it is this name by which a person gains access to use the computer. Some system services also run using restricted or privileged user accounts.<br />
<br />
Managing users is done for the purpose of security by limiting access in certain specific ways. The [[Wikipedia:Superuser|superuser]] (root) has complete access to the operating system and its configuration; it is intended for administrative use only. Unprivileged users can use the [[su]] and [[sudo]] programs for controlled privilege escalation.<br />
<br />
Any individual may have more than one account as long as they use a different name for each account they create. Further, there are some reserved names which may not be used such as "root".<br />
<br />
Users may be grouped together into a "group", and users may be added to an existing group to utilize the privileged access it grants.<br />
<br />
{{Note|The beginner should use these tools carefully and stay away from having anything to do with any other ''existing'' user account, other than their own.}}<br />
<br />
== Permissions and ownership ==<br />
<br />
From [http://ph7spot.com:80/musings/in-unix-everything-is-a-file In UNIX Everything is a File]:<br />
<br />
:The UNIX operating system crystallizes a couple of unifying ideas and concepts that shaped its design, user interface, culture and evolution. One of the most important of these is probably the mantra: "everything is a file," widely regarded as one of the defining points of UNIX.<br />
<br />
:This key design principle consists of providing a unified paradigm for accessing a wide range of input/output resources: documents, directories, hard-drives, CD-ROMs, modems, keyboards, printers, monitors, terminals and even some inter-process and network communications. The trick is to provide a common abstraction for all of these resources, each of which the UNIX fathers called a "file." Since every "file" is exposed through the same API, you can use the same set of basic commands to read/write to a disk, keyboard, document or network device.<br />
<br />
From [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.597.4699&rep=rep1&type=pdf Extending UNIX File Abstraction for General-Purpose Networking]:<br />
<br />
:A fundamental and very powerful, consistent abstraction provided in UNIX and compatible operating systems is the file abstraction. Many OS services and device interfaces are implemented to provide a file or file system metaphor to applications. This enables new uses for, and greatly increases the power of, existing applications — simple tools designed with specific uses in mind can, with UNIX file abstractions, be used in novel ways. A simple tool, such as cat, designed to read one or more files and output the contents to standard output, can be used to read from I/O devices through special device files, typically found under the {{ic|/dev}} directory. On many systems, audio recording and playback can be done simply with the commands, "{{ic|cat /dev/audio > myfile}}" and "{{ic|cat myfile > /dev/audio}}," respectively.<br />
<br />
Every file on a GNU/Linux system is owned by a user and a group. In addition, there are three types of access permissions: read, write, and execute. Different access permissions can be applied to a file's owning user, owning group, and others (those without ownership). One can determine a file's owners and permissions by viewing the long listing format of the [[ls]] command:<br />
<br />
{{hc|$ ls -l /boot/|total 13740<br />
drwxr-xr-x 2 root root 4096 Jan 12 00:33 grub<br />
-rw-r--r-- 1 root root 8570335 Jan 12 00:33 initramfs-linux-fallback.img<br />
-rw-r--r-- 1 root root 1821573 Jan 12 00:31 initramfs-linux.img<br />
-rw-r--r-- 1 root root 1457315 Jan 8 08:19 System.map26<br />
-rw-r--r-- 1 root root 2209920 Jan 8 08:19 vmlinuz-linux<br />
}}<br />
<br />
The first column displays the file's permissions (for example, the file {{ic|initramfs-linux.img}} has permissions {{ic|-rw-r--r--}}). The third and fourth columns display the file's owning user and group, respectively. In this example, all files are owned by the ''root'' user and the ''root'' group.<br />
<br />
{{hc|$ ls -l /media/|total 16<br />
drwxrwx--- 1 root vboxsf 16384 Jan 29 11:02 sf_Shared<br />
}}<br />
<br />
In this example, the {{ic|sf_Shared}} directory is owned by the ''root'' user and the ''vboxsf'' group. It is also possible to determine a file's owners and permissions using the stat command:<br />
<br />
Owning user:<br />
<br />
{{hc|$ stat -c %U /media/sf_Shared/|<br />
root<br />
}}<br />
<br />
Owning group:<br />
<br />
{{hc|$ stat -c %G /media/sf_Shared/|<br />
vboxsf<br />
}}<br />
<br />
Access rights:<br />
<br />
{{hc|$ stat -c %A /media/sf_Shared/|<br />
drwxrwx---<br />
}}<br />
<br />
Access permissions are displayed in three groups of characters, representing the permissions of the owning user, owning group, and others, respectively. For example, the characters {{ic|-rw-r--r--}} indicate that the file's owner has read and write permission, but not execute ({{ic|rw-}}), whilst users belonging to the owning group and other users have only read permission ({{ic|r--}} and {{ic|r--}}). Meanwhile, the characters {{ic|drwxrwx---}} indicate that the file's owner and users belonging to the owning group all have read, write, and execute permissions ({{ic|rwx}} and {{ic|rwx}}), whilst other users are denied access ({{ic|---}}). The first character represents the file's type.<br />
<br />
List files owned by a user or group with the ''find'' utility:<br />
<br />
# find / -group ''groupname''<br />
<br />
# find / -group ''groupnumber''<br />
<br />
# find / -user ''user''<br />
<br />
A file's owning user and group can be changed with the [[chown]] (change owner) command. A file's access permissions can be changed with the [[chmod]] (change mode) command.<br />
<br />
See {{man|1|chown}}, {{man|1|chmod}}, and [http://www.linux.com/tutorials/understanding-linux-file-permissions Linux file permissions] for additional detail.<br />
<br />
== Shadow ==<br />
<br />
{{Expansion|Mention {{ic|shadow.timer}} and {{ic|shadow.service}}.}}<br />
<br />
The user, group and password management tools on Arch Linux come from the {{Pkg|shadow}} package, which is a dependency of the {{Pkg|base}} [[meta package]].<br />
<br />
== File list ==<br />
<br />
{{Warning|Do not edit these files by hand. There are utilities that properly handle locking and avoid invalidating the format of the database. See [[#User management]] and [[#Group management]] for an overview.}}<br />
<br />
{| class="wikitable"<br />
! File || Purpose<br />
|-<br />
| {{ic|/etc/shadow}} || Secure user account information<br />
|-<br />
| {{ic|/etc/passwd}} || User account information<br />
|-<br />
| {{ic|/etc/gshadow}} || Contains the shadowed information for group accounts<br />
|-<br />
| {{ic|/etc/group}} || Defines the groups to which users belong<br />
|}<br />
<br />
== User management ==<br />
<br />
To list users currently logged on the system, the ''who'' command can be used. To list all existing user accounts including their properties stored in the [[#User database|user database]], run {{ic|passwd -Sa}} as root. See {{man|1|passwd}} for the description of the output format.<br />
<br />
To add a new user, use the ''useradd'' command:<br />
<br />
# useradd -m -G ''additional_groups'' -s ''login_shell'' ''username''<br />
<br />
; {{ic|-m}}/{{ic|--create-home}}: creates the user home directory as {{ic|/home/''username''}}. Within their home directory, a non-root user can write files, delete them, install programs, and so on.<br />
; {{ic|-G}}/{{ic|--groups}}: introduces a list of supplementary groups which the user is also a member of. Each group is separated from the next by a comma, with no intervening spaces. The default is for the user to belong only to the initial group.<br />
; {{ic|-s}}/{{ic|--shell}}: defines the path and file name of the user's default login shell. After the boot process is complete, the default login shell is the one specified here. Ensure the chosen shell package is installed if choosing something other than [[Bash]].<br />
<br />
{{Warning|In order to be able to log in, the login shell must be one of those listed in {{ic|/etc/shells}}, otherwise the [[PAM]] module {{ic|pam_shell}} will deny the login request. In particular, do not use the {{ic|/usr/bin/bash}} path instead of {{ic|/bin/bash}}, unless it is properly configured in {{ic|/etc/shells}}; see {{Bug|33677}}.}}<br />
<br />
{{Note|The password for the newly created user must then be defined, using ''passwd'' as shown in [[#Example adding a user]].}}<br />
<br />
If an initial login group is specified by name or number, it must refer to an already existing group. If not specified, the behaviour of ''useradd'' will depend on the {{ic|USERGROUPS_ENAB}} variable contained in {{ic|/etc/login.defs}}. The default behaviour ({{ic|USERGROUPS_ENAB yes}}) is to create a group with the same name as the username.<br />
<br />
When the login shell is intended to be non-functional, for example when the user account is created for a specific service, {{ic|/usr/bin/nologin}} may be specified in place of a regular shell to politely refuse a login (see {{man|8|nologin}}).<br />
<br />
See {{man|8|useradd}} for other supported options.<br />
<br />
=== Example adding a user ===<br />
<br />
To add a new user named {{ic|archie}}, creating its home directory and otherwise using all the defaults in terms of groups, folder names, shell used and various other parameters:<br />
<br />
# useradd -m archie<br />
<br />
{{Tip|The default value used for the login shell of the new account can be displayed using {{ic|useradd --defaults}}. The default is Bash, a different shell can be specified with the {{ic|-s}}/{{ic|--shell}} option; see {{ic|/etc/shells}} for valid login shells.}}<br />
<br />
Although it is not required to protect the newly created user {{ic|archie}} with a password, it is highly recommended to do so:<br />
<br />
# passwd archie<br />
<br />
The above ''useradd'' command will also automatically create a group called {{ic|archie}} and makes this the default group for the user {{ic|archie}}. Making each user have their own group (with the group name same as the user name) is the preferred way to add users.<br />
<br />
You could also make the default group something else using the {{ic|-g}} option, but note that, in multi-user systems, using a single default group (e.g. {{ic|users}}) for every user is not recommended. The reason is that typically, the method for facilitating shared write access for specific groups of users is setting user [[umask]] value to {{ic|002}}, which means that the default group will by default always have write access to any file you create. See also [https://security.ias.edu/how-and-why-user-private-groups-unix User Private Groups]. If a user must be a member of a specific group specify that group as a supplementary group when creating the user.<br />
<br />
In the recommended scenario, where the default group has the same name as the user name, all files are by default writeable only for the user who created them. To allow write access to a specific group, shared files/folders can be made writeable by default for everyone in this group and the owning group can be automatically fixed to the group which owns the parent directory by setting the setgid bit on this directory:<br />
<br />
# chmod g+s ''our_shared_directory''<br />
<br />
Otherwise the file creator's default group (usually the same as the user name) is used. <br />
<br />
If a GID change is required temporarily you can also use the ''newgrp'' command to change the user's default GID to another GID at runtime. For example, after executing {{ic|newgrp ''groupname''}} files created by the user will be associated with the {{ic|''groupname''}} GID, without requiring a re-login. To change back to the default GID, execute ''newgrp'' without a groupname.<br />
<br />
=== Example adding a system user ===<br />
<br />
System users can be used to run processes/daemons under a different user, protecting (e.g. with [[chown]]) files and/or directories and more examples of computer hardening.<br />
<br />
With the following command a system user without shell access and without a {{ic|home}} directory is created (optionally append the {{ic|-U}} parameter to create a group with the same name as the user, and add the user to this group):<br />
<br />
# useradd -r -s /usr/bin/nologin ''username''<br />
<br />
If the system user requires a specific user and group ID, specify them with the {{ic|-u}}/{{ic|--uid}} and {{ic|-g}}/{{ic|--gid}} options when creating the user:<br />
<br />
# useradd -r -u 850 -g 850 -s /usr/bin/nologin ''username''<br />
<br />
=== Change a user's login name or home directory ===<br />
<br />
To change a user's home directory:<br />
<br />
# usermod -d /my/new/home -m ''username''<br />
<br />
The {{ic|-m}} option also automatically creates the new directory and moves the content there.<br />
<br />
{{Tip|You can create a link from the user's former home directory to the new one. Doing this will allow programs to find files that have hardcoded paths.<br />
<br />
# ln -s /my/new/home/ /my/old/home<br />
<br />
Make sure there is '''no''' trailing {{ic|/}} on {{ic|/my/old/home}}.<br />
}}<br />
<br />
To change a user's login name:<br />
<br />
# usermod -l ''newname'' ''oldname''<br />
<br />
{{Warning|Make certain that you are not logged in as the user whose name you are about to change. Open a new tty (e.g. {{ic|Ctrl+Alt+F6}}) and log in as root or as another user and [[General recommendations#Privilege elevation|elevate to root]]. ''usermod'' should prevent you from doing this by mistake.}}<br />
<br />
Changing a username is safe and easy when done properly, just use the [[#Other examples of user management|usermod]] command. If the user is associated to a group with the same name, you can rename this with the [[#Group management|groupmod]] command.<br />
<br />
Alternatively, the {{ic|/etc/passwd}} file can be edited directly, see [[#User database]] for an introduction to its format.<br />
<br />
Also keep in mind the following notes:<br />
<br />
* If you are using [[sudo]] make sure you update your {{ic|/etc/sudoers}} to reflect the new username(s) (via the ''visudo'' command as root).<br />
* Personal [[cron#Crontab_format|crontabs]] need to be adjusted by renaming the user's file in {{ic|/var/spool/cron}} from the old to the new name, and then opening {{ic|crontab -e}} to change any relevant paths and have it adjust the file permissions accordingly.<br />
* [[Wine|Wine's]] personal folders/files' contents in {{ic|~/.wine/drive_c/users}}, {{ic|~/.local/share/applications/wine/Programs}} and possibly more need to be manually renamed/edited.<br />
* Certain Thunderbird addons, like [https://www.enigmail.net/index.php/en/ Enigmail], may need to be reinstalled.<br />
* Anything on your system (desktop shortcuts, shell scripts, etc.) that uses an absolute path to your home dir (i.e. {{ic|/home/oldname}}) will need to be changed to reflect your new name. To avoid these problems in shell scripts, simply use the {{ic|~}} or {{ic|$HOME}} variables for home directories.<br />
* Also do not forget to edit accordingly the configuration files in {{ic|/etc}} that relies on your absolute path (i.e. Samba, CUPS, so on). A nice way to learn what files you need to update involves using the grep command this way: {{ic|grep -r {old_user} *}}<br />
<br />
=== Other examples of user management ===<br />
<br />
To add a user to other groups use ({{ic|''additional_groups''}} is a comma-separated list):<br />
<br />
# usermod -aG ''additional_groups'' ''username''<br />
<br />
{{Warning|If the {{ic|-a}} option is omitted in the ''usermod'' command above, the user is removed from all groups not listed in {{ic|''additional_groups''}} (i.e. the user will be member only of those groups listed in {{ic|''additional_groups''}}).}}<br />
<br />
To enter user information for the [[#User database|GECOS]] comment (e.g. the full user name), type:<br />
<br />
# chfn ''username''<br />
<br />
(this way ''chfn'' runs in interactive mode).<br />
<br />
Alternatively the GECOS comment can be set more liberally with:<br />
<br />
# usermod -c "Comment" ''username''<br />
<br />
To mark a user's password as expired, requiring them to create a new password the first time they log in, type:<br />
<br />
# chage -d 0 ''username''<br />
<br />
User accounts may be deleted with the ''userdel'' command:<br />
<br />
# userdel -r ''username''<br />
<br />
The {{ic|-r}} option specifies that the user's home directory and mail spool should also be deleted.<br />
<br />
To change the user's login shell:<br />
<br />
# usermod -s ''/bin/bash'' ''username''<br />
<br />
{{Tip|The {{AUR|adduser}} script allows carrying out the jobs of ''useradd'', ''chfn'' and ''passwd'' interactively. See also {{Bug|32893}}.}}<br />
<br />
== User database ==<br />
<br />
Local user information is stored in the plain-text {{ic|/etc/passwd}} file: each of its lines represents a user account, and has seven fields delimited by colons.<br />
<br />
''account:password:UID:GID:GECOS:directory:shell''<br />
<br />
Where:<br />
<br />
* {{ic|''account''}} is the user name. This field can not be blank. Standard *NIX naming rules apply.<br />
* {{ic|''password''}} is the user password. {{Warning|1=The {{ic|passwd}} file is world-readable, so storing passwords (hashed or otherwise) in this file is insecure. Instead, Arch Linux uses [[Security#Password hashes|shadowed passwords]]: the {{ic|password}} field will contain a placeholder character ({{ic|x}}) indicating that the hashed password is saved in the access-restricted file {{ic|/etc/shadow}}. For this reason it is recommended to always change passwords using the '''passwd''' command.}}<br />
* {{ic|''UID''}} is the numerical user ID. In Arch, the first login name (after root) for a so called normal user, as opposed to services, is UID 1000 by default; subsequent UID entries for users should be greater than 1000.<br />
* {{ic|''GID''}} is the numerical primary group ID for the user. Numeric values for GIDs are listed in [[#Group management|/etc/group]].<br />
* {{ic|''GECOS''}} is an optional field used for informational purposes; usually it contains the full user name, but it can also be used by services such as ''finger'' and managed with the [[#Other examples of user management|chfn]] command. This field is optional and may be left blank.<br />
* {{ic|''directory''}} is used by the login command to set the {{ic|$HOME}} environment variable. Several services with their own users use {{ic|/}}, but normal users usually set a folder under {{ic|/home}}.<br />
* {{ic|''shell''}} is the path to the user's default [[command shell]]. This field is optional and defaults to {{ic|/bin/bash}}.<br />
<br />
Example:<br />
<br />
jack:x:1001:1003:Jack Smith,some comment here,,:/home/jack:/bin/bash<br />
<br />
Broken down, this means: user {{ic|jack}}, whose password is in {{ic|/etc/shadow}}, whose UID is 1001 and whose primary group is 1003. Jack Smith is his full name and there is a comment associated to his account; his home directory is {{ic|/home/jack}} and he is using [[Bash]].<br />
<br />
The ''pwck'' command can be used to verify the integrity of the user database. It can sort the user list by GID at the same time, which can be helpful for comparison:<br />
<br />
# pwck -s<br />
<br />
{{Warning|Arch Linux defaults of the files are created as ''.pacnew'' files by new releases of the {{Pkg|filesystem}} package. Unless Pacman outputs related messages for action, these ''.pacnew'' files can, and should, be disregarded/removed. New required default users and groups are added or re-added as needed by {{man|8|systemd-sysusers}} or the package install script.}}<br />
<br />
== Group management ==<br />
<br />
{{ic|/etc/group}} is the file that defines the groups on the system (see {{man|5|group}} for details). There is also its companion {{ic|gshadow}} which is rarely used. Its details are at {{man|5|gshadow}}.<br />
<br />
Display group membership with the {{ic|groups}} command:<br />
<br />
$ groups ''user''<br />
<br />
If {{ic|''user''}} is omitted, the current user's group names are displayed.<br />
<br />
The {{ic|id}} command provides additional detail, such as the user's UID and associated GIDs:<br />
<br />
$ id ''user''<br />
<br />
To list all groups on the system:<br />
<br />
$ cat /etc/group<br />
<br />
Create new groups with the {{ic|groupadd}} command:<br />
<br />
# groupadd ''group''<br />
<br />
Add users to a group with the {{ic|gpasswd}} command (see {{Bug|58262}} regarding errors; alternatively the {{ic|usermod}} command may be used):<br />
<br />
# gpasswd -a ''user'' ''group''<br />
<br />
Modify an existing group with {{ic|groupmod}}; e.g. to rename {{ic|old_group}} group to {{ic|new_group}} whilst preserving gid (all files previously owned by {{ic|old_group}} will be owned by {{ic|new_group}}):<br />
<br />
# groupmod -n ''new_group'' ''old_group''<br />
<br />
{{Note|This will change a group name but not the numerical GID of the group.}}<br />
<br />
To delete existing groups:<br />
<br />
# groupdel ''group''<br />
<br />
To remove users from a group:<br />
<br />
# gpasswd -d ''user'' ''group''<br />
<br />
{{Note|If the user is currently logged in, he must log out and in again for the change to take effect.}}<br />
<br />
The ''grpck'' command can be used to verify the integrity of the system's group files. <br />
<br />
{{Warning|Arch Linux defaults of the files are created as ''.pacnew'' files by new releases of the {{Pkg|filesystem}} package. Unless Pacman outputs related messages for action, these ''.pacnew'' files can, and should, be disregarded/removed. New required default users and groups are added or re-added as needed by {{man|8|systemd-sysusers}} or the package install script.}}<br />
<br />
== Group list ==<br />
<br />
This section explains the purpose of the essential groups from the [https://git.archlinux.org/svntogit/packages.git/tree/trunk/group?h=packages/filesystem core/filesystem] package. There are many other groups, which will be created with [[DeveloperWiki:UID / GID Database|correct GID]] when the relevant package is installed. See the main page for the software for details.<br />
<br />
{{Note|A later removal of a package does not remove the automatically created user/group (UID/GID) again. This is intentional because any files created during its usage would otherwise be left orphaned as a potential security risk.}}<br />
<br />
=== User groups ===<br />
<br />
Non-root workstation/desktop users often need to be added to some of following groups to allow access to hardware peripherals and facilitate system administration:<br />
<br />
{| class="wikitable"<br />
! Group || Affected files || Purpose<br />
|-<br />
| adm || || Administration group, commonly used to give read access to protected logs. It has full read access to [[journal]] files.<br />
|-<br />
| ftp || {{ic|/srv/ftp/}} || Access to files served by [[List of applications/Internet#FTP servers|FTP servers]].<br />
|-<br />
| games || {{ic|/var/games}} || Access to some game software.<br />
|-<br />
| http || {{ic|/srv/http/}} || Access to files served by [[List of applications/Internet#Web servers|HTTP servers]].<br />
|-<br />
| log || || Access to log files in {{ic|/var/log/}} created by [[syslog-ng]].<br />
|-<br />
| rfkill || {{ic|/dev/rfkill}} || Right to control wireless devices power state (used by [[rfkill]]).<br />
|-<br />
| sys || || Right to administer printers in [[CUPS]].<br />
|-<br />
| systemd-journal || {{ic|/var/log/journal/*}} || Can be used to provide read-only access to the systemd logs, as an alternative to {{ic|adm}} and {{ic|wheel}} [https://cgit.freedesktop.org/systemd/systemd/tree/README?id=fdbbf0eeda929451e2aaf34937a72f03a225e315#n190]. Otherwise, only user generated messages are displayed.<br />
|-<br />
| uucp || {{ic|/dev/ttyS[0-9]+}}, {{ic|/dev/tts/[0-9]+}}, {{ic|/dev/ttyUSB[0-9]+}}, {{ic|/dev/ttyACM[0-9]+}}, {{ic|/dev/rfcomm[0-9]+}} || RS-232 serial ports and devices connected to them.<br />
|-<br />
| wheel || || Administration group, commonly used to give privileges to perform administrative actions. It has full read access to [[journal]] files and the right to administer printers in [[CUPS]]. Can also be used to give access to the [[sudo]] and [[su]] utilities (neither uses it by default).<br />
|}<br />
<br />
=== System groups ===<br />
<br />
The following groups are used for system purposes, an assignment to users is only required for dedicated purposes:<br />
<br />
{| class="wikitable"<br />
! Group || Affected files || Purpose<br />
|-<br />
| dbus || || used internally by {{Pkg|dbus}}<br />
|-<br />
| kmem || {{ic|/dev/port}}, {{ic|/dev/mem}}, {{ic|/dev/kmem}} ||<br />
|-<br />
| locate || {{ic|/usr/bin/locate}}, {{ic|/var/lib/locate}}, {{ic|/var/lib/mlocate}}, {{ic|/var/lib/slocate}} || See [[Locate]].<br />
|-<br />
| lp || {{ic|/dev/lp[0-9]*}}, {{ic|/dev/parport[0-9]*}} || Access to parallel port devices (printers and others).<br />
|-<br />
| mail || {{ic|/usr/bin/mail}} ||<br />
|-<br />
| nobody || || Unprivileged group.<br />
|-<br />
| proc || {{ic|/proc/''pid''/}} || A group authorized to learn processes information otherwise prohibited by {{ic|1=hidepid=}} mount option of the [https://www.kernel.org/doc/Documentation/filesystems/proc.txt proc filesystem]. The group must be explicitly set with the {{ic|1=gid=}} mount option.<br />
|-<br />
| root || {{ic|/*}} || Complete system administration and control (root, admin).<br />
|-<br />
| smmsp || || [[sendmail]] group.<br />
|-<br />
| tty || {{ic|/dev/tty}}, {{ic|/dev/vcc}}, {{ic|/dev/vc}}, {{ic|/dev/ptmx}} ||<br />
|-<br />
| utmp || {{ic|/run/utmp}}, {{ic|/var/log/btmp}}, {{ic|/var/log/wtmp}} ||<br />
|-<br />
|}<br />
<br />
=== Pre-systemd groups ===<br />
<br />
Before arch migrated to [[systemd]], users had to be manually added to these groups in order to be able to access the corresponding devices. This way has been deprecated in favour of [[udev]] marking the devices with a {{ic|uaccess}} [https://github.com/systemd/systemd/blob/master/src/login/70-uaccess.rules.m4 tag] and ''logind'' assigning the permissions to users dynamically via [[ACL]]s according to which session is currently active. Note that the session must not be broken for this to work (see [[General troubleshooting#Session permissions]] to check it).<br />
<br />
There are some notable exceptions which require adding a user to some of these groups: for example if you want to allow users to access the device even when they are not logged in. However, note that adding users to the groups can even cause some functionality to break (for example, the {{ic|audio}} group will break fast user switching and allows applications to block software mixing).<br />
<br />
{| class="wikitable"<br />
! Group || Affected files || Purpose<br />
|-<br />
| audio || {{ic|/dev/audio}}, {{ic|/dev/snd/*}}, {{ic|/dev/rtc0}} || Direct access to sound hardware, for all sessions. It is still required to make [[ALSA]] and [[OSS]] work in remote sessions, see [[ALSA#User privileges]]. Also used in [[JACK]] to give users realtime processing permissions.<br />
|-<br />
| disk || {{ic|/dev/sd[a-zA-Z]*[1-9]*}} || Access to block devices not affected by other groups such as {{ic|optical}}, {{ic|floppy}}, and {{ic|storage}}.<br />
|-<br />
| floppy || {{ic|/dev/fd[0-9]*}} || Access to floppy drives.<br />
|-<br />
| input || {{ic|/dev/input/event[0-9]*}}, {{ic|/dev/input/mouse[0-9]*}} || Access to input devices. Introduced in systemd 215 [http://lists.freedesktop.org/archives/systemd-commits/2014-June/006343.html].<br />
|-<br />
| kvm || {{ic|/dev/kvm}} || Access to virtual machines using [[KVM]].<br />
|-<br />
| optical || {{ic|/dev/sr[0-9]}}, {{ic|/dev/sg[0-9]}} || Access to optical devices such as CD and DVD drives.<br />
|-<br />
| scanner || {{ic|/var/lock/sane}} || Access to scanner hardware.<br />
|-<br />
| storage || {{ic|/dev/st[0-9]*[lma]*}}, {{ic|/dev/nst[0-9]*[lma]*}}|| Used to gain access to removable drives such as USB hard drives, flash/jump drives, MP3 players; enables the user to mount storage devices.[https://lists.archlinux.org/pipermail/arch-dev-public/2011-August/021468.html] Now solely for direct access to tapes if no custom udev rules is involved.[https://git.archlinux.org/svntogit/packages.git/tree/trunk/0001-Use-Arch-Linux-device-access-groups.patch?h=packages/systemd][https://git.archlinux.org/svntogit/packages.git/commit/?id=26af78b6ad554dbd00615600edc7d53164ef8dee][https://github.com/systemd/systemd/blame/aaaf42cb44d4fcd598fca441a11856f3e8dd06d8/rules.d/50-udev-default.rules.in#L69][https://elixir.bootlin.com/linux/v5.5.6/source/include/scsi/scsi_proto.h#L251]<br />
|-<br />
| video || {{ic|/dev/fb/0}}, {{ic|/dev/misc/agpgart}} || Access to video capture devices, 2D/3D hardware acceleration, framebuffer ([[X]] can be used ''without'' belonging to this group).<br />
|}<br />
<br />
=== Unused groups ===<br />
<br />
The following groups are currently not used for any purpose: <br />
<br />
{| class="wikitable"<br />
! Group || Affected files || Purpose<br />
|-<br />
| bin || none || Historical<br />
|-<br />
| daemon || ||<br />
|-<br />
| lock || || Used for lockfile access. Required by e.g. {{Pkg|gnokii}}.<br />
|-<br />
| mem || ||<br />
|-<br />
| network || || Unused by default. Can be used e.g. for granting access to NetworkManager (see [[NetworkManager#Set up PolicyKit permissions]]). <br />
|-<br />
| power || ||<br />
|-<br />
| uuidd || ||<br />
|-<br />
| users || || The primary group for users when user private groups are not used (generally not recommended), e.g. when creating users with {{ic|USERGROUPS_ENAB no}} in {{ic|/etc/login.defs}} or the {{ic|-N}}/{{ic|--no-user-group}} option of ''useradd''.<br />
|}<br />
<br />
== Other tools related to these databases ==<br />
<br />
{{Merge|#Shadow|Separate section does not make sense.}}<br />
<br />
{{Accuracy|Does Arch Linux really prefer ''chage'' over {{ic|vipw -s}} and {{ic|vigr -s}}? The bug {{bug|31414}} (from 2012) simply mentions ''vipw'' and ''vigr'' are both provided by {{pkg|util-linux}} rather than {{Pkg|shadow}}.|section=Utilities to handle the shadow file in acrh}}<br />
<br />
{{man|1|getent}} can be used to read a particular record.<br />
<br />
% getent group tty<br />
<br />
As warned in [[#User database]], using specific utilities such as {{ic|passwd}} and {{ic|chfn}}, is a better way to change the databases. Nevertheless, there are times when editing them directly is looked after. For those times, {{ic|vipw}}, {{ic|vigr}} are provided. It is strongly recommended to use these tailored editors over using a general text editor as they lock the databases against concurrent editing. They also help prevent invalid entries and/or syntax errors. Note that Arch Linux prefers usage of specific tools, such as ''chage'', for modifying the shadow database over using {{ic|vipw -s}} and {{ic|vigr -s}} from the shadow-utils suite. See also {{bug|31414}}.</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Talk:Systemd-homed&diff=601142Talk:Systemd-homed2020-03-11T19:13:12Z<p>NeoTheFox: /* Enabling PAM modules: Don't make unnecessary changes to /etc/pam.d/system-auth */ replied to PAM question</p>
<hr />
<div>== About the category ==<br />
<br />
I view this page as an alternative to [[Users_and_groups]], and that one only has Security category. This was my only reasoning to pick a category, if it's not appropriate let's change it. [[User:NeoTheFox|NeoTheFox]] ([[User talk:NeoTheFox|talk]]) 12:41, 10 March 2020 (UTC)<br />
<br />
== Enabling PAM modules: Don't make unnecessary changes to /etc/pam.d/system-auth ==<br />
<br />
The PAM config on this page is copied from pam_systemd_home.8, but does not consider the other config files in /etc/pam.d. pam_loginuid and pam_keyinit should not be used by sudo [1][2] but will be if this config is used. Instead, the config should be something like the following to closer match the file from pambase [3]<br />
<br />
{{bc|#%PAM-1.0<br />
<br />
auth sufficient pam_unix.so try_first_pass nullok<br />
-auth sufficient pam_systemd_home.so<br />
auth optional pam_permit.so<br />
auth required pam_env.so<br />
<br />
-account sufficient pam_systemd_home.so<br />
account required pam_unix.so<br />
account optional pam_permit.so<br />
account required pam_time.so<br />
<br />
-password sufficient pam_systemd_home.so<br />
password required pam_unix.so try_first_pass nullok sha512 shadow<br />
password optional pam_permit.so<br />
<br />
-session optional pam_systemd_home.so<br />
session required pam_limits.so<br />
session required pam_unix.so<br />
session optional pam_permit.so}}<br />
<br />
[1] https://linux.die.net/man/8/pam_loginuid<br />
<br />
[2] https://linux.die.net/man/8/pam_keyinit<br />
<br />
[3] https://git.archlinux.org/svntogit/packages.git/tree/trunk/system-auth?h=packages/pambase<br />
<br />
[[User:Sambazley|Sambazley]] ([[User talk:Sambazley|talk]]) 19:06, 11 March 2020 (UTC)<br />
<br />
: This configuration leads to root being able to login without password. Just try logging in as root and pressing enter a few time instead of a password. Someone noticed it on reddit. [[User:NeoTheFox|NeoTheFox]] ([[User talk:NeoTheFox|talk]]) 19:12, 11 March 2020 (UTC)</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Systemd-homed&diff=601128Systemd-homed2020-03-11T13:29:55Z<p>NeoTheFox: /* Enabling PAM modules */ PAM section improvements</p>
<hr />
<div>{{lowercase title}}<br />
[[Category:Security]]<br />
{{Related articles start}}<br />
{{Related|fscrypt}}<br />
{{Related|Security}}<br />
{{Related|PAM}}<br />
{{Related articles end}}<br />
<br />
{{Style|See [[Help:Style]]. Why is it in [[:Category:Security]]?}}<br />
<br />
'''systemd-homed''' is a [[systemd]] service providing portable human-user accounts that aren't dependent on current system configuration.<br />
<br />
It achieves portability by moving all user-related information into a storage medium, preferably encrypted, and creating an {{ic|~/.identity}} file that contains signed information about the user - password, what groups they belong to, UID/GID and other information that would typically be scattered over multiple files in {{ic|/}}.<br />
<br />
This approach allows not only for a home directory portability, but also provides security by automatically managing {{ic|/home}} encryption on login and locking the folder if the system is suspended. <br />
<br />
== Installation ==<br />
<br />
''systemd-homed'' is part of and packaged with {{Pkg|systemd}}.<br />
<br />
== Utilities ==<br />
<br />
=== homectl === <br />
<br />
Create/update/inspect users, their home directories and {{ic|~/.identity}} files controlled by [[systemd-homed]] service.<br />
<br />
Create user named 'archie' using the first free UID, create group with the same name and GID equal to UID and make ''archie'' its member, set to {{ic|/bin/bash}} as default shell:<br />
<br />
{{bc|homectl create archie}}<br />
<br />
In above case the home directory mount point is set to {{ic|/home/archie}}, the storage mechanism is chosen in this order: '''LUKS''' if supported or '''subvolume''' if supported or '''directory'''. Image path for '''LUKS''' mechanism is set to {{ic|/home/archie.home}}, directory path for '''directory''' mechanism is set to {{ic|/home/archie.homedir}}.<br />
<br />
{{bc|1=homectl create archie --shell='/usr/bin/zsh'}}<br />
<br />
Same as before but with shell set to ''zsh''.<br />
<br />
{{bc|homectl update archie -G wheel}}<br />
<br />
Add the previously created user 'archie' to 'wheel' group.<br />
<br />
{{Note| '''homectl''' does not manage groups creation/deletion other than those matching the name and ID of users managed by it, use classic tools for that}}<br />
<br />
{{Warning|The {{ic|~/.identity}} files are signed and must not be edited directly with a text editor as it will break the signature and render them invalid. Use {{ic|1=homectl update --identity=/path/to/.identity}} to modify it.}}<br />
<br />
=== userdbctl ===<br />
<br />
A query tool used to inspect users, groups and group memberships provided by both classic unix mechanisms and [[systemd-homed]].<br />
<br />
== Storage mechanism ==<br />
<br />
=== Directory or [[btrfs]] subvolume ===<br />
<br />
A user home folder is stored in {{ic|/home/user.homedir}} and mounted to {{ic|/home/user}} using bind [[mount]] on unlocking. When this method is used no encryption is provided. <br />
To use this mechanism provide {{ic|1=--storage=directory}} or {{ic|1=--storage=subvolume}} to '''homectl'''.<br />
<br />
=== [[fscrypt]] directory ===<br />
<br />
{{Tip| Filesystems with fscrypt support include [[ext4]] and [[f2fs]]}}<br />
A user home folder is stored the same way as when using the above method, but this time a native filesystem encryption is used. <br />
To use this mechanism provide {{ic|1=--storage=fscrypt}} to '''homectl'''.<br />
<br />
=== [[LUKS]] home directory ===<br />
<br />
A user home folder is stored in a linux [[filesystem]], inside a [[LUKS]] volume inside a loopback file or any removable media. <br />
To use this mechanism provide {{ic|1=--storage=luks}} to ```homectl```.<br />
<br />
If you are using removable media make sure that these conditions are met:<br />
<br />
* The image contains a GPT partition table. For now it should only contain a single partition, and that partition must have the type {{ic|UUID 773f91ef-66d4-49b5-bd83-d683bf40ad16}}. It’s partition label must be the user name. <br />
<br />
* This partition must contain a LUKS2 volume, whose label must be the user name. The LUKS2 volume must contain a LUKS2 token field of type systemd-homed. The JSON data of this token must have a record field, containing a string with base64-encoded data. This data is the JSON user record, in the same serialization as in ~/.identity, though encrypted. The JSON data of this token must also have an iv field, which contains a base64-encoded binary initialization vector for the encryption. The encryption used is the same as the LUKS2 volume itself uses, unlocked by the same volume key, but based on its own IV.<br />
<br />
* Inside of this LUKS2 volume must be a Linux file system, one of [[ext4]], [[btrfs]] and [[xfs]]. The file system label must be the user name.<br />
<br />
* This file system should contain a single directory named after the user. This directory will become the home directory of the user when activated. It contains a second copy of the user record in the ~/.identity file, like in the other storage mechanisms.<br />
<br />
== Enabling PAM modules ==<br />
<br />
{{Accuracy|1=NSS records of ''systemd-homed'' users are provided by {{man|8|nss-systemd}}. See [https://www.reddit.com/r/linux/comments/fgc9n3/my_systemdhomed_experience/fk6prw6/?context=1] and {{man|1|homectl|DESCRIPTION}}.}}<br />
<br />
Since ''systemd-homed'' users don't have any regular [[NSS]] records and aren't present in {{ic|/etc/passwd}}, {{ic|/etc/shadow}} or {{ic|/etc/group}}, the system isn't aware of them by default. It is therefore necessary to use a [[PAM]] module provided.<br />
For example:<br />
{{hc |/etc/pam.d/system-auth |<br />
#%PAM-1.0<br />
<br />
auth sufficient pam_unix.so<br />
-auth sufficient pam_systemd_home.so<br />
auth required pam_deny.so<br />
<br />
account required pam_nologin.so<br />
-account sufficient pam_systemd_home.so<br />
account sufficient pam_unix.so<br />
account required pam_permit.so<br />
<br />
-password sufficient pam_systemd_home.so<br />
password sufficient pam_unix.so sha512 shadow try_first_pass try_authtok<br />
password required pam_deny.so<br />
<br />
-session optional pam_keyinit.so revoke<br />
-session optional pam_loginuid.so<br />
-session optional pam_systemd_home.so<br />
-session optional pam_systemd.so<br />
session required pam_unix.so }}<br />
{{Tip|Refer to {{man|8|pam_systemd_home}} for more info on options}}<br />
<br />
== Examples ==<br />
<br />
The following examples show how to use '''homectl'''<br />
<br />
=== User archie using fscrypt ===<br />
<br />
{{ic|1=# homectl create archie --storage=fscrypt}}<br />
<br />
=== User alan with a specific UID, shell and groups ===<br />
<br />
{{ic|1=# homectl create alan --shell=/bin/zsh --uid=1001 -G wheel,storage,games}}<br />
<br />
=== Delete a user ===<br />
{{Warning|The user deletion is instant, be careful!}}<br />
{{ic|# homectl remove archie}}<br />
<br />
== See also ==<br />
<br />
* https://systemd.io/HOME_DIRECTORY/</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Talk:Systemd-homed&diff=601025Talk:Systemd-homed2020-03-10T12:41:23Z<p>NeoTheFox: Category argument</p>
<hr />
<div>=== About the category ===<br />
<br />
I view this page as an alternative to [[Users_and_groups]], and that one only has Security category. This was my only reasoning to pick a category, if it's not appropriate let's change it. [[User:NeoTheFox|NeoTheFox]] ([[User talk:NeoTheFox|talk]]) 12:41, 10 March 2020 (UTC)</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Systemd-homed&diff=601013Systemd-homed2020-03-10T11:13:46Z<p>NeoTheFox: /* User archie using fscrypt */ missing "="</p>
<hr />
<div>{{lowercase title}}<br />
[[Category:Security]]<br />
{{Related articles start}}<br />
{{Related|fscrypt}}<br />
{{Related|Security}}<br />
{{Related|PAM}}<br />
{{Related articles end}}<br />
'''systemd-homed''' is a [[systemd]] service providing portable human-user accounts that aren't dependent on current system configuration.<br />
<br />
It archives portability by moving all user-related information into a storage medium, preferably encrypted, and creating an {{ic|~/.identity}} file that contains signed information about the user - password, what groups they belong to, UID/GID and other information that would typically be scattered over multiple files in {{ic|/}}.<br />
<br />
This approach allows not only for a home directory portability, but also provides security by automatically managing {{ic|/home}} encryption on login and locking the folder if the system is suspended. <br />
<br />
== Installation ==<br />
<br />
''systemd-homed'' is part of and packaged with {{Pkg|systemd}}.<br />
<br />
== Storage mechanism ==<br />
<br />
=== Directory or [[btrfs]] subvolume ===<br />
<br />
A user home folder is stored in {{ic|/home/user.homedir}} and mounted to {{ic|/home/user}} using bind [[mount]] on unlocking. When this method is used no encryption is provided. <br />
To use this mechanism provide {{ic|1=--storage=directory}} or {{ic|1=--storage=subvolume}} to '''homectl'''.<br />
<br />
=== [[fscrypt]] directory ===<br />
<br />
{{Tip| Filesystems with fscrypt support include [[ext4]] and [[f2fs]]}}<br />
A user home folder is stored the same way as when using the above method, but this time a native filesystem encryption is used. <br />
To use this mechanism provide {{ic|1=--storage=fscrypt}} to '''homectl'''.<br />
<br />
=== [[LUKS]] home directory ===<br />
<br />
A user home folder is stored in a linux [[filesystem]], inside a [[LUKS]] volume inside a loopback file or any removable media. <br />
To use this mechanism provide {{ic|1=--storage=luks}} to ```homectl```.<br />
<br />
If you are using removable media make sure that these conditions are met:<br />
<br />
* The image contains a GPT partition table. For now it should only contain a single partition, and that partition must have the type {{ic|UUID 773f91ef-66d4-49b5-bd83-d683bf40ad16}}. It’s partition label must be the user name. <br />
<br />
* This partition must contain a LUKS2 volume, whose label must be the user name. The LUKS2 volume must contain a LUKS2 token field of type systemd-homed. The JSON data of this token must have a record field, containing a string with base64-encoded data. This data is the JSON user record, in the same serialization as in ~/.identity, though encrypted. The JSON data of this token must also have an iv field, which contains a base64-encoded binary initialization vector for the encryption. The encryption used is the same as the LUKS2 volume itself uses, unlocked by the same volume key, but based on its own IV.<br />
<br />
* Inside of this LUKS2 volume must be a Linux file system, one of [[ext4]], [[btrfs]] and [[xfs]]. The file system label must be the user name.<br />
<br />
* This file system should contain a single directory named after the user. This directory will become the home directory of the user when activated. It contains a second copy of the user record in the ~/.identity file, like in the other storage mechanisms.<br />
<br />
== Enabling PAM modules ==<br />
<br />
Since ''systemd-homed'' users don't have any regular [[NSS]] records and aren't present in {{ic|/etc/passwd}}, {{ic|/etc/shadow}} or {{ic|/etc/group}}, the system isn't aware of them by default. It is therefore necessary to use a [[PAM]] module provided.<br />
For example:<br />
{{hc |/etc/pam.d/system-auth |<br />
#%PAM-1.0<br />
<br />
auth sufficient pam_unix.so try_first_pass nullok<br />
-auth sufficient pam_systemd_home.so <br />
auth optional pam_permit.so<br />
auth required pam_env.so<br />
<br />
-account sufficient pam_systemd_home.so<br />
account required pam_unix.so<br />
account optional pam_permit.so<br />
account required pam_time.so<br />
<br />
-password sufficient pam_systemd_home.so<br />
password required pam_unix.so try_first_pass nullok sha512 shadow<br />
password optional pam_permit.so<br />
<br />
-session optional pam_keyinit.so revoke<br />
-session optional pam_loginuid.so<br />
-session optional pam_systemd_home.so<br />
-session optional pam_systemd.so<br />
session required pam_unix.so<br />
session optional pam_permit.so }}<br />
{{Tip|Refer to {{man|5|pam_systemd_home}} for more info on options}}<br />
<br />
== Examples ==<br />
<br />
The following examples show how to use '''homectl'''<br />
<br />
=== User archie using fscrypt ===<br />
<br />
{{ic|1=# homectl create archie --storage=fscrypt}}<br />
<br />
=== User alan with a specific UID, shell and groups ===<br />
<br />
{{ic|1=# homectl create alan --shell=/bin/zsh --uid=1001 -G wheel,storage,games}}<br />
<br />
=== Delete a user ===<br />
{{Warning|The user deletion is instant, be careful!}}<br />
{{ic|# homectl remove archie}}<br />
<br />
== See also ==<br />
<br />
* https://systemd.io/HOME_DIRECTORY/</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Systemd-homed&diff=601012Systemd-homed2020-03-10T11:04:34Z<p>NeoTheFox: Created systemd-homed page</p>
<hr />
<div>{{lowercase title}}<br />
[[Category:Security]]<br />
{{Related articles start}}<br />
{{Related|fscrypt}}<br />
{{Related|Security}}<br />
{{Related|PAM}}<br />
{{Related articles end}}<br />
'''systemd-homed''' is a [[systemd]] service providing portable human-user accounts that aren't dependent on current system configuration.<br />
<br />
It archives portability by moving all user-related information into a storage medium, preferably encrypted, and creating an {{ic|~/.identity}} file that contains signed information about the user - password, what groups they belong to, UID/GID and other information that would typically be scattered over multiple files in {{ic|/}}.<br />
<br />
This approach allows not only for a home directory portability, but also provides security by automatically managing {{ic|/home}} encryption on login and locking the folder if the system is suspended. <br />
<br />
== Installation ==<br />
<br />
''systemd-homed'' is part of and packaged with {{Pkg|systemd}}.<br />
<br />
== Storage mechanism ==<br />
<br />
=== Directory or [[btrfs]] subvolume ===<br />
<br />
A user home folder is stored in {{ic|/home/user.homedir}} and mounted to {{ic|/home/user}} using bind [[mount]] on unlocking. When this method is used no encryption is provided. <br />
To use this mechanism provide {{ic|1=--storage=directory}} or {{ic|1=--storage=subvolume}} to '''homectl'''.<br />
<br />
=== [[fscrypt]] directory ===<br />
<br />
{{Tip| Filesystems with fscrypt support include [[ext4]] and [[f2fs]]}}<br />
A user home folder is stored the same way as when using the above method, but this time a native filesystem encryption is used. <br />
To use this mechanism provide {{ic|1=--storage=fscrypt}} to '''homectl'''.<br />
<br />
=== [[LUKS]] home directory ===<br />
<br />
A user home folder is stored in a linux [[filesystem]], inside a [[LUKS]] volume inside a loopback file or any removable media. <br />
To use this mechanism provide {{ic|1=--storage=luks}} to ```homectl```.<br />
<br />
If you are using removable media make sure that these conditions are met:<br />
<br />
* The image contains a GPT partition table. For now it should only contain a single partition, and that partition must have the type {{ic|UUID 773f91ef-66d4-49b5-bd83-d683bf40ad16}}. It’s partition label must be the user name. <br />
<br />
* This partition must contain a LUKS2 volume, whose label must be the user name. The LUKS2 volume must contain a LUKS2 token field of type systemd-homed. The JSON data of this token must have a record field, containing a string with base64-encoded data. This data is the JSON user record, in the same serialization as in ~/.identity, though encrypted. The JSON data of this token must also have an iv field, which contains a base64-encoded binary initialization vector for the encryption. The encryption used is the same as the LUKS2 volume itself uses, unlocked by the same volume key, but based on its own IV.<br />
<br />
* Inside of this LUKS2 volume must be a Linux file system, one of [[ext4]], [[btrfs]] and [[xfs]]. The file system label must be the user name.<br />
<br />
* This file system should contain a single directory named after the user. This directory will become the home directory of the user when activated. It contains a second copy of the user record in the ~/.identity file, like in the other storage mechanisms.<br />
<br />
== Enabling PAM modules ==<br />
<br />
Since ''systemd-homed'' users don't have any regular [[NSS]] records and aren't present in {{ic|/etc/passwd}}, {{ic|/etc/shadow}} or {{ic|/etc/group}}, the system isn't aware of them by default. It is therefore necessary to use a [[PAM]] module provided.<br />
For example:<br />
{{hc |/etc/pam.d/system-auth |<br />
#%PAM-1.0<br />
<br />
auth sufficient pam_unix.so try_first_pass nullok<br />
-auth sufficient pam_systemd_home.so <br />
auth optional pam_permit.so<br />
auth required pam_env.so<br />
<br />
-account sufficient pam_systemd_home.so<br />
account required pam_unix.so<br />
account optional pam_permit.so<br />
account required pam_time.so<br />
<br />
-password sufficient pam_systemd_home.so<br />
password required pam_unix.so try_first_pass nullok sha512 shadow<br />
password optional pam_permit.so<br />
<br />
-session optional pam_keyinit.so revoke<br />
-session optional pam_loginuid.so<br />
-session optional pam_systemd_home.so<br />
-session optional pam_systemd.so<br />
session required pam_unix.so<br />
session optional pam_permit.so }}<br />
{{Tip|Refer to {{man|5|pam_systemd_home}} for more info on options}}<br />
<br />
== Examples ==<br />
<br />
The following examples show how to use '''homectl'''<br />
<br />
=== User archie using fscrypt ===<br />
<br />
{{ic|# homectl create archie --storage fscrypt}}<br />
<br />
=== User alan with a specific UID, shell and groups ===<br />
<br />
{{ic|1=# homectl create alan --shell=/bin/zsh --uid=1001 -G wheel,storage,games}}<br />
<br />
=== Delete a user ===<br />
{{Warning|The user deletion is instant, be careful!}}<br />
{{ic|# homectl remove archie}}<br />
<br />
== See also ==<br />
<br />
* https://systemd.io/HOME_DIRECTORY/</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Talk:GPD_Pocket&diff=586561Talk:GPD Pocket2019-10-18T17:34:29Z<p>NeoTheFox: suggestion: rewrite to cover more GPD Pocket variants</p>
<hr />
<div>== gpd-pocket-support ==<br />
<br />
I maintain the repo at https://github.com/njkli/gpd-pocket<br />
<br />
Most of the how-to is now obsolete, that is - there's no need to do anything beyond 'pacman -Syyu gpd-pocket-support' and having the installed kernel in your boot loader configs.<br />
<br />
As of today, the new kernel build includes usb_storage drivers, so it's now possible to boot/install our devices without usb ethernet.<br />
<br />
If anybody wishes to see features/bugs added/removed, please open a github issue.<br />
<br />
Also, i need a little help with packaging - my pkg must overwrite a couple of files from another pkg, the pulseaudio ucm profiles (installed together with alsa-lib), this naturally produces an error message...<br />
<br />
PR is the best answer to the above question, alternatively - open an issue and explain what can be done.<br />
<br />
[[User:Voobscout|Voobscout]] ([[User talk:Voobscout|talk]]) 17:35, 24 September 2017 (UTC)<br />
:First off, thank you for maintaining the repo! It made installation much easier. <br />
:Is the new kernel in the Official Arch ISO? That is what I used and I just added your repo during the installation and installed the packages that way. Would it be better have the wiki just instruct to install the gpd-pocket-support package? If so I hesitate to add that specifically as I was experimenting with the newer bios and when attempting to flash back to the ubuntu bios my machine was at roughly 45% batter and plugged in but it powered off during the flash leaving me to go through the RMA process. [[User:The Idiot You yell at|The Idiot You yell at]] ([[User talk:The Idiot You yell at|talk]]) 14:16, 26 September 2017 (UTC)<br />
<br />
[[User:Farseerfc|Farseerfc]] ([[User talk:Farseerfc|talk]]) 19:12, 12 October 2017 (UTC)<br />
: Thank you for maintaining the repo. <br />
: For the how-to part of this wiki page, I think it's better to leave them on wiki and mention your repo in a note on top, because editing config files manually is more [[TheArchWay]] in my opinion.<br />
: And I think we should better listing the benefits of using a custom kernel (linux-jwrdegoede) over the vanilla kernel in this page. From my experience, most of the things is working including WiFi under 4.13.x vanilla kernel. All I get from linux-jwrdegoede is battery status and fan control.<br />
: BTW I added gpd-fan and linux-jwrdegoede packages into the [[Unofficial_user_repositories#archlinuxcn|unofficial [archlinuxcn] repo ]], because downloading from github is a pain from mainland China.<br />
<br />
::I'm not sure if Hans's kernel is in arch ISO, most likely not!<br />
<br />
::The gpd-pocket-support pkg is probably the easier way to get everything installed at one go, it's got all the xorg/sound etc... configs in it, someone should test it though, since "it works for me(TM)" - all I can say.<br />
<br />
::As to reflashing the BIOS - I didn't have any issues at all, tried it at different Bat% levels and it went smoothly, even at 5% Bat remaining, all done with flashrom from Arch.<br />
<br />
::Thank you very much for the Wiki page! The syntax of ArchWiki still escapes me and I don't wish to learn it either...<br />
::{{Unsigned|09:59, 7 October 2017|Voobscout}}<br />
<br />
== pwm_lpss and pwm_lpss_platform conflict with i915 in early KMS setup using plymouth ==<br />
<br />
I didn't want to update the wiki entry without checking here to see if there was something I'm missing. The wiki currently says to include the line:<br />
<code><br />
MODULES=(pwm_lpss pwm_lpss_platform i915)<br />
</code><br />
in the mkinitcpio.conf to enable early KMS. When I do this I get the classic i915 blank screen if I try to use KMS during the initrd (like with Plymouth). It actually gets the black screen of no return before plymouth runs (during its hook when it sets up the video devices using udev). I suggest the wiki either change to exclude the pwm modules in favor of command line options that make the blacklight work, or alternatively a note stating that it may cause conflicts. [[User:Blackhand314159|Blackhand314159]] ([[User talk:Blackhand314159|talk]]) 03:43, 28 January 2018 (UTC)<br />
<br />
== Suggestion: rewrite the article to be about GPD Pocket devices in general ==<br />
<br />
I think it is a good idea to refactor this article to cover GPD Pocket, GPD Pocket 2 and GPD Pocket 2 MAX devices, since a lot of the tips and info here would fit all these, but a better distincion on device-specific fixes would be great. For example, versions 1 and 2 share the same touchscreen, but Pocket 2 doesn't need a custom kernel or image.<br />
[[User:NeoTheFox|NeoTheFox]] ([[User talk:NeoTheFox|talk]]) 17:34, 18 October 2019 (UTC)</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Mpv&diff=586560Mpv2019-10-18T17:28:32Z<p>NeoTheFox: /* General settings */ added native Wayland support</p>
<hr />
<div>{{Lowercase title}}<br />
[[Category:Video]]<br />
[[Category:Audio]]<br />
[[Category:Streaming]]<br />
[[es:Mpv]]<br />
[[ja:Mpv]]<br />
[[ru:Mpv]]<br />
[[zh-hans:Mpv]]<br />
{{Related articles start}}<br />
{{Related|MPlayer}}<br />
{{Related|youtube-dl}}<br />
{{Related articles end}}<br />
<br />
[https://mpv.io/ mpv] is a media player based on [[MPlayer]] and the now unmaintained ''mplayer2''. It supports a wide variety of video file formats, audio and video codecs, and subtitle types. A comprehensive (although admittedly incomplete) list of differences between ''mpv'' and the aforementioned players can be found [https://github.com/mpv-player/mpv/blob/master/DOCS/mplayer-changes.rst here].<br />
<br />
== Installation ==<br />
<br />
[[Install]] the {{Pkg|mpv}} package or {{AUR|mpv-git}} for the development version.<br />
<br />
=== Front ends ===<br />
<br />
See [[List of applications/Multimedia#mpv-based]].<br />
<br />
== Configuration ==<br />
<br />
''mpv'' comes with good all-around defaults that should work well on computers with weaker/older video cards. However, if you have a computer with a more modern video card then ''mpv'' allows you to do a great deal of configuration to achieve better video quality (limited only by the power of your video card). To do this one only needs to create a few configuration files (they do not exist by default).<br />
<br />
{{Note| Configuration files are read system-wide from {{ic|/etc/mpv}} and per-user from {{ic|~/.config/mpv}} (unless the [[environment variable]] {{ic|XDG_CONFIG_HOME}} is set), where per-user settings override system-wide settings, all of which are overridden by the command line. User specific configuration is suggested since it may require some trial and error.}}<br />
<br />
To help you get started ''mpv'' provides sample configuration files with default settings. Copy them to use as a starting point:<br />
<br />
$ cp -r /usr/share/doc/mpv/ ~/.config/<br />
<br />
{{ic|mpv.conf}} contains the majority of ''mpv'''s settings, {{ic|input.conf}} contains key bindings. Read through both of them to get an idea of how they work and what options are available.<br />
<br />
=== General settings ===<br />
<br />
Add the following settings to {{ic|~/.config/mpv/mpv.conf}}.<br />
<br />
==== High quality configurations ====<br />
<br />
This loads high quality OpenGL options when using {{ic|1=vo=gpu}} as video output (default). Most users can run these without any problems, but they are not enabled by default to avoid causing problems for the few users who cannot run them:<br />
<br />
profile=gpu-hq<br />
<br />
The {{ic|gpu-hq}} profile defaults to the {{ic|spline36}} scaling filter for mid quality and speed. For the best quality video output, the manual states that if your hardware can run it, {{ic|ewa_lanczossharp}} is probably what you should use.<br />
<br />
profile=gpu-hq<br />
scale=ewa_lanczossharp<br />
cscale=ewa_lanczossharp<br />
<br />
These last three options are a little more complicated. The first option makes it so that if audio and video go out of sync then instead of dropping video frames it will resample the audio (a slight change in audio pitch is often less noticeable than dropped frames). The mpv wiki has an in depth article on it titled [https://github.com/mpv-player/mpv/wiki/Display-synchronization Display Synchronization]. The remaining two essentially make motion appear smoother on your display by changing the way that frames are shown so that the source framerate jives better with your display's refresh rate (not to be confused with SVP's technique which actually converts video to 60fps). The mpv wiki has an in depth article on it titled [https://github.com/mpv-player/mpv/wiki/Interpolation Interpolation] though it is also commonly known as ''smoothmotion''.<br />
<br />
profile=gpu-hq<br />
scale=ewa_lanczossharp<br />
cscale=ewa_lanczossharp<br />
video-sync=display-resample<br />
interpolation<br />
tscale=oversample<br />
<br />
Beyond this there is still a lot you can do but things become more complicated, require more powerful video cards, and are in constant development. As a brief overview, it is possible to load special shaders that perform exotic scaling and sharpening techniques including some that actually use deep neural networks trained on images (for both real world and animated content). To learn more about this take a look around the [https://github.com/mpv-player/mpv/wiki mpv wiki], particularly the [https://github.com/mpv-player/mpv/wiki/User-Scripts#user-shaders user shader's section].<br />
<br />
There are also plenty of other options you may find desirable as well. It is worthwhile taking a look at {{man|1|mpv}}. It is also helpful to run ''mpv'' from the command line to check for error messages about the config.<br />
<br />
==== Custom profiles ====<br />
<br />
In {{ic|mpv.conf}} it is possible to create ''profiles'' which are essentially just "groups of options" with which you can:<br />
<br />
* Quickly switch between different configurations without having to rewrite the file.<br />
* Create special profiles for special content.<br />
* ''nest'' profiles so that you can make more complicated ''profiles'' out of simpler ones.<br />
<br />
Creating a profile is easy. The area at the top of {{ic|mpv.conf}} is called the top level, any options you write there will kick into effect once ''mpv'' is started. However, once you define a profile by writing its name in brackets then every option you write below it (until you define a new profile) is considered part of that profile. Here is an example {{ic|mpv.conf}}:<br />
<br />
profile=myprofile2 #Top level area, load myprofile2<br />
ontop=yes #Always on top<br />
<br />
[myprofile1] #A simple profile, top level area ends here<br />
profile-desc="a profile" #Optional description for profile<br />
fs=yes #Start in full screen<br />
<br />
[myprofile2] #Another simple profile<br />
profile=gpu-hq #A built in profile that comes with mpv<br />
log-file=~~/log #Sets a location for writing a log file, ~~/ translates to ~/.config/mpv<br />
<br />
There are only two lines within the top level area and there are two separate profiles defined below it. When ''mpv'' starts it sees the first line, loads the options in {{ic|myprofile2}} (which means it loads the options in {{ic|gpu-hq}} and {{ic|1=log-file=~~/log}}) finally it loads {{ic|1=ontop=yes}} and finishes starting up. Note, {{ic|myprofile1}} is never loaded because it is never called in the top level area.<br />
<br />
Alternatively one could call ''mpv'' from the command line with:<br />
<br />
$ mpv --profile=myprofile1 video.mkv<br />
<br />
and it would ignore all options except the ones for {{ic|myprofile1}}.<br />
<br />
==== Native Wayland output ====<br />
<br />
By default mpv uses X even on [[Wayland]] clients, like [[Sway]]. <br />
<br />
In order to make it work on Wayland by default either specify it from the command line: <br />
<br />
$ mpv --gpu-context=wayland video.mkv <br />
<br />
It is also possible to add {{ic|1=gpu-context=wayland}} to your config.<br />
<br />
=== Key bindings ===<br />
<br />
Key bindings are fairly straightforward given the examples in {{ic|/usr/share/doc/mpv/input.conf}} and the relevant section in the [https://mpv.io/manual/master/#command-interface manual].<br />
<br />
Add the following examples to {{ic|~/.config/mpv/input.conf}}:<br />
<br />
shift+s screenshot each-frame<br />
Shift+UP seek 600<br />
Shift+DOWN seek -600<br />
= cycle video-unscaled<br />
- cycle-values window-scale 2 3 1 .5<br />
WHEEL_UP add volume 5<br />
WHEEL_DOWN add volume -5<br />
WHEEL_LEFT ignore<br />
WHEEL_RIGHT ignore<br />
Alt+RIGHT add video-rotate 90<br />
Alt+LEFT add video-rotate -90<br />
Alt+- add video-zoom -0.25<br />
Alt+= add video-zoom 0.25<br />
Alt+j add video-pan-x -0.05<br />
Alt+l add video-pan-x 0.05<br />
Alt+i add video-pan-y 0.05<br />
Alt+k add video-pan-y -0.05<br />
Alt+BS set video-zoom 0; set video-pan-x 0; set video-pan-y 0<br />
<br />
For an attempt to reproduce MPC-HC key bindings in mpv, see [https://github.com/dragons4life/MPC-HC-config-for-MPV/blob/master/input.conf].<br />
<br />
=== Additional configuration files ===<br />
<br />
In addition there are a few more configuration files and directories that can be created, among which:<br />
<br />
* {{ic|~/.config/mpv/script-opts/osc.conf}} manages the [https://mpv.io/manual/master/#on-screen-controller On Screen Controller].<br />
* {{ic|~/.config/mpv/scripts/''script-name''.lua}} for Lua scripts. See [https://github.com/mpv-player/mpv/issues/3500#issuecomment-305646994] for an example.<br />
<br />
See https://mpv.io/manual/master/#files for more.<br />
<br />
== Scripts ==<br />
<br />
''mpv'' has a [https://github.com/mpv-player/mpv/wiki/User-Scripts large variety of scripts] that extend the functionality of the player. To this end, it has internal bindings for both Lua and JavaScript (added recently).<br />
<br />
Scripts are typically installed by putting them in the {{ic|~/.config/mpv/scripts/}} directory (you may have to create it first). After that they will be automatically loaded when mpv starts (this behavior can be altered with other ''mpv'' options). Some scripts come with their own installation and configuration instructions, so make sure to have a look. In addition some scripts are old, broken, and unmaintained.<br />
<br />
=== JavaScript ===<br />
<br />
Since JavaScript support is still fairly new, there is currently very little in the way of scripts, but [https://mpv.io/manual/master/#javascript documentation exists] for anyone interested in making their own.<br />
<br />
JavaScript support is not available in the {{Pkg|mpv}} build, but it is supported by some AUR packages e.g. {{AUR|mpv-full}} and {{AUR|mpv-full-git}}.<br />
<br />
=== Lua ===<br />
<br />
There are a lot of interesting Lua scripts for mpv. If you would like to make your own, the relevant documentation may be found [https://github.com/mpv-player/mpv/blob/master/DOCS/man/lua.rst here].<br />
<br />
==== mpv-stats ====<br />
<br />
[https://github.com/Argon-/mpv-stats/ mpv-stats] (or simply ''stats'') is a Lua script that outputs a lot of live statistics showing how well ''mpv'' is currently doing. It is very useful for making sure that your hardware can keep up with your configuration and for comparing different configurations. Since version [https://github.com/mpv-player/mpv/releases/tag/v0.28.0 v0.28.0], the script is built into {{Pkg|mpv}} and can be toggled on or off with the {{ic|i}} or {{ic|I}} keys (by default).<br />
<br />
==== mpv-webm ====<br />
<br />
[https://github.com/ElegantMonkey/mpv-webm mpv-webm] (or simply ''webm'') is a very easy to use Lua script that allows one to create WebM files while watching videos. It includes several features and does not have any extra dependencies (relies entirely on mpv).<br />
<br />
=== C ===<br />
<br />
==== mpv-mpris ====<br />
<br />
The C plugin [https://github.com/hoyon/mpv-mpris mpv-mpris] allows other applications to integrate with ''mpv'' via the MPRIS protocol. For example, with mpv-mpris installed, {{pkg|kdeconnect}} can automatically pause video playback in ''mpv'' when a phone call arrives.<br />
<br />
Install {{AUR|mpv-mpris}} and follow the post-installation instructions displayed by Pacman.<br />
<br />
== Vapoursynth ==<br />
<br />
Vapoursynth is an alternative to AviSynth that can be used on Linux and allows for Video manipulation via python scripts. Vapoursynths python scripts can be used as video filters for ''mpv''.<br />
<br />
To use vapoursynth filters you have to install the {{Pkg|vapoursynth}} package (or {{AUR|vapoursynth-git}}) and compile ''mpv'' with the {{ic|--enable-vapoursynth}} build flag.<br />
<br />
This is easier to do by first installing Vapoursynth and then installing (or re-installing if it is already installed) {{AUR|mpv-git}}. The configure script for {{AUR|mpv-git}} will auto-detect Vapoursynth (as long as it has already been installed) and it will automatically compile ''mpv'' with support for Vapoursynth without having to manually change any configure options or anything (this makes it very easy to update ''mpv'' as well).<br />
<br />
=== SVP 4 Linux (SmoothVideoProject) ===<br />
<br />
[https://www.svp-team.com/wiki/Main_Page SmoothVideoProject SVP] is a program that is primarily known for converting video to 60fps. It is free [as in beer] and full featured for 64bit Linux (costs money for Windows and OS X and is incompatible with 32bit Linux).<br />
<br />
It has three main features and each one can be disabled/enabled as one chooses (you are not forced to use motion interpolation).<br />
<br />
# [https://www.svp-team.com/wiki/Manual:FRC Motion interpolation] ([https://www.youtube.com/watch?v=Wjb6CSe4708 youtube video]) - An algorithm that converts video to 60fps. This creates the somewhat controversial "soap opera effect" that some people love and others hate. Unfortunately the algorithm is not perfect and it also introduces more than its share of weird artifacts. The algorithm can be tuned (via a slider) for either performance or quality. It also has some artifact reduction settings that interpolate actual frames with the generated frames reducing the noticeability of the artifacts. The framerate detection can be set to automatic or manual (manual seems to resolve performance issues for some users).<br />
# [https://www.svp-team.com/wiki/Manual:Outer_lighting Black bar lighting] ([https://www.youtube.com/watch?v=yTzTpW3kTBE youtube video]) - If the image has an aspect ratio that produces black bars on your display then SVP will illuminate the black bars with "lights" generated by the content on the screen. It has some amount of customization but the defaults are pretty close to optimal.<br />
# [https://www.svp-team.com/wiki/Manual:SVPlight LED ambient lighting control] ([https://www.youtube.com/watch?v=UUM2n-8kIJ8 youtube video]) - Has the ability to control LED ambient lighting attached to your television.<br />
<br />
Once you have ''mpv'' compiled with Vapoursynth support it is fairly easy to get SVP working with ''mpv''. Simply install {{AUR|svp}}, open the SVP program to let it assess your system performance (you may want to close other programs first so that it gets an accurate reading), and finally add the following ''mpv'' profile to your mpv.conf[https://www.svp-team.com/wiki/SVP:mpv]:<br />
<br />
{{hc|1=mpv.conf|2=<br />
[svp]<br />
input-ipc-server=/tmp/mpvsocket # Receives input from SVP<br />
hr-seek-framedrop=no # Fixes audio desync<br />
resume-playback=no # Not compatible with SVP<br />
<br />
# Can fix stuttering in some cases, in other cases probably causes it. Try it if you experience stuttering.<br />
#opengl-early-flush=yes <br />
}}<br />
<br />
Then, in order to use SVP you must have the SVP program running in the background before opening the file using ''mpv'' with that profile. Either do:<br />
<br />
$ mpv --profile=svp video.mkv<br />
<br />
or set {{ic|1=profile=svp}} in the top-level portion of the ''mpv'' [[#Custom profiles|configuration]].<br />
<br />
If you want to use hardware decoding then you must use a copy-back decoder since normal decoders are not compatible with Vapoursynth (choose a {{ic|hwdec}} option that ends in {{ic|-copy}}). For instance:<br />
<br />
hwdec=auto-copy<br />
hwdec-codecs=all<br />
<br />
Either way, hardware decoding is discouraged by ''mpv'' developers and is not likely to make a significant difference in performance.<br />
<br />
== Tips and Tricks ==<br />
<br />
=== Hardware video acceleration ===<br />
<br />
See [[Hardware video acceleration]].<br />
<br />
Hardware accelerated video decoding is available via {{ic|1=--hwdec=''API''}} option. For list of all supported APIs and other required options see [https://mpv.io/manual/stable/#options-hwdec relevant manual section].<br />
<br />
For [[Wayland]] use {{ic|1=--gpu-context=wayland}} option. For list of other available GPU APIs see [https://mpv.io/manual/stable/#options-gpu-context manual].<br />
<br />
=== Save position on quit ===<br />
<br />
By default you can save the position and quit by pressing {{ic|Shift+q}}. The shortcut can be changed by setting {{ic|quit_watch_later}} in the key bindings configuration file.<br />
<br />
To automatically save the current playback position on quit, start ''mpv'' with {{ic|--save-position-on-quit}}, or add {{ic|save-position-on-quit}} to the configuration file.<br />
<br />
=== Volume is too low ===<br />
<br />
Set {{ic|1=volume-max=''value''}} in your configuration file to a reasonable amount, such as {{ic|1=volume-max=150}}, which then allows you to increase your volume up to 150%, which is more than twice as loud. Increasing your volume too high will result in clipping artefacts. Additionally (or alternatively), you can utilize [[Wikipedia:Dynamic range compression|dynamic range compression]] with {{ic|1=af=acompressor}}.<br />
<br />
=== Play a DVD ===<br />
<br />
To start the main stream of a video DVD, use the command:<br />
<br />
$ mpv dvd://<br />
<br />
Install {{Pkg|libdvdcss}}, to fix the error:<br />
<br />
[dvdnav] Error getting next block from DVD 1 (Error reading from DVD.)<br />
<br />
=== Quickly cycle between aspect ratios ===<br />
<br />
You can cycle between aspect ratios using {{ic|Shift+a}}.<br />
<br />
=== Ignoring aspect ratio ===<br />
<br />
You can ignore aspect ratio using {{ic|1=--keepaspect=''no''}}. To make option permanent, add line {{ic|1=keepaspect=''no''}} to configuration file.<br />
<br />
=== Draw to the root window ===<br />
<br />
Run ''mpv'' with {{ic|1=--wid=0}}. ''mpv'' will draw to the window with a window ID of 0.<br />
<br />
=== Always show application window ===<br />
<br />
To show application window even for audio files when launching mpv from command line use {{ic|--force-window}} option.<br />
<br />
=== Disable video output ===<br />
<br />
To disable video output when launching from command line use {{ic|1=--vid=no}} option, or its alias, {{ic|--no-video}}.<br />
<br />
=== Restoring old OSC ===<br />
<br />
Since version 0.21.0, ''mpv'' has replaced the on-screen controls by a bottombar. In case you want on-screen controls back, you can edit the ''mpv'' configuration [https://github.com/mpv-player/mpv/wiki/FAQ#i-want-the-old-osc-back as described here].<br />
<br />
=== Use as a browser plugin ===<br />
<br />
With the help of {{AUR|mozplugger}}, ''mpv'' can be used in a supported browser to play video. See [[Browser plugins#MozPlugger]] for configuration details. This coupled with a user script such as [http://isebaro.com/viewtube/?ln=en ViewTube], allows you to use ''mpv'' in place of a site's integrated video player.<br />
<br />
It may be needed to specify a valid user agent for HTTP streaming, e.g. {{ic|1=user-agent="Mozilla/5.0 (X11; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0"}}.<br />
<br />
[[Browser plugins#Multimedia playback]] page shows other easy ways to watch videos.<br />
<br />
=== Improving mpv as a music player with Lua scripts ===<br />
<br />
The development of ''mpv'''s Lua scripts are documented in [https://github.com/mpv-player/mpv/blob/master/DOCS/man/lua.rst DOCS/man/lua.rst]<br />
and examples are shown in [https://github.com/mpv-player/mpv/tree/master/TOOLS/lua TOOLS/lua]<br />
of the [https://github.com/mpv-player/mpv mpv repository].<br />
[https://web.archive.org/web/20160320001546/http://bamos.github.io/2014/07/05/mpv-lua-scripting/ This blog post] introduces the<br />
[https://github.com/bamos/dotfiles/blob/master/.mpv/scripts/music.lua music.lua] script,<br />
which shows how Lua scripts can be used to improve ''mpv'' as a music player.<br />
<br />
=== Twitch.tv streaming over mpv ===<br />
<br />
If [[youtube-dl]] is installed, ''mpv'' can directly open a Twitch livestream.<br />
<br />
Alternatively, see [[Streamlink#Twitch]].<br />
<br />
Another alternative based on Livestreamer is this Lua script: https://gist.github.com/ChrisK2/8701184fe3ea7701c9cc<br />
<br />
=== youtube-dl and choosing formats ===<br />
<br />
The default {{ic|--ytdl-format}} is {{ic|bestvideo+bestaudio/best}}. For youtube videos that have 4K resolutions available, this may mean that your device will struggle to decode 4K VP9 encoded video in software even if the attached monitor is much lower resolution.<br />
<br />
Setting the right youtube-dl format selectors can fix this easily though. In the following configuration example, only videos with a vertical resolution of 1080 pixels or less will be considered.<br />
<br />
ytdl-format="bestvideo[height<=?1080]+bestaudio/best"<br />
<br />
If you wish to avoid a certain codec altogether because you cannot hardware-decode it, you can add this to the format selector. For example, we can additionally choose to ignore VP9 as follows:<br />
<br />
ytdl-format="bestvideo[height<=?1080][vcodec!=vp9]+bestaudio/best"<br />
<br />
If you prefer best quality open codecs (VP9 and Opus), use:<br />
ytdl-format="((bestvideo[vcodec^=vp9]/bestvideo)+(bestaudio[acodec=opus]/bestaudio[acodec=vorbis]/bestaudio[acodec=aac]/bestaudio))/best"<br />
<br />
=== youtube-dl audio with search ===<br />
<br />
To find and stream audio from your terminal emulator with {{ic|yta ''search terms''}} put the following function in your {{ic|.bashrc}}:<br />
<br />
function yta() {<br />
mpv --ytdl-format=bestaudio ytdl://ytsearch:"$*"<br />
}<br />
<br />
=== Creating a single screenshot ===<br />
An example of creating a single screenshot, by using a start time ({{ic|HH:MM:SS}}):<br />
<br />
$ mpv --no-audio --start=00:01:30 --frames=1 /path/to/video/file --o=/path/to/screenshot.png<br />
<br />
Screenshots will be saved in /path/to/screenshot.png.<br />
<br />
== Troubleshooting ==<br />
<br />
=== General debugging ===<br />
<br />
If you are having trouble with ''mpv'''s playback (or if it is flat out failing to run) then the first three things you should do are:<br />
<br />
# Run ''mpv'' from the command line (the -v flag increases verbosity). If you are lucky there will be an error message there telling you what is wrong.<br>{{ic|$ mpv -v video.mkv}}<br />
# Have ''mpv'' output a log file. The log file might be difficult to sift through but if something is broken you might see it there.<br>{{ic|1=$ mpv -v --log-file=./log video.mkv}}<br />
# Run ''mpv'' without a configuration. If this runs well then the problem is somewhere in your configuration (perhaps your hardware cannot keep up with your settings).<br>{{ic|$ mpv --no-config video.mkv}}<br />
<br />
If ''mpv'' runs but it just does not run well then a fourth thing that might be worth taking a look at is installing the [[#mpv-stats|mpv-stats]] script and using it to see exactly how it is performing.<br />
<br />
=== Fix jerky playback and tearing ===<br />
<br />
''mpv'' defaults to using the OpenGL video output device setting on hardware that supports it. In cases such as trying to play video back on a 4K display using a Intel HD4XXX series card or similar, you will find video playback unreliable, jerky to the point of stopping entirely at times and with major tearing when using any OpenGL output setting. If you experience any of these issues, using the XV ([[Xorg]] only) video output device may help:<br />
<br />
{{hc|1=~/.config/mpv/mpv.conf|2=<br />
vo=xv<br />
}}<br />
<br />
{{Note|This is the most compatible VO on X, but may be low-quality, and has issues with OSD and subtitle display.}}<br />
<br />
It is possible to increase playback performance even more (especially on lower hardware), but this decreases the video quality dramatically in most cases.<br />
<br />
The following [[#Configuration|options]] may be considered to increase the video playback performance:<br />
<br />
{{hc|~/.config/mpv/mpv.conf|2=<br />
vd-lavc-fast<br />
vd-lavc-skiploopfilter=<skipvalue><br />
vd-lavc-skipframe=<skipvalue><br />
vd-lavc-framedrop=<skipvalue><br />
vd-lavc-threads=<threads><br />
}}<br />
<br />
=== Problems with window compositors ===<br />
<br />
Window compositors such as KWin or Mutter can cause trouble for playback smoothness. In such cases, it may help to set {{ic|1=x11-bypass-compositor=yes}} to make ''mpv'' also disable window compositing when playing in windowed mode (if supported by the compositor).<br />
<br />
With KWin compositing and hardware decoding, you may also want to set {{ic|1=x11-bypass-compositor=no}} to keep compositing enabled in fullscreen, since reanabling compositing after leaving fullscreen may introduce stutter for a period of time.<br />
<br />
=== No volume bar, cannot change volume ===<br />
<br />
Spin the mouse wheel over the volume icon.<br />
<br />
=== GNOME Blank screen (Wayland) ===<br />
<br />
''mpv'' may not suspend GNOME's Power Saving Settings if using Wayland resulting in screen saver turning off the monitor during video playback. A workaround is to add {{ic|gnome-session-inhibit}} to the beginning of the {{ic|1=Exec=}} line in {{ic|mpv.desktop}}.<br />
<br />
=== Use mpv with a compositor ===<br />
<br />
If you are using a compositor (e.g. in KDE Plasma 5) and find that composition is disabled (e.g. in Plasma this would make you unable to present windows or see window thumbnails in the default app switcher) when ''mpv'' is playing a video, try {{ic|1=x11-bypass-compositor=no}}</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=GPD_Pocket&diff=571841GPD Pocket2019-04-22T12:52:10Z<p>NeoTheFox: /* Xorg */ remapping 半/全 on Pocket 2</p>
<hr />
<div>[[Category:Laptops]]<br />
[[de:GPD Pocket]]<br />
[[fr:GPD Pocket]]<br />
[[ja:GPD Pocket]]<br />
Notes for the [https://www.indiegogo.com/projects/gpd-pocket-7-0-umpc-laptop-ubuntu-or-win-10-os-laptop--2#/ GPD Pocket].<br />
<br />
== Specs ==<br />
<br />
* Display: 7inch IPS 1920x1200<br />
* CPU: Intel Atom X7-Z8750<br />
* RAM: 8GB LPDDR3-1600<br />
* Storage: 128GB eMMC SSD (non-replaceable)<br />
* Battery: 7000mAh<br />
* WiFi: Broadcom 4356 802.11ac<br />
* Bluetooth: Broadcom 2045<br />
* Audio: Realtek ALC5645<br />
* Ports: 1 x USB 3 type A, 1 x MicroHDMI, 1 x USB 3 type C, 1 x 3.5mm Headphone Jack<br />
<br />
== Installation ==<br />
<br />
=== Automatic ===<br />
<br />
You can download a pre-patched ISO from [https://github.com/sigboe/GPD-ArchISO/releases here].<br />
<br />
=== Manual ===<br />
<br />
Because WiFi is not working with the default configuration, you have to fix WiFi first<sup>(see [[#WiFi]])</sup>, or use a supported USB Ethernet/WiFi dongle.<br />
<br />
== Configuration ==<br />
<br />
=== Automatic ===<br />
<br />
During install append the following to /etc/pacman.conf:<br />
<br />
{{hc|/etc/pacman.conf|<nowiki><br />
...<br />
[gpd-pocket-arch]<br />
SigLevel = Never<br />
Server = https://github.com/joshskidmore/gpd-pocket-arch/raw/master<br />
...<br />
</nowiki>}}<br />
<br />
Install the changes necessary for the GPD Pocket to run Arch properly by running:<br />
<br />
{{ic|# pacman -Syu gpd-pocket-support}}<br />
<br />
Because the patch for alsa-lib is an optional dependency, it must be installed manually to get audio to work:<br />
<br />
{{ic|# pacman -S gpd-pocket-alsa-lib}}<br />
<br />
=== Manual ===<br />
<br />
==== WiFi ====<br />
<br />
Install the package {{AUR|gpd-pocket-support-bcm4356-git}} and reload the WiFi kernel module:<br />
<br />
# modprobe -r brcmfmac<br />
# modprobe brcmfmac<br />
<br />
==== Backlight and KMS ====<br />
<br />
In order to enable backlight control with early KMS change {{ic|/etc/mkinitcpio.conf}} to match the following:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
...<br />
MODULES=(pwm_lpss pwm_lpss_platform i915)<br />
...<br />
}}<br />
<br />
==== Wayland ====<br />
<br />
===== Basic Configuration =====<br />
<br />
Create {{ic|/etc/udev/rules.d/99-goodix-touch.rules}} to rotate the touchscreen, and fill it with:<br />
<br />
ACTION=="add|change", KERNEL=="event[0-9]*", ATTRS{name}=="Goodix Capacitive TouchScreen", ENV{LIBINPUT_CALIBRATION_MATRIX}="0 1 0 -1 0 1"<br />
<br />
===== Right Click Emulation =====<br />
<br />
Unlike Xorg, under which right click emulation can be enabled by the standard Xorg configuration files, under Wayland, such configuration is supposed to be exposed by the compositor, and unfortunately, some compositors (e.g. GNOME Wayland) does not expose these configurations properly. However, the regarding functionality is still available in {{ic|libinput}}. Since these compositors normally loads {{ic|/etc/profile.d}}, {{ic|LD_PRELOAD}} can be used to hook into {{ic|libinput}} and force apply these configurations.<br />
<br />
A sample implementation of this approach is available [https://github.com/PeterCxy/scroll-emulation here].<br />
<br />
==== Xorg ====<br />
<br />
===== Basic Configuration =====<br />
<br />
Create {{ic|/etc/X11/xorg.conf.d/30-monitor.conf}} to rotate the monitor:<br />
<br />
{{Note|The Identifier may be different depending on your display driver of choice (either {{ic|<nowiki>DSI-1</nowiki>}} <sup>(modesetting)</sup> or {{ic|<nowiki>DSI1</nowiki>}} <sup>(xf86-video-intel)</sup>)}}<br />
<br />
{{hc|/etc/X11/xorg.conf.d/30-monitor.conf|<nowiki><br />
Section "Monitor"<br />
Identifier "DSI-1"<br />
Option "Rotate" "right"<br />
EndSection<br />
</nowiki>}}<br />
<br />
===== Gnome and GDM =====<br />
<br />
Edit {{ic|~/.config/monitors.xml}} (this file might not be present by default):<br />
<br />
{{hc|~/.config/monitors.xml|2=<br />
<monitors version="2"><br />
<configuration><br />
<logicalmonitor><br />
<x>0</x><br />
<y>0</y><br />
<scale>2</scale><br />
<primary>yes</primary><br />
<transform><br />
<rotation>right</rotation><br />
<flipped>no</flipped><br />
</transform><br />
<monitor><br />
<monitorspec><br />
<connector>DSI-1</connector><br />
<vendor>unknown</vendor><br />
<product>unknown</product><br />
<serial>unknown</serial><br />
</monitorspec><br />
<mode><br />
<width>1200</width><br />
<height>1920</height><br />
<rate>60.384620666503906</rate><br />
</mode><br />
</monitor><br />
</logicalmonitor><br />
</configuration><br />
</monitors><br />
}}<br />
<br />
This sets the correct rotation ({{ic|<rotation>right</rotation>}}) and a scale factor of 2 ({{ic|<scale>2</scale>}}). For fractional scaling, see [[HiDPI#GNOME]].<br />
<br />
For [[GDM]], copy the above {{ic|~/.config/monitors.xml}} to {{ic|/var/lib/gdm/.config/monitors.xml}} to set the correct rotation.<br />
<br />
===== KDE =====<br />
<br />
In ''System Settings > Display and Monitor'' change ''Orientation'' to ''90° Clockwise'', and ''Scale Display'' to a comfortable size.<br />
<br />
===== Right Click Emulation =====<br />
<br />
Create {{ic|/etc/X11/xorg.conf.d/50-trackpoint.conf}} to scroll while holding right click:<br />
<br />
{{hc|/etc/X11/xorg.conf.d/50-trackpoint.conf|<br />
Section "InputClass"<br />
Identifier "GPD trackpoint"<br />
MatchProduct "SINO WEALTH Gaming Keyboard"<br />
MatchIsPointer "on"<br />
Driver "libinput"<br />
Option "MiddleEmulation" "1"<br />
Option "ScrollButton" "3"<br />
Option "ScrollMethod" "button"<br />
EndSection<br />
}}<br />
<br />
===== SDDM =====<br />
<br />
Change DPI to be readable, append the following lines to {{ic|/usr/share/sddm/scripts/Xsetup}}:<br />
<br />
{{hc|/usr/share/sddm/scripts/Xsetup|<br />
# Set DPI <br />
xrandr --dpi 168" <br />
}}<br />
<br />
===== Touchscreen Gestures =====<br />
<br />
Install {{AUR|touchegg}}, then edit the following line in {{ic|/usr/share/touchegg/touchegg.conf}}:<br />
{{hc|/usr/share/touchegg/touchegg.conf|2=<br />
...<br />
<action type="SCROLL">SPEED=7:INVERTED=true</action><br />
...<br />
}}<br />
<br />
Create {{ic|/etc/X11/xinit/xinitrc.d/01_touchegg}}<br />
{{hc|/etc/X11/xinit/xinitrc.d/01_touchegg|2=<br />
...<br />
#!/bin/bash<br />
<br />
# starts touchegg application<br />
PREFIX="$HOME/.config/touchegg/.run"<br />
mkdir -p "$PREFIX"<br />
PIDFILE="$PREFIX/touchegg.$USER$DISPLAY.pid"<br />
LOCK="$PREFIX/touchegg.$USER$DISPLAY.lock"<br />
<br />
start_touchegg() {<br />
(<br />
flock -n 9 || exit 1<br />
touchegg 2>/dev/null >/dev/null &<br />
PID=$!<br />
echo "$!" >"$PIDFILE"<br />
wait $PID<br />
) 9>"$LOCK"<br />
}<br />
<br />
start_touchegg &<br />
}}<br />
<br />
Set the permissions on {{ic|/etc/X11/xinit/xinitrc.d/01_touchegg}}<br />
<br />
# chmod 0755 /etc/X11/xinit/xinitrc.d/01_touchegg<br />
<br />
===== Making use of 半/全 key on Pocket 2 =====<br />
<br />
Out of the box the {{ic|半/全}} key is configured to send the {{ic|`}} symbol and keycode 49, making it a duplicated of a regular {{ic|`/~}} key.<br />
In order to be able to map it to something more useful go to the [https://www.gpd.hk/gpdp2firmware GPD2 Firmware] and download "GPD Pocket 2 Keyboard Firmware (Japan)". This must be flashed with the Windows 10 OS supplied with the device. It is advised to have a USB mouse, since the keyboard and mouse will be disabled during the update.<br />
<br />
After the update is done {{ic|半/全}} will send keycode 49 and {{ic|`}} symbol, while the dedicated {{ic|`/~}} key will send keycode 132. In order to fix this create a file {{ic|~/.Xmodmap}}<br />
{{hc|~/.Xmodmap|2=<br />
keycode 132 = grave asciitilde<br />
keycode 49 = XF86Launch1 NoSymbol<br />
}}<br />
<br />
Test it immediately <br />
<br />
$ xmodmap ~/.Xmodmap<br />
<br />
For more information read [[xmodmap]]<br />
<br />
==== Fan ====<br />
<br />
With the latest kernel your fan should work out of the box.<br />
<br />
{{Note|If you are having issues with your fan not functioning as intended - try the following:}}<br />
<br />
# modprobe -r gpd-pocket-fan<br />
# modprobe gpd-pocket-fan temp_limits=40000,40001,40002<br />
<br />
Once this has been completed - you should hear your fan start up at 40c - if you hear a clicking sound - power off the device, remove the back panel and very gently push the fan around a few times. Then re-attach the panel and power on the device - running the above commands again once logged in. It seems to be an issue with some devices that the fan cannot start properly when it has not been powered on in a while.<br />
<br />
Once you have completed these steps and the fan is working properly - you should then either reboot or reload the fan kernel module in order to return the temperature limits to default:<br />
<br />
# modprobe -r gpd-pocket-fan<br />
# modprobe gpd-pocket-fan<br />
<br />
{{Note|By default fan is always spinning when on AC [https://github.com/stockmind/gpd-pocket-ubuntu-respin#gpd-fan-always-spinning-on-ac]. To override this behavior add {{ic|1=gpd-pocket-fan.speed_on_ac=0}} to the [[Kernel parameters]].}}<br />
<br />
==== Power Saving ====<br />
<br />
Install {{Pkg|tlp}} and then edit following lines in {{ic|/etc/default/tlp}}:<br />
<br />
{{hc|/etc/default/tlp|2=<br />
...<br />
# improve disk IO<br />
DISK_DEVICES="mmcblk0"<br />
DISK_IOSCHED="deadline"<br />
...<br />
# disable wifi power saving mode (wifi speed drops MASSIVELY!)<br />
WIFI_PWR_ON_AC=off<br />
WIFI_PWR_ON_BAT=off<br />
...<br />
}}<br />
<br />
==== PulseAudio ====<br />
<br />
Append the following lines into {{ic|/etc/pulse/default.pa}}:<br />
<br />
{{hc|/etc/pulse/default.pa|<br />
set-card-profile alsa_card.platform-cht-bsw-rt5645 HiFi<br />
set-default-sink alsa_output.platform-cht-bsw-rt5645.HiFi__hw_chtrt5645_0__sink<br />
set-sink-port alsa_output.platform-cht-bsw-rt5645.HiFi__hw_chtrt5645_0__sink [Out] Speaker<br />
}}<br />
<br />
Turn off realtime scheduling by editing {{ic|/etc/pulse/daemon.conf}}:<br />
<br />
{{hc|/etc/pulse/daemon.conf|2=<br />
realtime-scheduling = no<br />
}}<br />
<br />
== Known Issues ==<br />
==== USB-C Power source status ====<br />
<br />
USB-C power source status does not work on Kernel 4.14-15. Hans' kernel has a patch fixing this.<br />
<br />
==== systemd-gpt-auto-generator failed to dissect ====<br />
Due to [https://github.com/systemd/systemd/issues/5806 this issue], an error message appears at boot: <br />
<br />
{{ic|systemd-gpt-auto-generator[199]: Failed to dissect: Input/output error}}.<br />
<br />
To avoid the error message, add this boot parameter to your boot loader.<br />
<br />
systemd.gpt_auto=0<br />
<br />
== See also ==<br />
<br />
* https://github.com/njkli/gpd-pocket/<br />
* http://hansdegoede.livejournal.com/</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=510637PCI passthrough via OVMF2018-02-13T13:25:02Z<p>NeoTheFox: /* Using Looking Glass to stream guest screen to the host */ formating</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=processors&VTD=true List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Merge|QEMU#Enabling IOMMU (Intel VT-d/AMD-Vi) support|Same topic.}}<br />
<br />
IOMMU is a system specific IO mapping mechanism and can be used with most devices. IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
<br />
Before you enable IOMMU, you might have to first enable (non-IOMMU) virtualisation (Intel VT-x/"Vanderpool" or AMD-V/"Pacifica") in your BIOS settings.<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is enabled in your BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
You will also have to enable iommu support in the kernel itself through a [[Kernel parameters|bootloader kernel option]]. Depending on your type of CPU, use either {{ic|1=intel_iommu=on}} for Intel CPUs (VT-d) or {{ic|1=amd_iommu=on}} for AMD CPUs (AMD-Vi).<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for d in /sys/kernel/iommu_groups/*/devices/*; do <br />
n=${d#*/iommu_groups/*}; n=${n%%/*}<br />
printf 'IOMMU Group %s ' "$n"<br />
lspci -nns "${d##*/}"<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0104] (rev 09)<br />
IOMMU Group 1 00:16.0 Communication controller [0780]: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 [8086:1c3a] (rev 04)<br />
IOMMU Group 2 00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 04)<br />
IOMMU Group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev <br />
...<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will be appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the prefered method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this.}}<br />
<br />
=== Using vfio-pci ===<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use. If your system supports it, which you can try by running the following command, you should use it. If it returns an error, use pci-stub instead.<br />
<br />
{{hc|$ modinfo vfio-pci|<br />
filename: /lib/modules/4.4.5-1-ARCH/kernel/drivers/vfio/pci/vfio-pci.ko.gz<br />
description: VFIO PCI - User Level meta-driver<br />
author: Alex Williamson <alex.williamson@redhat.com><br />
...<br />
}}<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
You can then add those vendor-device ID pairs to the default parameters passed to vfio-pci whenever it is inserted into the kernel.<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.}}<br />
<br />
This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, {{ic|vfio_pci}} and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]]. Should you change the IDs of the devices in {{ic|/etc/modprobe.d/vfio.conf}}, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
=== Using pci-stub (legacy method, pre-4.1 kernels) ===<br />
<br />
If your kernel does not support vfio-pci, you can use the pci-stub module instead, which is a dummy driver used to claim a device and prevent other drivers from using it.<br />
<br />
Pci-stub normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
{{Pkg|linux}} and {{Pkg|linux-lts}} have pci-stub built as a module. You will need to add it to MODULES array in {{ic|mkinitpcio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... pci-stub ...)<br />
}}<br />
<br />
If you did need to add this module to your kernel image configuration manually, you must also [[regenerate the initramfs]].<br />
<br />
Add the relevant PCI device IDs to the {{ic|pci-stubs.ids}} [[kernel parameter]], e.g. {{ic|1=pci-stub.ids=10de:13c2,10de:0fbb}}.<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|pci-stub}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|pci-stub}} to bind with.}}<br />
<br />
Check dmesg output for successful assignment of the device to pci-stub:<br />
<br />
{{hc|dmesg {{!}} grep pci-stub|<br />
[ 2.390128] pci-stub: add 10DE:13C2 sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390143] pci-stub 0000:06:00.0: claimed by stub<br />
[ 2.390150] pci-stub: add 10DE:0FBB sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390159] pci-stub 0000:06:00.1: claimed by stub<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it's possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
After installing {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|ovmf}}, {{Pkg|firewalld}}, and {{Pkg|virt-manager}}, add the path to your OVMF firmware image and runtime variables template to your libvirt config so {{ic|virt-install}} or {{ic|virt-manager}} can find those later on.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
nvram = [<br />
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"<br />
]<br />
}}<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self-explanatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the libvirt [[group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps :<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that you have correctly specified the location of your firmware in {{ic|/etc/libvirt/qemu.conf}} and restart {{ic|libvirtd.service}}.<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, go into "Add Hardware" and add a Controller for SCSI drives of the "VirtIO SCSI" model. You can then change the default IDE disk for a SCSI disk, which will bind to said controller.<br />
** Windows VMs will not recognize those drives by default, so you need to download the ISO containing the drivers from [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ here] and add an IDE (or SATA for Windows 8.1 and newer) CD-ROM storage device linking to said ISO, otherwise you will not be able to get Windows to recognize it during the installation process. When prompted to select a disk to install windows on, load the drivers contained on the CD-ROM under ''vioscsi''.<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it's now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate linux/windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. Since switching between threads adds a bit of overhead (because context switching forces the core to change its cache between operations), this can noticeably harm performance on the guest. CPU pinning aims to resolve this as it overrides process scheduling and ensures that the VM threads will always run and only run on those specific cores. Here, for instance, the guest cores 0, 1, 2 and 3 are mapped to the host cores 4, 5, 6 and 7 respectively.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
==== The case of Hyper-threading ====<br />
<br />
If your CPU supports hardware multitasking, also known as Hyper-threading on Intel chips, there are two ways you can go with your CPU pinning. That is, Hyper-threading is simply a very efficient way of running two threads on one CPU at any given time, so while it may give you 8 logical cores on what would otherwise be a quad-core CPU, if the physical core is overloaded, the logical core will not be of any use. One could pin their VM threads on 2 physical cores and their 2 respective threads, but any task overloading those two cores will not be helped by the extra two logical cores, since in the end you are only passing through two cores out of four, not four out of eight. What you should do knowing this depends on what you intend to do with your host while your VM is running.<br />
<br />
This is the abridged content of {{ic|/proc/cpuinfo}} on a quad-core machine with hyper-threading.<br />
<br />
{{hc|$ grep -e "processor" -e "core id" -e "^$" /proc/cpuinfo|<br />
processor : 0<br />
core id : 0<br />
<br />
processor : 1<br />
core id : 1<br />
<br />
processor : 2<br />
core id : 2<br />
<br />
processor : 3<br />
core id : 3<br />
<br />
processor : 4<br />
core id : 0<br />
<br />
processor : 5<br />
core id : 1<br />
<br />
processor : 6<br />
core id : 2<br />
<br />
processor : 7<br />
core id : 3<br />
}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, it would probably be better to pin your VM threads across all of your logical cores, so that the VM can fully take advantage of the spare CPU time on all your cores.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='4' threads='1'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
If you would instead prefer to have the host and guest running intensive tasks at the same time, it would then be preferable to pin a limited amount of physical cores and their respective threads on the guest and leave the rest to the host to avoid the two competing for CPU time.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='6'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='2' threads='2'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
=== Static huge pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information accross multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4GB of memory divided into 4kB pages (which is the default size for normal pages), meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession. This is normally handeled with transparent huge pages, which dynamically manages hugepages to keep up with the demand.<br />
<br />
On a VM with a PCI passthrough, however, it is '''not possible''' to benefit from transparent huge pages, as IOMMU requires that the guest's memory be allocated and pinned as soon as the VM starts. It is therefore required to allocate huge pages statically in order to benefit from them. <br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4GBs worth of huge pages on a machine with 8GBs of memory will only leave you with 4GBs of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel comand line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048kB per huge page creates 2GBs worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GB huge page support could be verified by {{ic|<nowiki>grep pdpe1gb /proc/cpuinfo</nowiki>}}. Setting 1 GB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X transparent_hugepage=never}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== High DPC Latency ===<br />
<br />
{{Accuracy|As far as I can tell all virtio modules listed here are for virtual devices used when Linux runs as a guest. Loading them on the host serves no purpose.}}<br />
<br />
If you are experiencing high DPC and/or interrupt latency in your Guest VM, ensure you have [[Kernel modules#Manual module handling|loaded the needed virtio kernel modules]] on the host kernel. Loadable virtio kernel modules include: {{ic|virtio-pci}}, {{ic|virtio-net}}, {{ic|virtio-blk}}, {{ic|virtio-balloon}}, {{ic|virtio-ring}} and {{ic|virtio}}. <br />
<br />
After loading one or more of these modules, {{ic|lsmod {{!}} grep virtio}} executed on the host should not return empty.<br />
<br />
==== CPU pinning with isolcpus ====<br />
<br />
Alternatively, make sure that you have isolated CPUs properly. In this example, let us assume you are using CPUs 4-7.<br />
Use the kernel parameters {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. <br />
<br />
{{hc|sudo vim /etc/defaults/grub|<nowiki><br />
...<br />
GRUB_CMDLINE_LINUX="..your other params.. isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"<br />
...<br />
</nowiki>}}<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this reddit comment] for more info.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you're running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you're on the latest). If you're running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
=== Further Tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide]. This guide may be especially helpful if you are experiencing:<br />
<br />
* Moderate CPU load on the host during downloads/uploads from within the guest: See ''Bridge Zero Copy Transmit'' for a potential fix.<br />
* Guests capping out at certain network speeds during downloads/uploads despite virtio-net being used: See ''Multi-queue virtio-net'' for a potential fix.<br />
* Guests "stuttering" under high I/O, despite the same workload not affecting hosts to the same degree: See ''Multi-queue virtio-scsi'' for a potential fix.<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how both pci-stub and vfio-pci use your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/devices/pci*/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
GROUP="0000:00:03.0"<br />
DEVS="0000:03:00.0 0000:03:00.1 ."<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Create {{ic|/etc/modprobe.d/vfio.conf}} with the following:<br />
<br />
install vfio-pci /usr/bin/vfio-pci-override.sh<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}<br />
<br />
Remove any video drivers from MODULES, and add {{ic|vfio-pci}}, and {{ic|vfio_iommu_type1}}<br />
<br />
MODULES=(ext4 vfat vfio-pci vfio_iommu_type1)<br />
<br />
Add {{ic|/etc/modprobe.d/vfio.conf}} and {{ic|/usr/bin/vfio-pci-override.sh}} to FILES:<br />
<br />
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)<br />
<br />
[[Regenerate the initramfs]] and reboot:<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|Not possible at the time as far as I'm aware, but a common issue on various forums.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that's not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.hostfission.com/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total megabytes + 2<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MB + 2 = 17.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 17.82 is bigger than 16 we should choose 32.<br />
<br />
It is also possible to calculate this automatically with {{AUR|glasscalc}}.<br />
<br />
Next create a script to create a shared memory file.<br />
<br />
{{hc|/usr/local/bin/looking-glass-init.sh|2=<br />
#!/bin/sh<br />
<br />
touch /dev/shm/looking-glass<br />
chown '''user''':kvm /dev/shm/looking-glass<br />
chmod 660 /dev/shm/looking-glass<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Create a [[systemd]] unit to execute this script during boot<br />
<br />
{{hc|/etc/systemd/system/looking-glass-init.service|2=<br />
[Unit]<br />
Description=Create shared memory for looking glass<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/local/bin/looking-glass-init.sh<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
}}<br />
<br />
[[Start]] and [[enable]] {{ic|looking-glass-init.service}}<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to go into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download a signed driver [https://github.com/virtio-win/kvm-guest-drivers-windows/issues/217 from issue 217], as it is not yet available elsewhere.<br />
<br />
Once the driver is installed you must download the latest [https://looking-glass.hostfission.com/downloads looking-glass-host] from the looking glass website and start it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]<br />
It is also possible to make it start automatically on VM boot by editing the {{ic|HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run}} registry and adding a path to the downloaded executable.<br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you don't want to use Spice to control the guest mouse and keyboard you can disable the Spice server<br />
<br />
$ looking-glass-client -s<br />
<br />
Refer to <br />
<br />
$ looking-glass-client --help<br />
<br />
for more info<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream}} is typically sufficient.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it's not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It's recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing though other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting :[[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio environment variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
<qemu:commandline><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_AUDIO_DRV' value<nowiki>=</nowiki>'pa'/><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_PA_SERVER' value<nowiki>=</nowiki>'/run/user/1000/pulse/native'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
Change 1000 under the user directory to your user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and [[systemd/User|user service]] {{ic|pulseaudio.service}}.<br />
<br />
Virtual Machine audio will now be routed through the host as an application. The application {{Pkg|pavucontrol}} can be used to control the output device. Be aware that on Windows guests, this can cause audio crackling without [[#Slowed down audio pumped through HDMI on the video card|using Message-Signaled Interrupts.]]<br />
<br />
=== Physical Disk/Partition ===<br />
<br />
A whole disk or a partition may be used as a whole for improved I/O performance by adding an entry to XML<br />
<br />
{{hc|$ virsh edit [vmname]| <nowiki><br />
<devices><br />
...<br />
<disk type='block' device='disk'><br />
<driver name='qemu' type='raw' cache='none' io='native'/><br />
<source dev='/dev/sdXX'/><br />
<target dev='vda' bus='virtio'/><br />
<address type='pci' domain='0x0000' bus='0x02' slot='0x0a' function='0x0'/><br />
</disk><br />
...<br />
</devices><br />
</nowiki><br />
}}<br />
<br />
This would require a driver on Windows guests, refer to [[#Setting up the guest OS]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functionnal and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
If you have trouble configuring a certain mechanism in your setup, you might want to look up [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]]. A few users have described their setups and you might want to look up certain tricks from their configuration files.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the cpu parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. It may help for the ID to be set to a 12-character alphanumeric (e.g. '123456789ab') as opposed to longer or shorter strings.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
==== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ====<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
=== UEFI (OVMF) Compatability in VBIOS ===<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it's not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it's stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it's pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
Some tools named {{ic|MSI_util}} or similar are available on the Internet, but they do not work on Windows 10 64bit.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read {{ic|Graphics Problems?}} in [https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt Intel-IOMMU.txt] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X doesn't start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user account is added to the kvm and libvirt [[groups]].<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script from https://www.youtube.com/watch?v=37D2bRsthfI]<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://webchat.freenode.net/?channels=vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=510636PCI passthrough via OVMF2018-02-13T13:24:35Z<p>NeoTheFox: /* Adding IVSHMEM Device to VM */ Added glasscalc</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=processors&VTD=true List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Merge|QEMU#Enabling IOMMU (Intel VT-d/AMD-Vi) support|Same topic.}}<br />
<br />
IOMMU is a system specific IO mapping mechanism and can be used with most devices. IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
<br />
Before you enable IOMMU, you might have to first enable (non-IOMMU) virtualisation (Intel VT-x/"Vanderpool" or AMD-V/"Pacifica") in your BIOS settings.<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is enabled in your BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
You will also have to enable iommu support in the kernel itself through a [[Kernel parameters|bootloader kernel option]]. Depending on your type of CPU, use either {{ic|1=intel_iommu=on}} for Intel CPUs (VT-d) or {{ic|1=amd_iommu=on}} for AMD CPUs (AMD-Vi).<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for d in /sys/kernel/iommu_groups/*/devices/*; do <br />
n=${d#*/iommu_groups/*}; n=${n%%/*}<br />
printf 'IOMMU Group %s ' "$n"<br />
lspci -nns "${d##*/}"<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0104] (rev 09)<br />
IOMMU Group 1 00:16.0 Communication controller [0780]: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 [8086:1c3a] (rev 04)<br />
IOMMU Group 2 00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 04)<br />
IOMMU Group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev <br />
...<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will be appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the prefered method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this.}}<br />
<br />
=== Using vfio-pci ===<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use. If your system supports it, which you can try by running the following command, you should use it. If it returns an error, use pci-stub instead.<br />
<br />
{{hc|$ modinfo vfio-pci|<br />
filename: /lib/modules/4.4.5-1-ARCH/kernel/drivers/vfio/pci/vfio-pci.ko.gz<br />
description: VFIO PCI - User Level meta-driver<br />
author: Alex Williamson <alex.williamson@redhat.com><br />
...<br />
}}<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
You can then add those vendor-device ID pairs to the default parameters passed to vfio-pci whenever it is inserted into the kernel.<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.}}<br />
<br />
This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, {{ic|vfio_pci}} and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]]. Should you change the IDs of the devices in {{ic|/etc/modprobe.d/vfio.conf}}, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
=== Using pci-stub (legacy method, pre-4.1 kernels) ===<br />
<br />
If your kernel does not support vfio-pci, you can use the pci-stub module instead, which is a dummy driver used to claim a device and prevent other drivers from using it.<br />
<br />
Pci-stub normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
{{Pkg|linux}} and {{Pkg|linux-lts}} have pci-stub built as a module. You will need to add it to MODULES array in {{ic|mkinitpcio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... pci-stub ...)<br />
}}<br />
<br />
If you did need to add this module to your kernel image configuration manually, you must also [[regenerate the initramfs]].<br />
<br />
Add the relevant PCI device IDs to the {{ic|pci-stubs.ids}} [[kernel parameter]], e.g. {{ic|1=pci-stub.ids=10de:13c2,10de:0fbb}}.<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|pci-stub}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|pci-stub}} to bind with.}}<br />
<br />
Check dmesg output for successful assignment of the device to pci-stub:<br />
<br />
{{hc|dmesg {{!}} grep pci-stub|<br />
[ 2.390128] pci-stub: add 10DE:13C2 sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390143] pci-stub 0000:06:00.0: claimed by stub<br />
[ 2.390150] pci-stub: add 10DE:0FBB sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390159] pci-stub 0000:06:00.1: claimed by stub<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it's possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
After installing {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|ovmf}}, {{Pkg|firewalld}}, and {{Pkg|virt-manager}}, add the path to your OVMF firmware image and runtime variables template to your libvirt config so {{ic|virt-install}} or {{ic|virt-manager}} can find those later on.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
nvram = [<br />
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"<br />
]<br />
}}<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self-explanatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the libvirt [[group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps :<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that you have correctly specified the location of your firmware in {{ic|/etc/libvirt/qemu.conf}} and restart {{ic|libvirtd.service}}.<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, go into "Add Hardware" and add a Controller for SCSI drives of the "VirtIO SCSI" model. You can then change the default IDE disk for a SCSI disk, which will bind to said controller.<br />
** Windows VMs will not recognize those drives by default, so you need to download the ISO containing the drivers from [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ here] and add an IDE (or SATA for Windows 8.1 and newer) CD-ROM storage device linking to said ISO, otherwise you will not be able to get Windows to recognize it during the installation process. When prompted to select a disk to install windows on, load the drivers contained on the CD-ROM under ''vioscsi''.<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it's now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate linux/windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. Since switching between threads adds a bit of overhead (because context switching forces the core to change its cache between operations), this can noticeably harm performance on the guest. CPU pinning aims to resolve this as it overrides process scheduling and ensures that the VM threads will always run and only run on those specific cores. Here, for instance, the guest cores 0, 1, 2 and 3 are mapped to the host cores 4, 5, 6 and 7 respectively.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
==== The case of Hyper-threading ====<br />
<br />
If your CPU supports hardware multitasking, also known as Hyper-threading on Intel chips, there are two ways you can go with your CPU pinning. That is, Hyper-threading is simply a very efficient way of running two threads on one CPU at any given time, so while it may give you 8 logical cores on what would otherwise be a quad-core CPU, if the physical core is overloaded, the logical core will not be of any use. One could pin their VM threads on 2 physical cores and their 2 respective threads, but any task overloading those two cores will not be helped by the extra two logical cores, since in the end you are only passing through two cores out of four, not four out of eight. What you should do knowing this depends on what you intend to do with your host while your VM is running.<br />
<br />
This is the abridged content of {{ic|/proc/cpuinfo}} on a quad-core machine with hyper-threading.<br />
<br />
{{hc|$ grep -e "processor" -e "core id" -e "^$" /proc/cpuinfo|<br />
processor : 0<br />
core id : 0<br />
<br />
processor : 1<br />
core id : 1<br />
<br />
processor : 2<br />
core id : 2<br />
<br />
processor : 3<br />
core id : 3<br />
<br />
processor : 4<br />
core id : 0<br />
<br />
processor : 5<br />
core id : 1<br />
<br />
processor : 6<br />
core id : 2<br />
<br />
processor : 7<br />
core id : 3<br />
}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, it would probably be better to pin your VM threads across all of your logical cores, so that the VM can fully take advantage of the spare CPU time on all your cores.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='4' threads='1'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
If you would instead prefer to have the host and guest running intensive tasks at the same time, it would then be preferable to pin a limited amount of physical cores and their respective threads on the guest and leave the rest to the host to avoid the two competing for CPU time.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='6'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='2' threads='2'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
=== Static huge pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information accross multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4GB of memory divided into 4kB pages (which is the default size for normal pages), meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession. This is normally handeled with transparent huge pages, which dynamically manages hugepages to keep up with the demand.<br />
<br />
On a VM with a PCI passthrough, however, it is '''not possible''' to benefit from transparent huge pages, as IOMMU requires that the guest's memory be allocated and pinned as soon as the VM starts. It is therefore required to allocate huge pages statically in order to benefit from them. <br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4GBs worth of huge pages on a machine with 8GBs of memory will only leave you with 4GBs of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel comand line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048kB per huge page creates 2GBs worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GB huge page support could be verified by {{ic|<nowiki>grep pdpe1gb /proc/cpuinfo</nowiki>}}. Setting 1 GB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X transparent_hugepage=never}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== High DPC Latency ===<br />
<br />
{{Accuracy|As far as I can tell all virtio modules listed here are for virtual devices used when Linux runs as a guest. Loading them on the host serves no purpose.}}<br />
<br />
If you are experiencing high DPC and/or interrupt latency in your Guest VM, ensure you have [[Kernel modules#Manual module handling|loaded the needed virtio kernel modules]] on the host kernel. Loadable virtio kernel modules include: {{ic|virtio-pci}}, {{ic|virtio-net}}, {{ic|virtio-blk}}, {{ic|virtio-balloon}}, {{ic|virtio-ring}} and {{ic|virtio}}. <br />
<br />
After loading one or more of these modules, {{ic|lsmod {{!}} grep virtio}} executed on the host should not return empty.<br />
<br />
==== CPU pinning with isolcpus ====<br />
<br />
Alternatively, make sure that you have isolated CPUs properly. In this example, let us assume you are using CPUs 4-7.<br />
Use the kernel parameters {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. <br />
<br />
{{hc|sudo vim /etc/defaults/grub|<nowiki><br />
...<br />
GRUB_CMDLINE_LINUX="..your other params.. isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"<br />
...<br />
</nowiki>}}<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this reddit comment] for more info.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you're running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you're on the latest). If you're running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
=== Further Tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide]. This guide may be especially helpful if you are experiencing:<br />
<br />
* Moderate CPU load on the host during downloads/uploads from within the guest: See ''Bridge Zero Copy Transmit'' for a potential fix.<br />
* Guests capping out at certain network speeds during downloads/uploads despite virtio-net being used: See ''Multi-queue virtio-net'' for a potential fix.<br />
* Guests "stuttering" under high I/O, despite the same workload not affecting hosts to the same degree: See ''Multi-queue virtio-scsi'' for a potential fix.<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how both pci-stub and vfio-pci use your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/devices/pci*/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
GROUP="0000:00:03.0"<br />
DEVS="0000:03:00.0 0000:03:00.1 ."<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Create {{ic|/etc/modprobe.d/vfio.conf}} with the following:<br />
<br />
install vfio-pci /usr/bin/vfio-pci-override.sh<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}<br />
<br />
Remove any video drivers from MODULES, and add {{ic|vfio-pci}}, and {{ic|vfio_iommu_type1}}<br />
<br />
MODULES=(ext4 vfat vfio-pci vfio_iommu_type1)<br />
<br />
Add {{ic|/etc/modprobe.d/vfio.conf}} and {{ic|/usr/bin/vfio-pci-override.sh}} to FILES:<br />
<br />
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)<br />
<br />
[[Regenerate the initramfs]] and reboot:<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|Not possible at the time as far as I'm aware, but a common issue on various forums.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that's not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.hostfission.com/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total megabytes + 2<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MB + 2 = 17.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 17.82 is bigger than 16 we should choose 32<br />
It is also possible to calculate this automatically with {{AUR|glasscalc}}.<br />
<br />
Next create a script to create a shared memory file.<br />
<br />
{{hc|/usr/local/bin/looking-glass-init.sh|2=<br />
#!/bin/sh<br />
<br />
touch /dev/shm/looking-glass<br />
chown '''user''':kvm /dev/shm/looking-glass<br />
chmod 660 /dev/shm/looking-glass<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Create a [[systemd]] unit to execute this script during boot<br />
<br />
{{hc|/etc/systemd/system/looking-glass-init.service|2=<br />
[Unit]<br />
Description=Create shared memory for looking glass<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/local/bin/looking-glass-init.sh<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
}}<br />
<br />
[[Start]] and [[enable]] {{ic|looking-glass-init.service}}<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to go into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download a signed driver [https://github.com/virtio-win/kvm-guest-drivers-windows/issues/217 from issue 217], as it is not yet available elsewhere.<br />
<br />
Once the driver is installed you must download the latest [https://looking-glass.hostfission.com/downloads looking-glass-host] from the looking glass website and start it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]<br />
It is also possible to make it start automatically on VM boot by editing the {{ic|HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run}} registry and adding a path to the downloaded executable.<br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you don't want to use Spice to control the guest mouse and keyboard you can disable the Spice server<br />
<br />
$ looking-glass-client -s<br />
<br />
Refer to <br />
<br />
$ looking-glass-client --help<br />
<br />
for more info<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream}} is typically sufficient.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it's not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It's recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing though other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting :[[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio environment variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
<qemu:commandline><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_AUDIO_DRV' value<nowiki>=</nowiki>'pa'/><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_PA_SERVER' value<nowiki>=</nowiki>'/run/user/1000/pulse/native'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
Change 1000 under the user directory to your user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and [[systemd/User|user service]] {{ic|pulseaudio.service}}.<br />
<br />
Virtual Machine audio will now be routed through the host as an application. The application {{Pkg|pavucontrol}} can be used to control the output device. Be aware that on Windows guests, this can cause audio crackling without [[#Slowed down audio pumped through HDMI on the video card|using Message-Signaled Interrupts.]]<br />
<br />
=== Physical Disk/Partition ===<br />
<br />
A whole disk or a partition may be used as a whole for improved I/O performance by adding an entry to XML<br />
<br />
{{hc|$ virsh edit [vmname]| <nowiki><br />
<devices><br />
...<br />
<disk type='block' device='disk'><br />
<driver name='qemu' type='raw' cache='none' io='native'/><br />
<source dev='/dev/sdXX'/><br />
<target dev='vda' bus='virtio'/><br />
<address type='pci' domain='0x0000' bus='0x02' slot='0x0a' function='0x0'/><br />
</disk><br />
...<br />
</devices><br />
</nowiki><br />
}}<br />
<br />
This would require a driver on Windows guests, refer to [[#Setting up the guest OS]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functionnal and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
If you have trouble configuring a certain mechanism in your setup, you might want to look up [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]]. A few users have described their setups and you might want to look up certain tricks from their configuration files.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the cpu parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. It may help for the ID to be set to a 12-character alphanumeric (e.g. '123456789ab') as opposed to longer or shorter strings.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
==== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ====<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
=== UEFI (OVMF) Compatability in VBIOS ===<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it's not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it's stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it's pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
Some tools named {{ic|MSI_util}} or similar are available on the Internet, but they do not work on Windows 10 64bit.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read {{ic|Graphics Problems?}} in [https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt Intel-IOMMU.txt] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X doesn't start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user account is added to the kvm and libvirt [[groups]].<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script from https://www.youtube.com/watch?v=37D2bRsthfI]<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://webchat.freenode.net/?channels=vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=510067PCI passthrough via OVMF2018-02-07T21:43:29Z<p>NeoTheFox: Added a link to the upper part of the article that has driver cd link</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=processors&VTD=true List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Merge|QEMU#Enabling IOMMU (Intel VT-d/AMD-Vi) support|Same topic.}}<br />
<br />
IOMMU is a system specific IO mapping mechanism and can be used with most devices. IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
<br />
Before you enable IOMMU, you might have to first enable (non-IOMMU) virtualisation (Intel VT-x/"Vanderpool" or AMD-V/"Pacifica") in your BIOS settings.<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is enabled in your BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
You will also have to enable iommu support in the kernel itself through a [[Kernel parameters|bootloader kernel option]]. Depending on your type of CPU, use either {{ic|1=intel_iommu=on}} for Intel CPUs (VT-d) or {{ic|1=amd_iommu=on}} for AMD CPUs (AMD-Vi).<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for d in /sys/kernel/iommu_groups/*/devices/*; do <br />
n=${d#*/iommu_groups/*}; n=${n%%/*}<br />
printf 'IOMMU Group %s ' "$n"<br />
lspci -nns "${d##*/}"<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0104] (rev 09)<br />
IOMMU Group 1 00:16.0 Communication controller [0780]: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 [8086:1c3a] (rev 04)<br />
IOMMU Group 2 00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 04)<br />
IOMMU Group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev <br />
...<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will be appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the prefered method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this.}}<br />
<br />
=== Using vfio-pci ===<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use. If your system supports it, which you can try by running the following command, you should use it. If it returns an error, use pci-stub instead.<br />
<br />
{{hc|$ modinfo vfio-pci|<br />
filename: /lib/modules/4.4.5-1-ARCH/kernel/drivers/vfio/pci/vfio-pci.ko.gz<br />
description: VFIO PCI - User Level meta-driver<br />
author: Alex Williamson <alex.williamson@redhat.com><br />
...<br />
}}<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
You can then add those vendor-device ID pairs to the default parameters passed to vfio-pci whenever it is inserted into the kernel.<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.}}<br />
<br />
This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, {{ic|vfio_pci}} and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]]. Should you change the IDs of the devices in {{ic|/etc/modprobe.d/vfio.conf}}, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
=== Using pci-stub (legacy method, pre-4.1 kernels) ===<br />
<br />
If your kernel does not support vfio-pci, you can use the pci-stub module instead, which is a dummy driver used to claim a device and prevent other drivers from using it.<br />
<br />
Pci-stub normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
{{Pkg|linux}} and {{Pkg|linux-lts}} have pci-stub built as a module. You will need to add it to MODULES array in {{ic|mkinitpcio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... pci-stub ...)<br />
}}<br />
<br />
If you did need to add this module to your kernel image configuration manually, you must also [[regenerate the initramfs]].<br />
<br />
Add the relevant PCI device IDs to the {{ic|pci-stubs.ids}} [[kernel parameter]], e.g. {{ic|1=pci-stub.ids=10de:13c2,10de:0fbb}}.<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|pci-stub}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|pci-stub}} to bind with.}}<br />
<br />
Check dmesg output for successful assignment of the device to pci-stub:<br />
<br />
{{hc|dmesg {{!}} grep pci-stub|<br />
[ 2.390128] pci-stub: add 10DE:13C2 sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390143] pci-stub 0000:06:00.0: claimed by stub<br />
[ 2.390150] pci-stub: add 10DE:0FBB sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390159] pci-stub 0000:06:00.1: claimed by stub<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it's possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
After installing {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|ovmf}}, {{Pkg|firewalld}}, and {{Pkg|virt-manager}}, add the path to your OVMF firmware image and runtime variables template to your libvirt config so {{ic|virt-install}} or {{ic|virt-manager}} can find those later on.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
nvram = [<br />
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"<br />
]<br />
}}<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self-explanatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the libvirt [[group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps :<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that you have correctly specified the location of your firmware in {{ic|/etc/libvirt/qemu.conf}} and restart {{ic|libvirtd.service}}.<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, go into "Add Hardware" and add a Controller for SCSI drives of the "VirtIO SCSI" model. You can then change the default IDE disk for a SCSI disk, which will bind to said controller.<br />
** Windows VMs will not recognize those drives by default, so you need to download the ISO containing the drivers from [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ here] and add an IDE (or SATA for Windows 8.1 and newer) CD-ROM storage device linking to said ISO, otherwise you will not be able to get Windows to recognize it during the installation process. When prompted to select a disk to install windows on, load the drivers contained on the CD-ROM under ''vioscsi''.<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it's now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate linux/windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. Since switching between threads adds a bit of overhead (because context switching forces the core to change its cache between operations), this can noticeably harm performance on the guest. CPU pinning aims to resolve this as it overrides process scheduling and ensures that the VM threads will always run and only run on those specific cores. Here, for instance, the guest cores 0, 1, 2 and 3 are mapped to the host cores 4, 5, 6 and 7 respectively.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
==== The case of Hyper-threading ====<br />
<br />
If your CPU supports hardware multitasking, also known as Hyper-threading on Intel chips, there are two ways you can go with your CPU pinning. That is, Hyper-threading is simply a very efficient way of running two threads on one CPU at any given time, so while it may give you 8 logical cores on what would otherwise be a quad-core CPU, if the physical core is overloaded, the logical core will not be of any use. One could pin their VM threads on 2 physical cores and their 2 respective threads, but any task overloading those two cores will not be helped by the extra two logical cores, since in the end you are only passing through two cores out of four, not four out of eight. What you should do knowing this depends on what you intend to do with your host while your VM is running.<br />
<br />
This is the abridged content of {{ic|/proc/cpuinfo}} on a quad-core machine with hyper-threading.<br />
<br />
{{hc|$ grep -e "processor" -e "core id" -e "^$" /proc/cpuinfo|<br />
processor : 0<br />
core id : 0<br />
<br />
processor : 1<br />
core id : 1<br />
<br />
processor : 2<br />
core id : 2<br />
<br />
processor : 3<br />
core id : 3<br />
<br />
processor : 4<br />
core id : 0<br />
<br />
processor : 5<br />
core id : 1<br />
<br />
processor : 6<br />
core id : 2<br />
<br />
processor : 7<br />
core id : 3<br />
}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, it would probably be better to pin your VM threads across all of your logical cores, so that the VM can fully take advantage of the spare CPU time on all your cores.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='4' threads='1'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
If you would instead prefer to have the host and guest running intensive tasks at the same time, it would then be preferable to pin a limited amount of physical cores and their respective threads on the guest and leave the rest to the host to avoid the two competing for CPU time.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='6'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='2' threads='2'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
=== Static huge pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information accross multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4GB of memory divided into 4kB pages (which is the default size for normal pages), meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession. This is normally handeled with transparent huge pages, which dynamically manages hugepages to keep up with the demand.<br />
<br />
On a VM with a PCI passthrough, however, it is '''not possible''' to benefit from transparent huge pages, as IOMMU requires that the guest's memory be allocated and pinned as soon as the VM starts. It is therefore required to allocate huge pages statically in order to benefit from them. <br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4GBs worth of huge pages on a machine with 8GBs of memory will only leave you with 4GBs of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel comand line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048kB per huge page creates 2GBs worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GB huge page support could be verified by {{ic|<nowiki>grep pdpe1gb /proc/cpuinfo</nowiki>}}. Setting 1 GB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X transparent_hugepage=never}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== High DPC Latency ===<br />
<br />
{{Accuracy|As far as I can tell all virtio modules listed here are for virtual devices used when Linux runs as a guest. Loading them on the host serves no purpose.}}<br />
<br />
If you are experiencing high DPC and/or interrupt latency in your Guest VM, ensure you have [[Kernel modules#Manual module handling|loaded the needed virtio kernel modules]] on the host kernel. Loadable virtio kernel modules include: {{ic|virtio-pci}}, {{ic|virtio-net}}, {{ic|virtio-blk}}, {{ic|virtio-balloon}}, {{ic|virtio-ring}} and {{ic|virtio}}. <br />
<br />
After loading one or more of these modules, {{ic|lsmod {{!}} grep virtio}} executed on the host should not return empty.<br />
<br />
==== CPU pinning with isolcpus ====<br />
<br />
Alternatively, make sure that you have isolated CPUs properly. In this example, let us assume you are using CPUs 4-7.<br />
Use the kernel parameters {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. <br />
<br />
{{hc|sudo vim /etc/defaults/grub|<nowiki><br />
...<br />
GRUB_CMDLINE_LINUX="..your other params.. isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"<br />
...<br />
</nowiki>}}<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this reddit comment] for more info.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you're running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you're on the latest). If you're running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
=== Further Tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide]. This guide may be especially helpful if you are experiencing:<br />
<br />
* Moderate CPU load on the host during downloads/uploads from within the guest: See ''Bridge Zero Copy Transmit'' for a potential fix.<br />
* Guests capping out at certain network speeds during downloads/uploads despite virtio-net being used: See ''Multi-queue virtio-net'' for a potential fix.<br />
* Guests "stuttering" under high I/O, despite the same workload not affecting hosts to the same degree: See ''Multi-queue virtio-scsi'' for a potential fix.<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how both pci-stub and vfio-pci use your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/devices/pci*/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
GROUP="0000:00:03.0"<br />
DEVS="0000:03:00.0 0000:03:00.1 ."<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Create {{ic|/etc/modprobe.d/vfio.conf}} with the following:<br />
<br />
install vfio-pci /usr/bin/vfio-pci-override.sh<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}<br />
<br />
Remove any video drivers from MODULES, and add {{ic|vfio-pci}}, and {{ic|vfio_iommu_type1}}<br />
<br />
MODULES=(ext4 vfat vfio-pci vfio_iommu_type1)<br />
<br />
Add {{ic|/etc/modprobe.d/vfio.conf}} and {{ic|/usr/bin/vfio-pci-override.sh}} to FILES:<br />
<br />
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)<br />
<br />
[[Regenerate the initramfs]] and reboot:<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|Not possible at the time as far as I'm aware, but a common issue on various forums.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that's not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.hostfission.com/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total megabytes + 2<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MB + 2 = 17.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 17.82 is bigger than 16 we should choose 32<br />
<br />
Next create a script to create a shared memory file.<br />
<br />
{{hc|/usr/local/bin/looking-glass-init.sh|2=<br />
#!/bin/sh<br />
<br />
touch /dev/shm/looking-glass<br />
chown '''user''':kvm /dev/shm/looking-glass<br />
chmod 660 /dev/shm/looking-glass<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Create a [[systemd]] unit to execute this script during boot<br />
<br />
{{hc|/etc/systemd/system/looking-glass-init.service|2=<br />
[Unit]<br />
Description=Create shared memory for looking glass<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/local/bin/looking-glass-init.sh<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
}}<br />
<br />
[[Start]] and [[enable]] {{ic|looking-glass-init.service}}<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to go into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download a signed driver [https://github.com/virtio-win/kvm-guest-drivers-windows/issues/217 from issue 217], as it is not yet available elsewhere.<br />
<br />
Once the driver is installed you must download the latest [https://looking-glass.hostfission.com/downloads looking-glass-host] from the looking glass website and start it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]<br />
It is also possible to make it start automatically on VM boot by editing the {{ic|HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run}} registry and adding a path to the downloaded executable.<br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you don't want to use Spice to control the guest mouse and keyboard you can disable the Spice server<br />
<br />
$ looking-glass-client -s<br />
<br />
Refer to <br />
<br />
$ looking-glass-client --help<br />
<br />
for more info<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream}} is typically sufficient.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it's not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It's recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing though other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting :[[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio environment variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
<qemu:commandline><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_AUDIO_DRV' value<nowiki>=</nowiki>'pa'/><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_PA_SERVER' value<nowiki>=</nowiki>'/run/user/1000/pulse/native'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
Change 1000 under the user directory to your user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and [[systemd/User|user service]] {{ic|pulseaudio.service}}.<br />
<br />
Virtual Machine audio will now be routed through the host as an application. The application {{Pkg|pavucontrol}} can be used to control the output device. Be aware that on Windows guests, this can cause audio crackling without [[#Slowed down audio pumped through HDMI on the video card|using Message-Signaled Interrupts.]]<br />
<br />
=== Physical Disk/Partition ===<br />
<br />
A whole disk or a partition may be used as a whole for improved I/O performance by adding an entry to XML<br />
<br />
{{hc|$ virsh edit [vmname]| <nowiki><br />
<devices><br />
...<br />
<disk type='block' device='disk'><br />
<driver name='qemu' type='raw' cache='none' io='native'/><br />
<source dev='/dev/sdXX'/><br />
<target dev='vda' bus='virtio'/><br />
<address type='pci' domain='0x0000' bus='0x02' slot='0x0a' function='0x0'/><br />
</disk><br />
...<br />
</devices><br />
</nowiki><br />
}}<br />
<br />
This would require a driver on Windows guests, refer to [[PCI_passthrough_via_OVMF#Setting_up_the_guest_OS]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functionnal and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
If you have trouble configuring a certain mechanism in your setup, you might want to look up [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]]. A few users have described their setups and you might want to look up certain tricks from their configuration files.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the cpu parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. It may help for the ID to be set to a 12-character alphanumeric (e.g. '123456789ab') as opposed to longer or shorter strings.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
==== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ====<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
=== UEFI (OVMF) Compatability in VBIOS ===<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it's not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it's stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it's pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
Some tools named {{ic|MSI_util}} or similar are available on the Internet, but they do not work on Windows 10 64bit.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read {{ic|Graphics Problems?}} in [https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt Intel-IOMMU.txt] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X doesn't start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user account is added to the kvm and libvirt [[groups]].<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script from https://www.youtube.com/watch?v=37D2bRsthfI]<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://webchat.freenode.net/?channels=vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=510066PCI passthrough via OVMF2018-02-07T21:38:55Z<p>NeoTheFox: /* Passing though other devices */ Added a raw partition/disk passthrough</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=processors&VTD=true List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Merge|QEMU#Enabling IOMMU (Intel VT-d/AMD-Vi) support|Same topic.}}<br />
<br />
IOMMU is a system specific IO mapping mechanism and can be used with most devices. IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
<br />
Before you enable IOMMU, you might have to first enable (non-IOMMU) virtualisation (Intel VT-x/"Vanderpool" or AMD-V/"Pacifica") in your BIOS settings.<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is enabled in your BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
You will also have to enable iommu support in the kernel itself through a [[Kernel parameters|bootloader kernel option]]. Depending on your type of CPU, use either {{ic|1=intel_iommu=on}} for Intel CPUs (VT-d) or {{ic|1=amd_iommu=on}} for AMD CPUs (AMD-Vi).<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for d in /sys/kernel/iommu_groups/*/devices/*; do <br />
n=${d#*/iommu_groups/*}; n=${n%%/*}<br />
printf 'IOMMU Group %s ' "$n"<br />
lspci -nns "${d##*/}"<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0104] (rev 09)<br />
IOMMU Group 1 00:16.0 Communication controller [0780]: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 [8086:1c3a] (rev 04)<br />
IOMMU Group 2 00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 04)<br />
IOMMU Group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev <br />
...<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will be appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the prefered method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this.}}<br />
<br />
=== Using vfio-pci ===<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use. If your system supports it, which you can try by running the following command, you should use it. If it returns an error, use pci-stub instead.<br />
<br />
{{hc|$ modinfo vfio-pci|<br />
filename: /lib/modules/4.4.5-1-ARCH/kernel/drivers/vfio/pci/vfio-pci.ko.gz<br />
description: VFIO PCI - User Level meta-driver<br />
author: Alex Williamson <alex.williamson@redhat.com><br />
...<br />
}}<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
You can then add those vendor-device ID pairs to the default parameters passed to vfio-pci whenever it is inserted into the kernel.<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.}}<br />
<br />
This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, {{ic|vfio_pci}} and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]]. Should you change the IDs of the devices in {{ic|/etc/modprobe.d/vfio.conf}}, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
=== Using pci-stub (legacy method, pre-4.1 kernels) ===<br />
<br />
If your kernel does not support vfio-pci, you can use the pci-stub module instead, which is a dummy driver used to claim a device and prevent other drivers from using it.<br />
<br />
Pci-stub normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
{{Pkg|linux}} and {{Pkg|linux-lts}} have pci-stub built as a module. You will need to add it to MODULES array in {{ic|mkinitpcio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... pci-stub ...)<br />
}}<br />
<br />
If you did need to add this module to your kernel image configuration manually, you must also [[regenerate the initramfs]].<br />
<br />
Add the relevant PCI device IDs to the {{ic|pci-stubs.ids}} [[kernel parameter]], e.g. {{ic|1=pci-stub.ids=10de:13c2,10de:0fbb}}.<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|pci-stub}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|pci-stub}} to bind with.}}<br />
<br />
Check dmesg output for successful assignment of the device to pci-stub:<br />
<br />
{{hc|dmesg {{!}} grep pci-stub|<br />
[ 2.390128] pci-stub: add 10DE:13C2 sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390143] pci-stub 0000:06:00.0: claimed by stub<br />
[ 2.390150] pci-stub: add 10DE:0FBB sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390159] pci-stub 0000:06:00.1: claimed by stub<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it's possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
After installing {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|ovmf}}, {{Pkg|firewalld}}, and {{Pkg|virt-manager}}, add the path to your OVMF firmware image and runtime variables template to your libvirt config so {{ic|virt-install}} or {{ic|virt-manager}} can find those later on.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
nvram = [<br />
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"<br />
]<br />
}}<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self-explanatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the libvirt [[group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps :<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that you have correctly specified the location of your firmware in {{ic|/etc/libvirt/qemu.conf}} and restart {{ic|libvirtd.service}}.<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, go into "Add Hardware" and add a Controller for SCSI drives of the "VirtIO SCSI" model. You can then change the default IDE disk for a SCSI disk, which will bind to said controller.<br />
** Windows VMs will not recognize those drives by default, so you need to download the ISO containing the drivers from [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ here] and add an IDE (or SATA for Windows 8.1 and newer) CD-ROM storage device linking to said ISO, otherwise you will not be able to get Windows to recognize it during the installation process. When prompted to select a disk to install windows on, load the drivers contained on the CD-ROM under ''vioscsi''.<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it's now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate linux/windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. Since switching between threads adds a bit of overhead (because context switching forces the core to change its cache between operations), this can noticeably harm performance on the guest. CPU pinning aims to resolve this as it overrides process scheduling and ensures that the VM threads will always run and only run on those specific cores. Here, for instance, the guest cores 0, 1, 2 and 3 are mapped to the host cores 4, 5, 6 and 7 respectively.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
==== The case of Hyper-threading ====<br />
<br />
If your CPU supports hardware multitasking, also known as Hyper-threading on Intel chips, there are two ways you can go with your CPU pinning. That is, Hyper-threading is simply a very efficient way of running two threads on one CPU at any given time, so while it may give you 8 logical cores on what would otherwise be a quad-core CPU, if the physical core is overloaded, the logical core will not be of any use. One could pin their VM threads on 2 physical cores and their 2 respective threads, but any task overloading those two cores will not be helped by the extra two logical cores, since in the end you are only passing through two cores out of four, not four out of eight. What you should do knowing this depends on what you intend to do with your host while your VM is running.<br />
<br />
This is the abridged content of {{ic|/proc/cpuinfo}} on a quad-core machine with hyper-threading.<br />
<br />
{{hc|$ grep -e "processor" -e "core id" -e "^$" /proc/cpuinfo|<br />
processor : 0<br />
core id : 0<br />
<br />
processor : 1<br />
core id : 1<br />
<br />
processor : 2<br />
core id : 2<br />
<br />
processor : 3<br />
core id : 3<br />
<br />
processor : 4<br />
core id : 0<br />
<br />
processor : 5<br />
core id : 1<br />
<br />
processor : 6<br />
core id : 2<br />
<br />
processor : 7<br />
core id : 3<br />
}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, it would probably be better to pin your VM threads across all of your logical cores, so that the VM can fully take advantage of the spare CPU time on all your cores.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='4' threads='1'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
If you would instead prefer to have the host and guest running intensive tasks at the same time, it would then be preferable to pin a limited amount of physical cores and their respective threads on the guest and leave the rest to the host to avoid the two competing for CPU time.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='6'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='2' threads='2'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
=== Static huge pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information accross multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4GB of memory divided into 4kB pages (which is the default size for normal pages), meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession. This is normally handeled with transparent huge pages, which dynamically manages hugepages to keep up with the demand.<br />
<br />
On a VM with a PCI passthrough, however, it is '''not possible''' to benefit from transparent huge pages, as IOMMU requires that the guest's memory be allocated and pinned as soon as the VM starts. It is therefore required to allocate huge pages statically in order to benefit from them. <br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4GBs worth of huge pages on a machine with 8GBs of memory will only leave you with 4GBs of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel comand line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048kB per huge page creates 2GBs worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GB huge page support could be verified by {{ic|<nowiki>grep pdpe1gb /proc/cpuinfo</nowiki>}}. Setting 1 GB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X transparent_hugepage=never}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== High DPC Latency ===<br />
<br />
{{Accuracy|As far as I can tell all virtio modules listed here are for virtual devices used when Linux runs as a guest. Loading them on the host serves no purpose.}}<br />
<br />
If you are experiencing high DPC and/or interrupt latency in your Guest VM, ensure you have [[Kernel modules#Manual module handling|loaded the needed virtio kernel modules]] on the host kernel. Loadable virtio kernel modules include: {{ic|virtio-pci}}, {{ic|virtio-net}}, {{ic|virtio-blk}}, {{ic|virtio-balloon}}, {{ic|virtio-ring}} and {{ic|virtio}}. <br />
<br />
After loading one or more of these modules, {{ic|lsmod {{!}} grep virtio}} executed on the host should not return empty.<br />
<br />
==== CPU pinning with isolcpus ====<br />
<br />
Alternatively, make sure that you have isolated CPUs properly. In this example, let us assume you are using CPUs 4-7.<br />
Use the kernel parameters {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. <br />
<br />
{{hc|sudo vim /etc/defaults/grub|<nowiki><br />
...<br />
GRUB_CMDLINE_LINUX="..your other params.. isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"<br />
...<br />
</nowiki>}}<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this reddit comment] for more info.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you're running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you're on the latest). If you're running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
=== Further Tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide]. This guide may be especially helpful if you are experiencing:<br />
<br />
* Moderate CPU load on the host during downloads/uploads from within the guest: See ''Bridge Zero Copy Transmit'' for a potential fix.<br />
* Guests capping out at certain network speeds during downloads/uploads despite virtio-net being used: See ''Multi-queue virtio-net'' for a potential fix.<br />
* Guests "stuttering" under high I/O, despite the same workload not affecting hosts to the same degree: See ''Multi-queue virtio-scsi'' for a potential fix.<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how both pci-stub and vfio-pci use your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/devices/pci*/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
GROUP="0000:00:03.0"<br />
DEVS="0000:03:00.0 0000:03:00.1 ."<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Create {{ic|/etc/modprobe.d/vfio.conf}} with the following:<br />
<br />
install vfio-pci /usr/bin/vfio-pci-override.sh<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}<br />
<br />
Remove any video drivers from MODULES, and add {{ic|vfio-pci}}, and {{ic|vfio_iommu_type1}}<br />
<br />
MODULES=(ext4 vfat vfio-pci vfio_iommu_type1)<br />
<br />
Add {{ic|/etc/modprobe.d/vfio.conf}} and {{ic|/usr/bin/vfio-pci-override.sh}} to FILES:<br />
<br />
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)<br />
<br />
[[Regenerate the initramfs]] and reboot:<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|Not possible at the time as far as I'm aware, but a common issue on various forums.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that's not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.hostfission.com/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total megabytes + 2<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MB + 2 = 17.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 17.82 is bigger than 16 we should choose 32<br />
<br />
Next create a script to create a shared memory file.<br />
<br />
{{hc|/usr/local/bin/looking-glass-init.sh|2=<br />
#!/bin/sh<br />
<br />
touch /dev/shm/looking-glass<br />
chown '''user''':kvm /dev/shm/looking-glass<br />
chmod 660 /dev/shm/looking-glass<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Create a [[systemd]] unit to execute this script during boot<br />
<br />
{{hc|/etc/systemd/system/looking-glass-init.service|2=<br />
[Unit]<br />
Description=Create shared memory for looking glass<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/local/bin/looking-glass-init.sh<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
}}<br />
<br />
[[Start]] and [[enable]] {{ic|looking-glass-init.service}}<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to go into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download a signed driver [https://github.com/virtio-win/kvm-guest-drivers-windows/issues/217 from issue 217], as it is not yet available elsewhere.<br />
<br />
Once the driver is installed you must download the latest [https://looking-glass.hostfission.com/downloads looking-glass-host] from the looking glass website and start it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]<br />
It is also possible to make it start automatically on VM boot by editing the {{ic|HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run}} registry and adding a path to the downloaded executable.<br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you don't want to use Spice to control the guest mouse and keyboard you can disable the Spice server<br />
<br />
$ looking-glass-client -s<br />
<br />
Refer to <br />
<br />
$ looking-glass-client --help<br />
<br />
for more info<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream}} is typically sufficient.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it's not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It's recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing though other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting :[[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio environment variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
<qemu:commandline><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_AUDIO_DRV' value<nowiki>=</nowiki>'pa'/><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_PA_SERVER' value<nowiki>=</nowiki>'/run/user/1000/pulse/native'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
Change 1000 under the user directory to your user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and [[systemd/User|user service]] {{ic|pulseaudio.service}}.<br />
<br />
Virtual Machine audio will now be routed through the host as an application. The application {{Pkg|pavucontrol}} can be used to control the output device. Be aware that on Windows guests, this can cause audio crackling without [[#Slowed down audio pumped through HDMI on the video card|using Message-Signaled Interrupts.]]<br />
<br />
=== Physical Disk/Partition ===<br />
<br />
A whole disk or a partition may be used as a whole for improved I/O performance by adding an entry to XML<br />
<br />
{{hc|$ virsh edit [vmname]| <nowiki><br />
<devices><br />
...<br />
<disk type='block' device='disk'><br />
<driver name='qemu' type='raw' cache='none' io='native'/><br />
<source dev='/dev/sdXX'/><br />
<target dev='vda' bus='virtio'/><br />
<address type='pci' domain='0x0000' bus='0x02' slot='0x0a' function='0x0'/><br />
</disk><br />
...<br />
</devices><br />
</nowiki><br />
}}<br />
<br />
This would require a driver on Windows guests<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functionnal and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
If you have trouble configuring a certain mechanism in your setup, you might want to look up [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]]. A few users have described their setups and you might want to look up certain tricks from their configuration files.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the cpu parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. It may help for the ID to be set to a 12-character alphanumeric (e.g. '123456789ab') as opposed to longer or shorter strings.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
==== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ====<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
=== UEFI (OVMF) Compatability in VBIOS ===<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it's not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it's stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it's pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
Some tools named {{ic|MSI_util}} or similar are available on the Internet, but they do not work on Windows 10 64bit.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read {{ic|Graphics Problems?}} in [https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt Intel-IOMMU.txt] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X doesn't start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user account is added to the kvm and libvirt [[groups]].<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script from https://www.youtube.com/watch?v=37D2bRsthfI]<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://webchat.freenode.net/?channels=vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=508576PCI passthrough via OVMF2018-01-26T19:05:43Z<p>NeoTheFox: /* Installing the IVSHMEM Host to Windows guest */ minor typo fix</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=processors&VTD=true List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Merge|QEMU#Enabling IOMMU (Intel VT-d/AMD-Vi) support|Same topic.}}<br />
<br />
IOMMU is a system specific IO mapping mechanism and can be used with most devices. IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
<br />
Before you enable IOMMU, you might have to first enable (non-IOMMU) virtualisation (Intel VT-x/"Vanderpool" or AMD-V/"Pacifica") in your BIOS settings.<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is enabled in your BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
You will also have to enable iommu support in the kernel itself through a [[Kernel parameters|bootloader kernel option]]. Depending on your type of CPU, use either {{ic|1=intel_iommu=on}} for Intel CPUs (VT-d) or {{ic|1=amd_iommu=on}} for AMD CPUs (AMD-Vi).<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for d in /sys/kernel/iommu_groups/*/devices/*; do <br />
n=${d#*/iommu_groups/*}; n=${n%%/*}<br />
printf 'IOMMU Group %s ' "$n"<br />
lspci -nns "${d##*/}"<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0104] (rev 09)<br />
IOMMU Group 1 00:16.0 Communication controller [0780]: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 [8086:1c3a] (rev 04)<br />
IOMMU Group 2 00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 04)<br />
IOMMU Group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev <br />
...<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will be appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the prefered method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this.}}<br />
<br />
=== Using vfio-pci ===<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use. If your system supports it, which you can try by running the following command, you should use it. If it returns an error, use pci-stub instead.<br />
<br />
{{hc|$ modinfo vfio-pci|<br />
filename: /lib/modules/4.4.5-1-ARCH/kernel/drivers/vfio/pci/vfio-pci.ko.gz<br />
description: VFIO PCI - User Level meta-driver<br />
author: Alex Williamson <alex.williamson@redhat.com><br />
...<br />
}}<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
You can then add those vendor-device ID pairs to the default parameters passed to vfio-pci whenever it is inserted into the kernel.<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.}}<br />
<br />
This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, {{ic|vfio_pci}} and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]]. Should you change the IDs of the devices in {{ic|/etc/modprobe.d/vfio.conf}}, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
=== Using pci-stub (legacy method, pre-4.1 kernels) ===<br />
<br />
If your kernel does not support vfio-pci, you can use the pci-stub module instead, which is a dummy driver used to claim a device and prevent other drivers from using it.<br />
<br />
Pci-stub normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
{{Pkg|linux}} and {{Pkg|linux-lts}} have pci-stub built as a module. You will need to add it to MODULES array in {{ic|mkinitpcio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... pci-stub ...)<br />
}}<br />
<br />
If you did need to add this module to your kernel image configuration manually, you must also [[regenerate the initramfs]].<br />
<br />
Add the relevant PCI device IDs to the {{ic|pci-stubs.ids}} [[kernel parameter]], e.g. {{ic|1=pci-stub.ids=10de:13c2,10de:0fbb}}.<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|pci-stub}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|pci-stub}} to bind with.}}<br />
<br />
Check dmesg output for successful assignment of the device to pci-stub:<br />
<br />
{{hc|dmesg {{!}} grep pci-stub|<br />
[ 2.390128] pci-stub: add 10DE:13C2 sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390143] pci-stub 0000:06:00.0: claimed by stub<br />
[ 2.390150] pci-stub: add 10DE:0FBB sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390159] pci-stub 0000:06:00.1: claimed by stub<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it's possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
After installing {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|ovmf}} and {{Pkg|virt-manager}}, add the path to your OVMF firmware image and runtime variables template to your libvirt config so {{ic|virt-install}} or {{ic|virt-manager}} can find those later on.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
nvram = [<br />
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"<br />
]<br />
}}<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self explainatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the libvirt [[group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps :<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that you have correctly specified the location of your firmware in {{ic|/etc/libvirt/qemu.conf}} and restart {{ic|libvirtd.service}}.<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, go into "Add Hardware" and add a Controller for SCSI drives of the "VirtIO SCSI" model. You can then change the default IDE disk for a SCSI disk, which will bind to said controller.<br />
** Windows VMs will not recognize those drives by default, so you need to download the ISO containing the drivers from [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ here] and add an IDE (or SATA for Windows 8.1 and newer) CD-ROM storage device linking to said ISO, otherwise you will not be able to get Windows to recognize it during the installation process. When prompted to select a disk to install windows on, load the drivers contained on the CD-ROM under ''vioscsi''.<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it's now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate linux/windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. Since switching between threads adds a bit of overhead (because context switching forces the core to change its cache between operations), this can noticeably harm performance on the guest. CPU pinning aims to resolve this as it overrides process scheduling and ensures that the VM threads will always run and only run on those specific cores. Here, for instance, the guest cores 0, 1, 2 and 3 are mapped to the host cores 4, 5, 6 and 7 respectively.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
==== The case of Hyper-threading ====<br />
<br />
If your CPU supports hardware multitasking, also known as Hyper-threading on Intel chips, there are two ways you can go with your CPU pinning. That is, Hyper-threading is simply a very efficient way of running two threads on one CPU at any given time, so while it may give you 8 logical cores on what would otherwise be a quad-core CPU, if the physical core is overloaded, the logical core will not be of any use. One could pin their VM threads on 2 physical cores and their 2 respective threads, but any task overloading those two cores will not be helped by the extra two logical cores, since in the end you are only passing through two cores out of four, not four out of eight. What you should do knowing this depends on what you intend to do with your host while your VM is running.<br />
<br />
This is the abridged content of {{ic|/proc/cpuinfo}} on a quad-core machine with hyper-threading.<br />
<br />
{{hc|$ grep -e "processor" -e "core id" -e "^$" /proc/cpuinfo|<br />
processor : 0<br />
core id : 0<br />
<br />
processor : 1<br />
core id : 1<br />
<br />
processor : 2<br />
core id : 2<br />
<br />
processor : 3<br />
core id : 3<br />
<br />
processor : 4<br />
core id : 0<br />
<br />
processor : 5<br />
core id : 1<br />
<br />
processor : 6<br />
core id : 2<br />
<br />
processor : 7<br />
core id : 3<br />
}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, it would probably be better to pin your VM threads across all of your logical cores, so that the VM can fully take advantage of the spare CPU time on all your cores.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='4' threads='1'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
If you would instead prefer to have the host and guest running intensive tasks at the same time, it would then be preferable to pin a limited amount of physical cores and their respective threads on the guest and leave the rest to the host to avoid the two competing for CPU time.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='6'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='2' threads='2'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
=== Static huge pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information accross multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4GB of memory divided into 4kB pages (which is the default size for normal pages), meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession. This is normally handeled with transparent huge pages, which dynamically manages hugepages to keep up with the demand.<br />
<br />
On a VM with a PCI passthrough, however, it is '''not possible''' to benefit from transparent huge pages, as IOMMU requires that the guest's memory be allocated and pinned as soon as the VM starts. It is therefore required to allocate huge pages statically in order to benefit from them. <br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4GBs worth of huge pages on a machine with 8GBs of memory will only leave you with 4GBs of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel comand line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048kB per huge page creates 2GBs worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GB huge page support could be verified by {{ic|<nowiki>grep pdpe1gb /proc/cpuinfo</nowiki>}}. Setting 1 GB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X transparent_hugepage=never}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== High DPC Latency ===<br />
<br />
{{Accuracy|As far as I can tell all virtio modules listed here are for virtual devices used when Linux runs as a guest. Loading them on the host serves no purpose.}}<br />
<br />
If you are experiencing high DPC and/or interrupt latency in your Guest VM, ensure you have [[Kernel modules#Manual module handling|loaded the needed virtio kernel modules]] on the host kernel. Loadable virtio kernel modules include: {{ic|virtio-pci}}, {{ic|virtio-net}}, {{ic|virtio-blk}}, {{ic|virtio-balloon}}, {{ic|virtio-ring}} and {{ic|virtio}}. <br />
<br />
After loading one or more of these modules, {{ic|lsmod {{!}} grep virtio}} executed on the host should not return empty.<br />
<br />
==== CPU pinning with isolcpus ====<br />
<br />
Alternatively, make sure that you have isolated CPUs properly. In this example, let us assume you are using CPUs 4-7.<br />
Use the kernel parameters {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. <br />
<br />
{{hc|sudo vim /etc/defaults/grub|<nowiki><br />
...<br />
GRUB_CMDLINE_LINUX="..your other params.. isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"<br />
...<br />
</nowiki>}}<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this reddit comment] for more info.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you're running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you're on the latest). If you're running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
=== Further Tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide]. This guide may be especially helpful if you are experiencing:<br />
<br />
* Moderate CPU load on the host during downloads/uploads from within the guest: See ''Bridge Zero Copy Transmit'' for a potential fix.<br />
* Guests capping out at certain network speeds during downloads/uploads despite virtio-net being used: See ''Multi-queue virtio-net'' for a potential fix.<br />
* Guests "stuttering" under high I/O, despite the same workload not affecting hosts to the same degree: See ''Multi-queue virtio-scsi'' for a potential fix.<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how both pci-stub and vfio-pci use your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/devices/pci*/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
GROUP="0000:00:03.0"<br />
DEVS="0000:03:00.0 0000:03:00.1 ."<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Create {{ic|/etc/modprobe.d/vfio.conf}} with the following:<br />
<br />
install vfio-pci /usr/bin/vfio-pci-override.sh<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}<br />
<br />
Remove any video drivers from MODULES, and add {{ic|vfio-pci}}, and {{ic|vfio_iommu_type1}}<br />
<br />
MODULES=(ext4 vfat vfio-pci vfio_iommu_type1)<br />
<br />
Add {{ic|/etc/modprobe.d/vfio.conf}} and {{ic|/usr/bin/vfio-pci-override.sh}} to FILES:<br />
<br />
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)<br />
<br />
[[Regenerate the initramfs]] and reboot:<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|Not possible at the time as far as I'm aware, but a common issue on various forums.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that's not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.hostfission.com/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total megabytes + 2<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MB + 2 = 17.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 17.82 is bigger than 16 we should choose 32<br />
<br />
Next create a script to create a shared memory file.<br />
<br />
{{hc|/usr/local/bin/looking-glass-init.sh|2=<br />
#!/bin/sh<br />
<br />
touch /dev/shm/looking-glass<br />
chown '''user''':kvm /dev/shm/looking-glass<br />
chmod 660 /dev/shm/looking-glass<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Create a [[systemd]] unit to execute this script during boot<br />
<br />
{{hc|/etc/systemd/system/looking-glass-init.service|2=<br />
[Unit]<br />
Description=Create shared memory for looking glass<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/local/bin/looking-glass-init.sh<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
}}<br />
<br />
[[Start]] and [[enable]] {{ic|looking-glass-init.service}}<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to go into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download a signed driver [https://github.com/virtio-win/kvm-guest-drivers-windows/issues/217 from issue 217], as it is not yet available elsewhere.<br />
<br />
Once the driver is installed you must download the latest [https://looking-glass.hostfission.com/downloads looking-glass-host] from the looking glass website and start it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]<br />
It is also possible to make it start automatically on VM boot by editing the {{ic|HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run}} registry and adding a path to the downloaded executable.<br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you don't want to use Spice to control the guest mouse and keyboard you can disable the Spice server<br />
<br />
$ looking-glass-client -s<br />
<br />
Refer to <br />
<br />
$ looking-glass-client --help<br />
<br />
for more info<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream}} is typically sufficient.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it's not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It's recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing though other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting :[[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio environment variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
<qemu:commandline><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_AUDIO_DRV' value<nowiki>=</nowiki>'pa'/><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_PA_SERVER' value<nowiki>=</nowiki>'/run/user/1000/pulse/native'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
Change 1000 under the user directory to your user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and [[systemd/User|user service]] {{ic|pulseaudio.service}}.<br />
<br />
Virtual Machine audio will now be routed through the host as an application. The application {{Pkg|pavucontrol}} can be used to control the output device. Be aware that on Windows guests, this can cause audio crackling without [[#Slowed down audio pumped through HDMI on the video card|using Message-Signaled Interrupts.]]<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functionnal and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
If you have trouble configuring a certain mechanism in your setup, you might want to look up [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]]. A few users have described their setups and you might want to look up certain tricks from their configuration files.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the cpu parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. It may help for the ID to be set to a 12-character alphanumeric (e.g. '123456789ab') as opposed to longer or shorter strings.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
==== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ====<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
=== UEFI (OVMF) Compatability in VBIOS ===<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it's not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it's stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it's pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
Some tools named {{ic|MSI_util}} or similar are available on the Internet, but they do not work on Windows 10 64bit.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read {{ic|Graphics Problems?}} in [https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt Intel-IOMMU.txt] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X doesn't start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user account is added to the kvm and libvirt [[groups]].<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script from https://www.youtube.com/watch?v=37D2bRsthfI]<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://webchat.freenode.net/?channels=vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Nextcloud&diff=508575Nextcloud2018-01-26T19:04:24Z<p>NeoTheFox: Undo revision 508573 by NeoTheFox (talk) the template is not there yet</p>
<hr />
<div>[[Category:File sharing]]<br />
[[Category:Web applications]]<br />
[[fr:Owncloud]]<br />
[[ja:Nextcloud]]<br />
{{Related articles start}}<br />
{{Related|Apache HTTP Server}}<br />
{{Related|Nginx}}<br />
{{Related|OpenSSL}}<br />
{{Related|WebDAV}}<br />
{{Related articles end}}<br />
<br />
From [[Wikipedia:Nextcloud|Wikipedia]]:<br />
<br />
:Nextcloud is a suite of client-server software for creating and using file hosting services. It is functionally similar to Dropbox, although Nextcloud is free and open-source, allowing anyone to install and operate it on a private server. In contrast to proprietary services like Dropbox, the open architecture allows adding additional functionality to the server in form of applications.<br />
<br />
Nextcloud is a fork of ownCloud. For differences between the two, see [[wikipedia:Nextcloud#Differences from ownCloud]].<br />
<br />
== Prerequisites ==<br />
Nextcloud requires several components:<br />
* A web server: [[Apache]] or [[nginx]]<br />
* A database: [[MariaDB]] or [[PostgreSQL]]<br />
* [[PHP]] with [[#PHP_setup|additional modules]].<br />
<br />
These will be configured in [[#Setup]].<br />
<br />
Make sure the required components are installed before proceeding.<br />
<br />
== Installation ==<br />
<br />
[[Install]] the {{Pkg|nextcloud}} package.<br />
<br />
== Setup ==<br />
<br />
As stated above, in order to setup Nextcloud, you must set up the appropriate PHP requirements;<br />
additionally, you must configure a database and a webserver.<br />
<br />
=== Pacman hook ===<br />
<br />
To do nextcloud database upgrade automatically you may set up pacman post upgrade hook based on following example:<br />
<br />
# Update Nextcloud when core or -apps are touched<br />
<br />
[Trigger]<br />
Operation = Install<br />
Operation = Upgrade<br />
Type = Package<br />
Target = nextcloud<br />
Target = nextcloud-app-*<br />
<br />
[Action]<br />
Description = Updating Nextcloud installation<br />
When = PostTransaction<br />
Exec = /usr/bin/runuser -u http -- /usr/bin/php /usr/share/webapps/nextcloud/occ upgrade<br />
<br />
You need to put it into /etc/pacman.d/hooks/nextcloud.hook if you did not customize HookDir in pacman.conf.<br />
<br />
See also [[Pacman#Hooks]]<br />
<br />
=== PHP setup ===<br />
<br />
{{Note|Nextcloud does not support PHP 7.2 as of version 12.0.4. [https://github.com/nextcloud/server/issues/6786] Until version 13 is released, [[install]] the {{pkg|php71}} package or update to {{AUR|nextcloud-testing}}.}}<br />
<br />
{{Tip|For all prerequisite PHP modules, see upstream documentation: [https://docs.nextcloud.com/server/12/admin_manual/installation/source_installation.html#prerequisites-label Nextcloud 12.0].}}<br />
<br />
Install [[PHP#gd]] and {{pkg|php-intl}} as additional modules.<br />
<br />
Some apps(News for example) require the iconv extension, if you wish to use these apps, uncomment the extension in {{ic|/etc/php/php.ini}}.<br />
<br />
Depending on which database backend will be used:<br />
* For [[MySQL]], see [[PHP#MySQL/MariaDB]].<br />
* For [[PostgreSQL]], see [[PHP#PostgreSQL]].<br />
* For [[SQLite]], see [[PHP#Sqlite]].<br />
<br />
Performance may be improved through the implementation of [[PHP#Caching|caching]], see<br />
[https://docs.nextcloud.com/server/12/admin_manual/configuration_server/caching_configuration.html Configuring Memory Caching]<br />
on the official documentation for details.<br />
<br />
=== Database setup ===<br />
<br />
An SQL database must be setup and used for your Nextcloud installation. After setting up the database here,<br />
you will be prompted for its information when you first create an administrator account.<br />
<br />
==== MariaDB ====<br />
<br />
{{Note|It's is highly recommended to set {{ic|binlog_format}} to ''mixed'' [https://docs.nextcloud.com/server/12/admin_manual/configuration_database/linux_database_configuration.html#db-binlog-label] in {{ic|/etc/mysql/my.cnf}}.}}<br />
<br />
The following is an example of setting up a [[MariaDB]] database and user:<br />
<br />
{{hc|$ mysql -u root -p|2=<br />
mysql> CREATE DATABASE `'''nextcloud'''` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`;<br />
mysql> CREATE USER `'''nextcloud'''`@'localhost' IDENTIFIED BY ''''password'''';<br />
mysql> GRANT ALL PRIVILEGES ON `'''nextcloud'''`.* TO `'''nextcloud'''`@`localhost`;<br />
mysql> \q<br />
}}<br />
<br />
==== PostgreSQL ====<br />
{{Warning|1=Nextcloud 12 is unable to do migration with PostgreSQL version 10 (currently in the repositories), as noted in this [https://bugs.archlinux.org/task/56302?project=5&string=nextcloud bug report].<br />
While not officially supported, once the migration done, Nextcloud 12 seems to work as expected with PostgreSQL 10.}}<br />
<br />
The following is an example of setting up a [[PostgreSQL]] user and database:<br />
{{hc|1=$ sudo -u postgres createuser -h localhost -P nextcloud|2=<br />
Enter password for new role:<br />
Enter it again:<br />
}}<br />
<br />
$ sudo -u postgres createdb -O nextcloud nextcloud<br />
<br />
=== Webserver setup ===<br />
<br />
{{Warning|It is recommended to use TLS/SSL (HTTPS) over plain HTTP, see [[Apache#TLS/SSL]] or [[Nginx#TLS/SSL]] for examples and implement this in the examples given below.}}<br />
<br />
Depending on which [[webserver]] you are using, further setup is required, indicated below.<br />
<br />
==== Apache ====<br />
<br />
If you haven't already, install [[Apache]] and install and enable [[Apache#PHP|Apache's PHP module]]<br />
<br />
Copy the Apache configuration file to the configuration directory:<br />
<br />
# cp /etc/webapps/nextcloud/apache.example.conf /etc/httpd/conf/extra/nextcloud.conf<br />
<br />
Modify the file according to your preferences. By default it includes an alias<br />
for {{ic|/nextcloud}} pointing to {{ic|/usr/share/webapps/nextcloud}}.<br />
<br />
And include it in {{ic|/etc/httpd/conf/httpd.conf}}:<br />
Include conf/extra/nextcloud.conf<br />
<br />
Ensure that the root location of your Nextcloud installation (e.g., {{ic|/usr/share/webapps/nextcloud}}) is accessible by the webserver's user {{ic|http}}.<br />
<br />
Now restart Apache ({{ic|httpd.service}}).<br />
<br />
===== WebDAV =====<br />
Nextcloud comes with its own [[WebDAV]] implementation enabled, which may conflict with the one shipped with Apache. If you have enabled WebDAV in Apache (not enabled by default), disable the modules {{ic|mod_dav}} and {{ic|mod_dav_fs}} in {{ic|/etc/httpd/conf/httpd.conf}}. See [https://forum.owncloud.org/viewtopic.php?f=17&t=7240] for details.<br />
<br />
==== Nginx ====<br />
Create an empty directory to hold the cloud-specific config file:<br />
# mkdir /etc/nginx/conf.d/<br />
<br />
In {{ic|/etc/nginx/nginx.conf}}, add the following lines under the "http" section:<br />
<br />
http {<br />
...<br />
...<br />
server_names_hash_bucket_size 64;<br />
include conf.d/*.conf;<br />
}<br />
<br />
<br />
Create a config file {{ic|/etc/nginx/conf.d/nextcloud.conf}} according to the [https://docs.nextcloud.com/server/12/admin_manual/installation/nginx.html documentation]. You will have to change the {{ic|root}} location, as the Arch package installs to {{ic|/usr/share/webapps/nextcloud}} instead of {{ic|/var/www/nextcloud}}.<br />
<br />
Addtitionally, you change the php-handler block so it looks like this one <br />
<br />
upstream php-handler {<br />
server unix:/run/php-fpm/php-fpm.sock;<br />
}<br />
<br />
in the {{ic|/etc/nginx/conf.d/nextcloud.conf}} file.<br />
<br />
From this point on, it is recommended to obtain a secure-certificates using [[Let's Encrypt]], see [[#Security Hardening]].<br />
<br />
===== PHP-FPM configuration =====<br />
<br />
Make sure PHP-FPM has been configured correctly as described in [[Nginx#FastCGI]].<br />
<br />
Uncomment {{ic|1=env[PATH] = /usr/local/bin:/usr/bin:/bin}} in {{ic|/etc/php/php-fpm.d/www.conf}} and [[restart]] {{ic|php-fpm.service}} to apply the changes.<br />
<br />
== Initialize ==<br />
<br />
Open the address where you have installed Nextcloud in a web browser (e.g., https://www.examples.com/nextcloud).<br />
<br />
=== Create storage directories ===<br />
<br />
You will now see "Cannot write into "apps" directory". This is because Arch packages Nextcloud in a way where the ''apps'' folder only has the webserver as a group without web permissions and the ''data'' folder is nonexistent.<br />
<br />
The easiest non-conflicting way is to create a new writable folder for apps and also create a writable data folder. Replace the http group with the group your webserver uses if needed.<br />
<br />
# mkdir -p /usr/share/webapps/nextcloud/data<br />
# mkdir -p /usr/share/webapps/nextcloud/apps2<br />
# chown http:http /usr/share/webapps/nextcloud/data<br />
# chown http:http /usr/share/webapps/nextcloud/apps2<br />
# chmod 700 /usr/share/webapps/nextcloud/data<br />
# chmod 700 /usr/share/webapps/nextcloud/apps2<br />
<br />
Next edit the configuration file at {{ic|/etc/webapps/nextcloud/config/config.php}} and add following lines before the closing {{ic|);}} of the file.<br />
<br />
{{hc|/etc/webapps/nextcloud/config/config.php|2=<br />
'apps_paths' =><br />
array (<br />
0 =><br />
array (<br />
'path' => '/usr/share/webapps/nextcloud/apps',<br />
'url' => '/apps',<br />
'writable' => false,<br />
),<br />
1 =><br />
array (<br />
'path' => '/usr/share/webapps/nextcloud/apps2',<br />
'url' => '/apps2',<br />
'writable' => true,<br />
),<br />
),<br />
'datadirectory' => '/usr/share/webapps/nextcloud/data'<br />
}}<br />
<br />
Refresh the page and the error should be gone.<br />
<br />
From there follow the instructions in adding an administrator account as well as selecting the database you created earlier.<br />
<br />
== Security Hardening ==<br />
<br />
The [https://docs.nextcloud.com/server/12/admin_manual/configuration_server/harden_server.html Nextcloud Hardening and Security]<br />
article guides along generic topics. See also the project's [https://scan.nextcloud.com/ Security scanner].<br />
<br />
=== Let's Encrypt ===<br />
==== nginx ====<br />
{{Move|Let's Encrypt|A simple example of configuring an 'empty' domain for Let's Encrypt should be created. The example file given is too big and could be far easier.}}<br />
1. Create the cloud configuration {{ic|/etc/nginx/conf.d/cloud-initial.conf}} using [https://github.com/graysky2/configs/blob/master/nginx/nextcloud-initial.conf this initial file] as a template. Substitute the literal "@@FQDN@@" in the template file with the actual [[wikipedia:Fully_qualified_domain_name|FQDN]] to be used. The certs for the server need to be generated using this unencrypted configuration initially. Follow the steps outlined on [[Let’s Encrypt]] to generate the server encryption certificates.<br />
<br />
2. Upon successfully generating certificates, replace {{ic|/etc/nginx/conf.d/cloud-initial.conf}} (it may be safely renamed so long as it does not end in ".conf" or simply deleted) with a new file, {{ic|/etc/nginx/conf.d/cloud.conf}} using [https://github.com/graysky2/configs/blob/master/nginx/nextcloud.conf this file] as a template. Again, substitute the literal "@@FQDN@@" in the template file with the actual [[wikipedia:Fully_qualified_domain_name|FQDN]] to be used. [[Start]] and optionally [[enable]] {{ic|nginx.service}}.<br />
<br />
=== uWSGI ===<br />
<br />
You can run Nextcloud in its own process and service by using the [[uWSGI]] application server with {{pkg|uwsgi-plugin-php}}. This allows you to define a [[PHP#Configuration|PHP configuration]] only for this instance of PHP, without the need to edit the global {{ic|php.ini}} and thus keeping your web application configurations compartmentalized. ''uWSGI'' itself has a wealth of features to limit the resource use and to harden the security of the application, and by being a separate process it can run under its own user.<br />
<br />
The only part that differs from [[#php-fpm configuration]]{{Broken section link}} is the {{ic|<nowiki>location ~ \.php(?:$|/) {}</nowiki>}} block:<br />
{{bc|<nowiki><br />
location ~ \.php(?:$|/) {<br />
include uwsgi_params;<br />
uwsgi_modifier1 14;<br />
# Avoid duplicate headers confusing OC checks<br />
uwsgi_hide_header X-Frame-Options;<br />
uwsgi_hide_header X-XSS-Protection;<br />
uwsgi_hide_header X-Content-Type-Options;<br />
uwsgi_hide_header X-Robots-Tag;<br />
uwsgi_pass unix:/run/uwsgi/nextcloud.sock;<br />
}<br />
</nowiki>}}<br />
<br />
Then create a config file for ''uWSGI'':<br />
<br />
{{hc|/etc/uwsgi/nextcloud.ini|<nowiki><br />
[uwsgi]<br />
; load the required plugins<br />
plugins = php<br />
; force the sapi name to 'apache', this will enable the opcode cache <br />
php-sapi-name = apache<br />
<br />
; set master process name and socket<br />
; '%n' refers to the name of this configuration file without extension<br />
procname-master = uwsgi %n<br />
master = true<br />
socket = /run/uwsgi/%n.sock<br />
<br />
; drop privileges<br />
uid = http<br />
gid = http<br />
umask = 027<br />
<br />
; run with at least 1 process but increase up to 4 when needed<br />
processes = 4<br />
cheaper = 1<br />
<br />
; reload whenever this config file changes<br />
; %p is the full path of the current config file<br />
touch-reload = %p<br />
<br />
; disable uWSGI request logging<br />
;disable-logging = true<br />
<br />
; enforce a DOCUMENT_ROOT<br />
php-docroot = /usr/share/webapps/%n<br />
; limit allowed extensions<br />
php-allowed-ext = .php<br />
; and search for index.php if required<br />
php-index = index.php<br />
<br />
; set php configuration for this instance of php, no need to edit global php.ini<br />
php-set = date.timezone=Etc/UTC<br />
;php-set = open_basedir=/tmp/:/usr/share/webapps/nextcloud:/etc/webapps/nextcloud:/dev/urandom<br />
php-set = expose_php=false<br />
; avoid security risk of leaving sessions in world-readable /tmp<br />
php-set = session.save_path=/usr/share/webapps/nextcloud/data<br />
<br />
; port of php directives set upstream in /usr/share/webapps/nextcloud/.user.ini for use with PHP-FPM<br />
php-set = upload_max_filesize=513M<br />
php-set = post_max_size=513M<br />
php-set = memory_limit=512M<br />
php-set = output_buffering=off<br />
<br />
; load all extensions only in this instance of php, no need to edit global php.ini<br />
;; required core modules<br />
php-set = extension=gd<br />
php-set = extension=iconv<br />
;php-set = extension=zip # enabled by default in global php.ini<br />
<br />
;; database connectors<br />
;; uncomment your selected driver<br />
;php-set = extension=pdo_sqlite<br />
;php-set = extension=pdo_mysql<br />
;php-set = extension=pdo_pgsql<br />
<br />
;; recommended extensions<br />
;php-set = extension=curl # enabled by default in global php.ini<br />
php-set = extension=bz2<br />
php-set = extension=intl<br />
<br />
;; required for specific apps<br />
;php-set = extension=ldap # for LDAP integration<br />
;php-set = extension=ftp # for FTP storage / external user authentication<br />
;php-set = extension=imap # for external user authentication, requires php-imap<br />
<br />
;; recommended for specific apps<br />
;php-set = extension=exif # for image rotation in pictures app, requires exiv2<br />
;php-set = extension=gmp # for SFTP storage<br />
<br />
;; for preview generation<br />
;; provided by packages in AUR<br />
; php-set = extension=imagick<br />
<br />
; opcache<br />
php-set = zend_extension=opcache<br />
<br />
; user cache<br />
; provided by php-acpu, to be enabled **either** here **or** in /etc/php/conf.d/apcu.ini<br />
php-set = extension=apcu<br />
; per https://github.com/krakjoe/apcu/blob/simplify/INSTALL<br />
php-set = apc.ttl=7200<br />
php-set = apc.enable_cli=1<br />
<br />
cron2 = minute=-15,unique=1 /usr/bin/php -f /usr/share/webapps/nextcloud/cron.php 1>/dev/null<br />
</nowiki>}}<br />
<br />
{{Note|1=<br />
* Do not forget to set your timezone and uncomment the required database connector in the uWSGI config file<br />
* The [[PHP#Configuration|open_basedir]] directive is optional and commented out. You can uncomment to harden security. Be aware that it may [https://github.com/owncloud/core/search?q=open_basedir&type=Issues occasionally break things].<br />
* Use {{ic|1=php-docroot = /usr/share/webapps}} if placing nextcloud in /nextcloud subdirectory. }}<br />
<br />
{{Warning|The way the [https://docs.nextcloud.com/server/12/admin_manual/configuration_server/background_jobs_configuration.html Nextcloud background job] is currently set up with [https://uwsgi-docs.readthedocs.org/en/latest/Cron.html uWSGI cron] will make use of the default global configuration from {{ic|/etc/php/php.ini}}. This means that none of the specific parameters defined (e.g. required modules) will be enabled, [https://github.com/owncloud/core/issues/12678#issuecomment-66114448 leading to various issues]. One solution is to copy {{ic|/etc/php/php.ini}} to e.g. {{ic|/etc/uwsgi/cron-php.ini}}, make the required modifications there (mirroring {{ic|/etc/uwsgi/nextcloud.ini}} parameters) and referencing it in the cron directive by adding the {{ic|-c /etc/uwsgi/cron-php.ini}} option to ''php'' invocation.}}<br />
<br />
==== Activation ====<br />
<br />
[[uWSGI]] provides a [[Systemd#Using_units|template unit]] that allows to start and enable application using their configuration file name as instance identifier. For example:<br />
# systemctl start uwsgi@nextcloud.socket<br />
would start it on demand referencing the configuration file {{ic|/etc/uwsgi/nextcloud.ini}}. <br />
<br />
To enable the uwsgi service by default at start-up, run:<br />
# systemctl enable uwsgi@nextcloud.socket<br />
<br />
{{Note|Here we make use of [http://0pointer.de/blog/projects/socket-activation.html systemd socket activation] to prevent unnecessary resources consumption when no connections are made to the instance. If you would rather have it constantly active, simply remove the {{ic|.socket}} part to start and enable the service instead.}}<br />
<br />
See also [[UWSGI#Starting service]]{{Broken section link}}.<br />
<br />
=== Setting strong permissions for the filesystem ===<br />
<br />
You should set the permissions for {{ic|config/}}, {{ic|data/}} and {{ic|apps/}} as strict as possible. That means that your HTTP user (''http'' in case of {{pkg|apache}}) should own them, and the should have {{ic|700}} permissions. You can use the following script to achieve this.<br />
<br />
{{Style|Complex script for trivial one-time task.}}<br />
<br />
{{hc|oc-perms|2=<nowiki><br />
#!/bin/bash<br />
ocpath='/usr/share/webapps/nextcloud'<br />
htuser='http'<br />
htgroup='http'<br />
rootuser='root'<br />
<br />
printf "Creating possible missing Directories\n"<br />
mkdir -p $ocpath/data<br />
mkdir -p $ocpath/assets<br />
<br />
printf "chmod Files and Directories\n"<br />
find ${ocpath}/ -type f -print0 | xargs -0 chmod 0640<br />
find ${ocpath}/ -type d -print0 | xargs -0 chmod 0750<br />
<br />
printf "chown Directories\n"<br />
chown -R ${rootuser}:${htgroup} ${ocpath}/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/apps/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/assets/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/config/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/data/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/themes/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/updater/<br />
<br />
chmod +x ${ocpath}/occ<br />
<br />
printf "chmod/chown .htaccess\n"<br />
if [ -f ${ocpath}/.htaccess ]<br />
then<br />
chmod 0644 ${ocpath}/.htaccess<br />
chown ${rootuser}:${htgroup} ${ocpath}/.htaccess<br />
fi<br />
if [ -f ${ocpath}/data/.htaccess ]<br />
then<br />
chmod 0644 ${ocpath}/data/.htaccess<br />
chown ${rootuser}:${htgroup} ${ocpath}/data/.htaccess<br />
fi<br />
</nowiki>}}<br />
<br />
If you have customized your Nextcloud installation and your filepaths are different than the standard installation, then modify this script accordingly.<br />
<br />
== Synchronization ==<br />
<br />
=== Desktop ===<br />
<br />
The official client can be installed with the {{Pkg|owncloud-client}} or {{aur|nextcloud-client}} package.<br />
Alternative versions are available in the [[AUR]]: {{AUR|owncloud-client-git}}.<br />
<br />
==== Calendar ====<br />
<br />
To access your Nextcloud calendars using Mozilla [[Thunderbird]]'s Lightning calendar you would use the following URL:<br />
<br />
<nowiki>https://ADDRESS/remote.php/caldav/calendars/USERNAME/CALENDARNAME</nowiki><br />
<br />
To access your Nextcloud calendars using CalDAV-compatible programs like Kontact or [[Evolution]], you would use the following URL:<br />
<br />
<nowiki>https://ADDRESS/remote.php/caldav</nowiki><br />
<br />
For details see the [https://docs.nextcloud.com/server/12/user_manual/pim/index.html official documentation].<br />
<br />
==== Contacts ====<br />
<br />
To sync contacts with [[Thunderbird]], see [https://docs.nextcloud.com/server/12/user_manual/pim/sync_thunderbird.html these instructions] from the official doc.<br />
<br />
==== Mounting files with davfs2 ====<br />
<br />
If you want to mount your ownCloud permanently install {{Pkg|davfs2}} (as described in [[davfs2]]) first.<br />
<br />
Considering your ownCloud were at {{ic|https://own.example.com}}, your WebDAV URL would be {{ic|https://own.example.com/remote.php/webdav}} (as of ownCloud 6.0).<br />
<br />
To mount your ownCloud, use:<br />
<br />
# mount -t davfs https://own.example.com/remote.php/webdav /path/to/mount<br />
<br />
You can also create an entry for this in {{ic|/etc/fstab}}<br />
<br />
{{hc|/etc/fstab|<br />
https://own.example.com/remote.php/webdav /path/to/mount davfs rw,user,noauto 0 0<br />
}}<br />
<br />
{{Tip|In order to allow automount you can also store your username (and password if you like) in a file as described in [[davfs2#Storing credentials]].}}<br />
<br />
{{Note|If creating/copying files is not possible, while the same operations work on directories, see [[davfs2#Creating/copying files not possible and/or freezes]].}}<br />
<br />
=== Mounting files in GNOME Files (Nautilus) ===<br />
<br />
You can access the files directly in Nautilus ('+ Other Locations') through WebDAV protocol - use the link as shown in your Nextcloud installation Web GUI (typically: https://example.org/remote.php/webdav/) but replace the protocol name from 'https' to 'davs'. Nautilus will ask for user name and password when trying to connect.<br />
<br />
=== Android ===<br />
<br />
Download the official Nextcloud app from [https://play.google.com/store/apps/details?id=com.nextcloud.client Google Play].<br />
<br />
To enable contacts and calendar sync (Android 4+):<br />
# download [https://davdroid.bitfire.at/ DAVdroid] ([https://play.google.com/store/apps/details?id=at.bitfire.davdroid Play Store], [https://f-droid.org/app/at.bitfire.davdroid F-Droid])<br />
# Enable mod_rewrite.so in httpd.conf<br />
# create a new DAVdroid account in the ''Account'' settings, and specify your "short" server address and login/password couple, e.g. {{ic|<nowiki>https://cloud.example.com</nowiki>}} (there is no need for the {{ic|<nowiki>/remote.php/{carddav,webdav}</nowiki>}} part if you configured your web server with the proper redirections, as illustrated previously in the article; ''DAVdroid'' will find itself the right URLs)<br />
<br />
=== iOS ===<br />
<br />
Download the official Nextcloud app from the [https://itunes.apple.com/us/app/nextcloud/id1125420102 App Store].<br />
<br />
=== SABnzbd ===<br />
<br />
When using [[SABnzbd]], you might want to set<br />
folder_rename 0<br />
in your sabnzbd.ini file, because ownCloud will scan the files as soon as they get uploaded, preventing SABnzbd from removing UNPACKING prefixes etc.<br />
<br />
== Troubleshooting ==<br />
{{Out of date|A lot of references to OwnCloud, are these still valid with Nextcloud?}}<br />
<br />
=== Self-signed certificate not accepted ===<br />
<br />
ownCloud uses [[Wikipedia:cURL]] and [[Wikipedia:SabreDAV]] to check if WebDAV is enabled.<br />
If you use SSL/TLS with a self-signed certificate, e.g. as shown in [[LAMP]], and access ownCloud's admin panel, you will see the following error message:<br />
<br />
Your web server is not yet properly setup to allow files synchronization because the WebDAV interface seems to be broken.<br />
<br />
Assuming that you followed the [[LAMP]] tutorial, execute the following steps:<br />
<br />
Create a local directory for non-distribution certificates and copy [[LAMP]]s certificate there. This will prevent {{ic|ca-certificates}}-updates from overwriting it.<br />
<br />
# cp /etc/httpd/conf/server.crt /usr/share/ca-certificates/''WWW.EXAMPLE.COM.crt''<br />
<br />
Add ''WWW.EXAMPLE.COM.crt'' to {{ic|/etc/ca-certificates.conf}}:<br />
<br />
''WWW.EXAMPLE.COM.crt''<br />
<br />
Now, regenerate your certificate store:<br />
<br />
# update-ca-certificates<br />
<br />
Restart the httpd service to activate your certificate.<br />
<br />
=== Self-signed certificate for Android devices ===<br />
<br />
Once you have followed the setup for SSL, as on [[LAMP#TLS.2FSSL|LAMP]]{{Broken section link}} for example, early versions of DAVdroid will<br />
reject the connection because the certificate is not trusted. A certificate can be made as follows on your server:<br />
<br />
# openssl x509 -req -days 365 -in /etc/httpd/conf/server.csr -signkey /etc/httpd/conf/server.key -extfile android.txt -out CA.crt<br />
# openssl x509 -inform PEM -outform DER -in CA.crt -out CA.der.crt <br />
<br />
The file {{ic|android.txt}} should contain the following:<br />
<br />
basicConstraints=CA:true<br />
<br />
Then import {{ic|CA.der.crt}} to your Android device:<br />
<br />
Put the {{ic|CA.der.crt}} file onto the sdcard of your Android device (usually to the internal one, e.g. save from a mail attachment).<br />
It should be in the root directory. Go to ''Settings > Security > Credential storage'' and select ''Install from device storage''.<br />
The {{ic|.crt}} file will be detected and you will be prompted to enter a certificate name. After importing the certificate,<br />
you will find it in ''Settings > Security > Credential storage > Trusted credentials > User''.<br />
<br />
Thanks to: [http://www.leftbrainthings.com/2013/10/13/creating-and-importing-self-signed-certificate-to-android-device/]<br />
<br />
Another way is to import the certificate directly from your server via [https://play.google.com/store/apps/details?id=at.bitfire.cadroid CAdroid] and follow the instructions there.<br />
<br />
=== Cannot write into config directory! ===<br />
<br />
If you have set {{ic|open_basedir}} in your PHP/web server configuration file (e.g. {{ic|/etc/httpd/conf/extra/nextcloud.conf}}), make sure that it includes {{ic|/etc/webapps}}.<br />
<br />
Restart the web server to apply the change.<br />
<br />
=== Cannot create data directory ===<br />
<br />
If you have set {{ic|open_basedir}} in your PHP/web server configuration file (e.g. {{ic|/etc/httpd/conf/extra/nextcloud.conf}}), make sure that it includes the data directory.<br />
<br />
Restart the web server to apply the change.<br />
<br />
=== CSync failed to find a specific file. ===<br />
<br />
This is most likely a certificate issue. Recreate it, and do not leave the common name empty or you will see the error again.<br />
<br />
# openssl req -new -x509 -nodes -newkey rsa:4096 -keyout server.key -out server.crt<br />
<br />
=== Seeing white page after login ===<br />
<br />
The cause is probably a new app that you installed. To fix that, you can use the occ command as described<br />
[https://docs.nextcloud.com/server/12/admin_manual/configuration_server/occ_command.html here]. So with<br />
sudo -u http php /usr/share/webapps/nextcloud/occ app:list<br />
you can list all apps (if you installed nextcloud in the standard directory), and with <br />
sudo -u http php /usr/share/webapps/nextcloud/occ app:disable <nameOfExtension><br />
you can disable the troubling app.<br />
<br />
Alternatively, you can either use [[phpMyAdmin]] to edit the {{ic|oc_appconfig}} table (if you got lucky and the table has an edit option), or do it by hand with mysql:<br />
<br />
mysql -u root -p owncloud<br />
MariaDB [owncloud]> '''delete from''' oc_appconfig '''where''' appid='<nameOfExtension>' '''and''' configkey='enabled' '''and''' configvalue='yes';<br />
MariaDB [owncloud]> '''insert into''' oc_appconfig (appid,configkey,configvalue) '''values''' ('<nameOfExtension>','enabled','no');<br />
<br />
This should delete the relevant configuration from the table and add it again.<br />
<br />
=== GUI sync client fails to connect ===<br />
<br />
If using HTTP basic authentication, make sure to exclude "status.php", which must be publicly accessible. [https://github.com/owncloud/mirall/issues/734]<br />
<br />
=== Some files upload, but give an error 'Integrity constraint violation...' ===<br />
<br />
You may see the following error in the ownCloud sync client:<br />
<br />
SQLSTATE[23000]: Integrity constraint violation: ... Duplicate entry '...' for key 'fs_storage_path_hash')...<br />
<br />
This is caused by an issue with the File Locking app, which is often not sufficient to keep conflicts from occurring on some webserver configurations.<br />
A more complete [https://docs.nextcloud.com/server/12/admin_manual/configuration_files/files_locking_transactional.html Transactional File Locking]<br />
is available that rids these errors, but you must be using the Redis php-caching method. Install {{Pkg|redis}} and {{AUR|php-redis}}, comment out<br />
your current php-cache mechanism, and then in {{ic|/etc/php/conf.d/redis.ini}} uncomment {{ic|1=extension=redis}}.<br />
Then in {{ic|config.php}} make the following changes:<br />
<br />
'memcache.local' => '\OC\Memcache\Redis',<br />
'filelocking.enabled' => 'true',<br />
'memcache.locking' => '\OC\Memcache\Redis',<br />
'redis' => array(<br />
'host' => 'localhost',<br />
'port' => 6379,<br />
'timeout' => 0.0,<br />
),<br />
<br />
and [[start/enable]] {{ic|redis.service}}.<br />
<br />
Finally, disable the File Locking App, as the Transational File Locking will take care of it (and would conflict).<br />
<br />
If everything is working, you should see 'Transactional File Locking Enabled' under Server Status on the Admin page, and syncs should no longer cause issues.<br />
<br />
=== "Cannot write into apps directory" ===<br />
<br />
As mentioned in the [https://docs.nextcloud.com/server/12/admin_manual/installation/apps_management_installation.html official admin manual],<br />
either you need an apps directory that is writable by the http user, or you need to set {{ic|appstoreenabled}} to {{ic|false}}. <br />
<br />
=== Installed apps get blocked because of MIME type error ===<br />
<br />
If you're putting your apps folder outside of the nextcloud installation directory make sure your webserver serves it properly.<br />
<br />
In nginx this is accomplished by adding a location block to the nginx configuration as the folder will not be included in it by default.<br />
<br />
location ~ /apps2/(.*)$ {<br />
alias /var/www/nextcloud/apps/$1;<br />
}<br />
<br />
=== Security warnings even though the recommended settings have been included in nginx.conf ===<br />
<br />
At the top of the admin page there might be a warning to set the {{ic|Strict-Transport-Security}}, {{ic|X-Content-Type-Options}},<br />
{{ic|X-Frame-Options}}, {{ic|X-XSS-Protection}} and {{ic|X-Robots-Tag}} according to https://docs.nextcloud.com/server/12/admin_manual/configuration_server/harden_server.html<br />
even though they are already set like that.<br />
<br />
A possible cause could be that because owncloud sets those settings, uwsgi passed them along and nginx added them again:<br />
<br />
{{hc|$ curl -I https://domain.tld|<br />
<nowiki>...<br />
X-XSS-Protection: 1; mode=block<br />
X-Content-Type-Options: nosniff<br />
X-Frame-Options: Sameorigin<br />
X-Robots-Tag: none<br />
Strict-Transport-Security: max-age=15768000; includeSubDomains; preload;<br />
X-Content-Type-Options: nosniff<br />
X-Frame-Options: SAMEORIGIN<br />
X-XSS-Protection: 1; mode=block<br />
X-Robots-Tag: none</nowiki>}}<br />
<br />
While the fast_cgi sample config has a parameter to avoid that ( {{ic|fastcgi_param modHeadersAvailable true; #Avoid sending the security headers twice}} ), when using uwsgi and nginx the following modification of the uwsgi part in nginx.conf could help:<br />
<br />
{{hc| /etc/nginx/nginx.conf|<br />
<nowiki>...<br />
# pass all .php or .php/path urls to uWSGI<br />
location ~ ^(.+\.php)(.*)$ {<br />
include uwsgi_params;<br />
uwsgi_modifier1 14;<br />
# hode following headers received from uwsgi, because otherwise we would send them twice since we already add them in nginx itself<br />
uwsgi_hide_header X-Frame-Options;<br />
uwsgi_hide_header X-XSS-Protection;<br />
uwsgi_hide_header X-Content-Type-Options;<br />
uwsgi_hide_header X-Robots-Tag;<br />
uwsgi_hide_header X-Frame-Options;<br />
#Uncomment line below if you get connection refused error. Remember to commet out line with "uwsgi_pass 127.0.0.1:3001;" below<br />
uwsgi_pass unix:/run/uwsgi/owncloud.sock;<br />
#uwsgi_pass 127.0.0.1:3001;<br />
}<br />
...</nowiki>}}<br />
<br />
=== "Reading from keychain failed with error: 'No keychain service available'" ===<br />
<br />
Can be fixed for Gnome by installing the following 2 packages, {{Pkg|libgnome-keyring}} and {{Pkg|gnome-keyring}}.<br />
Or the following for KDE, {{Pkg|libgnome-keyring}} and {{Pkg|qtkeychain}}.<br />
<br />
=== FolderSync: "Method Not Allowed" ===<br />
<br />
FolderSync needs access to {{ic|/owncloud/remote.php/webdav}}, so you could create another alias for owncloud in your {{ic|/etc/httpd/conf/extra/nextcloud.conf}}<br />
<IfModule mod_alias.c><br />
Alias /nextcloud /usr/share/webapps/nextcloud/<br />
Alias /owncloud /usr/share/webapps/nextcloud/<br />
</IfModule><br />
<br />
== Tips and tricks ==<br />
<br />
=== Running ownCloud in a subdirectory ===<br />
<br />
By including the default {{ic|owncloud.conf}} in {{ic|httpd.conf}}, ownCloud will take control of port 80 and your localhost domain. <br />
<br />
If you would like to have ownCloud run in a subdirectory, then edit the {{ic|/etc/httpd/conf/extra/owncloud.conf}}<br />
you included and comment out the {{ic|<nowiki><VirtualHost *:80> ... </VirtualHost></nowiki>}} part of the include file.<br />
<br />
You can use the following nginx config when using owncloud with uwsgi:<br />
{{hc|head=/etc/nginx/conf.d/owncloud.conf|output=<nowiki><br />
location = /.well-known/carddav {<br />
return 301 $scheme://$host/owncloud/remote.php/dav;<br />
}<br />
<br />
location = /.well-known/caldav {<br />
return 301 $scheme://$host/owncloud/remote.php/dav;<br />
}<br />
<br />
location /.well-known/acme-challenge { }<br />
<br />
location ^~ /owncloud {<br />
<br />
root /usr/share/webapps;<br />
<br />
# set max upload size<br />
client_max_body_size 512M;<br />
fastcgi_buffers 64 4K;<br />
<br />
# Disable gzip to avoid the removal of the ETag header<br />
gzip off;<br />
<br />
# Uncomment if your server is build with the ngx_pagespeed module<br />
# This module is currently not supported.<br />
#pagespeed off;<br />
<br />
location /owncloud {<br />
rewrite ^ /owncloud/index.php$uri;<br />
}<br />
<br />
location ~ ^/owncloud/(?:build|tests|config|lib|3rdparty|templates|data)/ {<br />
deny all;<br />
}<br />
<br />
location ~ ^/owncloud/(?:\.|autotest|occ|issue|indie|db_|console) {<br />
deny all;<br />
}<br />
<br />
location ~ ^/owncloud/(?:updater|ocs-provider)(?:$|/) {<br />
try_files $uri/ =404;<br />
index index.php;<br />
}<br />
<br />
location ~ ^/owncloud/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|core/templates/40[34])\.php(?:$|/) {<br />
include uwsgi_params;<br />
uwsgi_modifier1 14;<br />
# Avoid duplicate headers confusing OC checks<br />
uwsgi_hide_header X-Frame-Options;<br />
uwsgi_hide_header X-XSS-Protection;<br />
uwsgi_hide_header X-Content-Type-Options;<br />
uwsgi_hide_header X-Robots-Tag;<br />
uwsgi_pass unix:/run/uwsgi/owncloud.sock;<br />
}<br />
<br />
# Adding the cache control header for js and css files<br />
# Make sure it is BELOW the PHP block<br />
location ~* \.(?:css|js) {<br />
try_files $uri /owncloud/index.php$uri$is_args$args;<br />
add_header Cache-Control "public, max-age=7200";<br />
# Add headers to serve security related headers (It is intended<br />
# to have those duplicated to the ones above)<br />
# Before enabling Strict-Transport-Security headers please read<br />
# into this topic first.<br />
# add_header Strict-Transport-Security "max-age=15768000;<br />
# includeSubDomains; preload;";<br />
add_header X-Content-Type-Options nosniff;<br />
add_header X-Frame-Options "SAMEORIGIN";<br />
add_header X-XSS-Protection "1; mode=block";<br />
add_header X-Robots-Tag none;<br />
add_header X-Download-Options noopen;<br />
add_header X-Permitted-Cross-Domain-Policies none;<br />
# Optional: Don't log access to assets<br />
access_log off;<br />
}<br />
<br />
location ~* \.(?:svg|gif|png|html|ttf|woff|ico|jpg|jpeg) {<br />
try_files $uri /owncloud/index.php$uri$is_args$args;<br />
# Optional: Don't log access to other assets<br />
access_log off;<br />
}<br />
}<br />
</nowiki>}}<br />
<br />
=== Docker ===<br />
See the [https://hub.docker.com/_/owncloud/ ownCloud] or [https://github.com/nextcloud/docker Nextcloud] repository for [[Docker]].<br />
<br />
=== Upload and share from File Manager ===<br />
[https://github.com/schiesbn/shareLinkCreator shareLinkCreator] provides the ability to upload a file to<br />
OwnCloud via a supported file manager and receive a link to the uploaded file which can then be emailed or shared in another way.<br />
<br />
=== Defining Background Jobs ===<br />
<br />
Nextcloud requires scheduled execution of some tasks, and by default it achieves this by using AJAX,<br />
however AJAX is the least reliable method, and it is recommended to use [[Cron]] instead. However,<br />
ArchLinux ships with {{Pkg|systemd}}, so the preferred way of executing scheduled tasks is a [[Systemd#Timers|systemd timer]]. <br />
<br />
First create a service:<br />
<br />
{{hc|/etc/systemd/system/nextcloudcron.service|<nowiki><br />
[Unit]<br />
Description=Nextcloud cron.php job<br />
<br />
[Service]<br />
User=http<br />
ExecStart=/usr/bin/php -f /usr/share/webapps/nextcloud/cron.php<br />
<br />
[Install]<br />
WantedBy=basic.target<br />
</nowiki>}}<br />
<br />
Then create a timer for that service:<br />
<br />
{{hc|/etc/systemd/system/nextcloudcron.timer|<nowiki><br />
[Unit]<br />
Description=Run Nextcloud cron.php every 15 minutes<br />
<br />
[Timer]<br />
OnBootSec=5min<br />
OnUnitActiveSec=15min<br />
Unit=nextcloudcron.service<br />
<br />
[Install]<br />
WantedBy=timers.target<br />
</nowiki>}}<br />
<br />
[[Start/enable]] {{ic|nextcloudcron.timer}}.<br />
<br />
Confirm that it is running by running<br />
# systemctl list-timers<br />
<br />
=== Collabora Online Office integration ===<br />
'''Solution with Docker: ''CODE backend using the official Docker image'''''<br />
<br />
The first, install a {{Pkg|docker}} package to provide collabora files and setup a Collabora server.<br />
<br />
[[Start/enable]] docker.service<br />
<br />
# systemctl enable docker.service<br />
# systemctl start docker.service<br />
<br />
Then, download the required binares :<br />
<br />
# docker pull collabora/code<br />
<br />
And, installing a Collabora server. Make sure {{ic|cloud//.example//.com}} is your nextcloud's domain, not a collabora :<br />
<br />
# docker run -t -d -p 127.0.0.1:9980:9980 -e 'domain=cloud//.example//.com' --restart always --cap-add MKNOD collabora/code<br />
<br />
Also make sure to escape all dots with double backslashes (\), since this string will be evaluated as a regular expression (and your bash 'eats' the first backslash.) If you want to use the docker container with more than one Nextcloud, you'll need to use 'domain=cloud\\.example\\.com\|second\\.example\\.com' instead. (All hosts are separated by \|.) When using `localhost` as domain for testing you need to add {ic|--net host}} to ensure the docker container can access your Nextcloud server. <br />
<br />
If you need to delete or reinstall Collabora server use:<br />
<br />
For recognition CONTAINER_ID of server<br />
# docker ps<br />
<br />
Stop and delete<br />
# docker stop CONTAINER_ID<br />
# docker rm CONTAINER_ID<br />
<br />
Futher, follow the instruction of webserver you are using:<br />
<br />
'''Nginx setup example:'''<br />
<br />
Add following to your nextcloud domain config or add new config file in /etc/nginx/conf.d/ directory, (Don't forget to change {{ic|office.example.com}} and {{ic|ssl_certificate}} to the right values:<br />
<br />
{{hc|/etc/nginx/conf.d/example.conf|<nowiki><br />
upstream office.example.com {<br />
server 127.0.0.1:9980;<br />
}<br />
<br />
server {<br />
listen 443 ssl;<br />
server_name office.example.com;<br />
<br />
ssl_certificate /etc/letsencrypt/live/office.example.com/fullchain.pem;<br />
ssl_certificate_key /etc/letsencrypt/live/office.example.com/privkey.pem;<br />
<br />
# static files<br />
location ^~ /loleaflet {<br />
proxy_pass http://127.0.0.1:9980;<br />
proxy_set_header Host $host;<br />
}<br />
<br />
# WOPI discovery URL<br />
location ^~ /hosting/discovery {<br />
proxy_pass http://127.0.0.1:9980;<br />
proxy_set_header Host $host;<br />
}<br />
<br />
# Main websocket<br />
location ~ /lool/(.*)/ws$ {<br />
proxy_pass http://127.0.0.1:9980;<br />
proxy_set_header Upgrade $http_upgrade;<br />
proxy_set_header Connection "Upgrade";<br />
proxy_set_header Host $host;<br />
proxy_read_timeout 36000s;<br />
}<br />
<br />
# Admin Console websocket<br />
location ^~ /lool/adminws {<br />
proxy_buffering off;<br />
proxy_pass http://127.0.0.1:9980;<br />
proxy_set_header Upgrade $http_upgrade;<br />
proxy_set_header Connection "Upgrade";<br />
proxy_set_header Host $host;<br />
proxy_read_timeout 36000s;<br />
}<br />
<br />
# download, presentation and image upload<br />
location ~ /lool {<br />
proxy_pass http://127.0.0.1:9980;<br />
proxy_set_header Host $host;<br />
}<br />
}<br />
</nowiki>}}<br />
<br />
Restart a nginx:<br />
<br />
# nginx -s reload<br />
or<br />
# systemctl restart nginx.service<br />
<br />
'''Apache setup example:'''<br />
<br />
Add following to nextcloud config file. Don't forget to change to the right values<br />
<br />
{{hc|/etc/httpd/conf/extra/nextcloud.conf|<nowiki><br />
<VirtualHost *:443><br />
ServerName office.nextcloud.com:443<br />
<br />
# SSL configuration, you may want to take the easy route instead and use Lets Encrypt!<br />
SSLEngine on<br />
SSLCertificateFile /path/to/signed_certificate<br />
SSLCertificateChainFile /path/to/intermediate_certificate<br />
SSLCertificateKeyFile /path/to/private/key<br />
SSLProtocol all -SSLv2 -SSLv3<br />
SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS<br />
SSLHonorCipherOrder on<br />
<br />
# Encoded slashes need to be allowed<br />
AllowEncodedSlashes NoDecode<br />
<br />
# Container uses a unique non-signed certificate<br />
SSLProxyEngine On<br />
SSLProxyVerify None<br />
SSLProxyCheckPeerCN Off<br />
SSLProxyCheckPeerName Off<br />
<br />
# keep the host<br />
ProxyPreserveHost On<br />
<br />
# static html, js, images, etc. served from loolwsd<br />
# loleaflet is the client part of LibreOffice Online<br />
ProxyPass /loleaflet https://127.0.0.1:9980/loleaflet retry=0<br />
ProxyPassReverse /loleaflet https://127.0.0.1:9980/loleaflet<br />
<br />
# WOPI discovery URL<br />
ProxyPass /hosting/discovery https://127.0.0.1:9980/hosting/discovery retry=0<br />
ProxyPassReverse /hosting/discovery https://127.0.0.1:9980/hosting/discovery<br />
<br />
# Main websocket<br />
ProxyPassMatch "/lool/(.*)/ws$" wss://127.0.0.1:9980/lool/$1/ws nocanon<br />
<br />
# Admin Console websocket<br />
ProxyPass /lool/adminws wss://127.0.0.1:9980/lool/adminws<br />
<br />
# Download as, Fullscreen presentation and Image upload operations<br />
ProxyPass /lool https://127.0.0.1:9980/lool<br />
ProxyPassReverse /lool https://127.0.0.1:9980/lool<br />
</VirtualHost><br />
</nowiki>}}<br />
<br />
After configuring these do restart your apache:<br />
# systemctl restart httpd<br />
<br />
'''Install the Nextcloud app'''<br />
<br />
Go to the Apps section and choose “Office & Text”, install the “Collabora Online” app. In admin panel select Collabora Online tab and specific the server's domain you have setup before.<br />
<br />
'''Solution without Docker: ''CODE backend using an Archlinux package'''''<br />
<br />
The {{AUR|collabora-online-server-nodocker}} package brings to your Archlinux installation 1º Collabora Office (the desktop suite), and 2º the “CODE” (Collabora Online Development Edition) server, which is based on “lool” (LibreOffice OnLine).<br />
<br />
Alter the `/etc/loolwsd/loolwsd.xml` file, so that:<br />
* `config > server_name` contains the host and port of the public Nextcloud address, separated by a colon (eg. `example.org:443`),<br />
* `config > ssl > enable` is false (ie. web browser —HTTPS→ proxy —HTTP→ loolwsd),<br />
* `config > ssl > termination` is true (I suppose you’ll manage TLS at the proxy level),<br />
* `config > storage > wopi > host` reflects the actual hostname (or pattern) of the proxy server (eg. `(?:.*\.)?example\.org`),<br />
* `config > admin_console > username` and `config > admin_console > password` are set to values of your choice.<br />
<br />
Then:<br />
* start and enable `loolwsd.service`;<br />
* configure Nginx as showed in /usr/share/doc/loolwsd/example.nginx.conf, and restart it.<br />
<br />
== See also ==<br />
* [https://docs.nextcloud.com/ nextcloud official website]<br />
* [https://docs.nextcloud.com/server/12/admin_manual/ nextcloud 12.0 Admin Documentation]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Nextcloud&diff=508573Nextcloud2018-01-26T18:55:59Z<p>NeoTheFox: /* Desktop */ Added flatpak nextcloud-client</p>
<hr />
<div>[[Category:File sharing]]<br />
[[Category:Web applications]]<br />
[[fr:Owncloud]]<br />
[[ja:Nextcloud]]<br />
{{Related articles start}}<br />
{{Related|Apache HTTP Server}}<br />
{{Related|Nginx}}<br />
{{Related|OpenSSL}}<br />
{{Related|WebDAV}}<br />
{{Related articles end}}<br />
<br />
From [[Wikipedia:Nextcloud|Wikipedia]]:<br />
<br />
:Nextcloud is a suite of client-server software for creating and using file hosting services. It is functionally similar to Dropbox, although Nextcloud is free and open-source, allowing anyone to install and operate it on a private server. In contrast to proprietary services like Dropbox, the open architecture allows adding additional functionality to the server in form of applications.<br />
<br />
Nextcloud is a fork of ownCloud. For differences between the two, see [[wikipedia:Nextcloud#Differences from ownCloud]].<br />
<br />
== Prerequisites ==<br />
Nextcloud requires several components:<br />
* A web server: [[Apache]] or [[nginx]]<br />
* A database: [[MariaDB]] or [[PostgreSQL]]<br />
* [[PHP]] with [[#PHP_setup|additional modules]].<br />
<br />
These will be configured in [[#Setup]].<br />
<br />
Make sure the required components are installed before proceeding.<br />
<br />
== Installation ==<br />
<br />
[[Install]] the {{Pkg|nextcloud}} package.<br />
<br />
== Setup ==<br />
<br />
As stated above, in order to setup Nextcloud, you must set up the appropriate PHP requirements;<br />
additionally, you must configure a database and a webserver.<br />
<br />
=== Pacman hook ===<br />
<br />
To do nextcloud database upgrade automatically you may set up pacman post upgrade hook based on following example:<br />
<br />
# Update Nextcloud when core or -apps are touched<br />
<br />
[Trigger]<br />
Operation = Install<br />
Operation = Upgrade<br />
Type = Package<br />
Target = nextcloud<br />
Target = nextcloud-app-*<br />
<br />
[Action]<br />
Description = Updating Nextcloud installation<br />
When = PostTransaction<br />
Exec = /usr/bin/runuser -u http -- /usr/bin/php /usr/share/webapps/nextcloud/occ upgrade<br />
<br />
You need to put it into /etc/pacman.d/hooks/nextcloud.hook if you did not customize HookDir in pacman.conf.<br />
<br />
See also [[Pacman#Hooks]]<br />
<br />
=== PHP setup ===<br />
<br />
{{Note|Nextcloud does not support PHP 7.2 as of version 12.0.4. [https://github.com/nextcloud/server/issues/6786] Until version 13 is released, [[install]] the {{pkg|php71}} package or update to {{AUR|nextcloud-testing}}.}}<br />
<br />
{{Tip|For all prerequisite PHP modules, see upstream documentation: [https://docs.nextcloud.com/server/12/admin_manual/installation/source_installation.html#prerequisites-label Nextcloud 12.0].}}<br />
<br />
Install [[PHP#gd]] and {{pkg|php-intl}} as additional modules.<br />
<br />
Some apps(News for example) require the iconv extension, if you wish to use these apps, uncomment the extension in {{ic|/etc/php/php.ini}}.<br />
<br />
Depending on which database backend will be used:<br />
* For [[MySQL]], see [[PHP#MySQL/MariaDB]].<br />
* For [[PostgreSQL]], see [[PHP#PostgreSQL]].<br />
* For [[SQLite]], see [[PHP#Sqlite]].<br />
<br />
Performance may be improved through the implementation of [[PHP#Caching|caching]], see<br />
[https://docs.nextcloud.com/server/12/admin_manual/configuration_server/caching_configuration.html Configuring Memory Caching]<br />
on the official documentation for details.<br />
<br />
=== Database setup ===<br />
<br />
An SQL database must be setup and used for your Nextcloud installation. After setting up the database here,<br />
you will be prompted for its information when you first create an administrator account.<br />
<br />
==== MariaDB ====<br />
<br />
{{Note|It's is highly recommended to set {{ic|binlog_format}} to ''mixed'' [https://docs.nextcloud.com/server/12/admin_manual/configuration_database/linux_database_configuration.html#db-binlog-label] in {{ic|/etc/mysql/my.cnf}}.}}<br />
<br />
The following is an example of setting up a [[MariaDB]] database and user:<br />
<br />
{{hc|$ mysql -u root -p|2=<br />
mysql> CREATE DATABASE `'''nextcloud'''` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`;<br />
mysql> CREATE USER `'''nextcloud'''`@'localhost' IDENTIFIED BY ''''password'''';<br />
mysql> GRANT ALL PRIVILEGES ON `'''nextcloud'''`.* TO `'''nextcloud'''`@`localhost`;<br />
mysql> \q<br />
}}<br />
<br />
==== PostgreSQL ====<br />
{{Warning|1=Nextcloud 12 is unable to do migration with PostgreSQL version 10 (currently in the repositories), as noted in this [https://bugs.archlinux.org/task/56302?project=5&string=nextcloud bug report].<br />
While not officially supported, once the migration done, Nextcloud 12 seems to work as expected with PostgreSQL 10.}}<br />
<br />
The following is an example of setting up a [[PostgreSQL]] user and database:<br />
{{hc|1=$ sudo -u postgres createuser -h localhost -P nextcloud|2=<br />
Enter password for new role:<br />
Enter it again:<br />
}}<br />
<br />
$ sudo -u postgres createdb -O nextcloud nextcloud<br />
<br />
=== Webserver setup ===<br />
<br />
{{Warning|It is recommended to use TLS/SSL (HTTPS) over plain HTTP, see [[Apache#TLS/SSL]] or [[Nginx#TLS/SSL]] for examples and implement this in the examples given below.}}<br />
<br />
Depending on which [[webserver]] you are using, further setup is required, indicated below.<br />
<br />
==== Apache ====<br />
<br />
If you haven't already, install [[Apache]] and install and enable [[Apache#PHP|Apache's PHP module]]<br />
<br />
Copy the Apache configuration file to the configuration directory:<br />
<br />
# cp /etc/webapps/nextcloud/apache.example.conf /etc/httpd/conf/extra/nextcloud.conf<br />
<br />
Modify the file according to your preferences. By default it includes an alias<br />
for {{ic|/nextcloud}} pointing to {{ic|/usr/share/webapps/nextcloud}}.<br />
<br />
And include it in {{ic|/etc/httpd/conf/httpd.conf}}:<br />
Include conf/extra/nextcloud.conf<br />
<br />
Ensure that the root location of your Nextcloud installation (e.g., {{ic|/usr/share/webapps/nextcloud}}) is accessible by the webserver's user {{ic|http}}.<br />
<br />
Now restart Apache ({{ic|httpd.service}}).<br />
<br />
===== WebDAV =====<br />
Nextcloud comes with its own [[WebDAV]] implementation enabled, which may conflict with the one shipped with Apache. If you have enabled WebDAV in Apache (not enabled by default), disable the modules {{ic|mod_dav}} and {{ic|mod_dav_fs}} in {{ic|/etc/httpd/conf/httpd.conf}}. See [https://forum.owncloud.org/viewtopic.php?f=17&t=7240] for details.<br />
<br />
==== Nginx ====<br />
Create an empty directory to hold the cloud-specific config file:<br />
# mkdir /etc/nginx/conf.d/<br />
<br />
In {{ic|/etc/nginx/nginx.conf}}, add the following lines under the "http" section:<br />
<br />
http {<br />
...<br />
...<br />
server_names_hash_bucket_size 64;<br />
include conf.d/*.conf;<br />
}<br />
<br />
<br />
Create a config file {{ic|/etc/nginx/conf.d/nextcloud.conf}} according to the [https://docs.nextcloud.com/server/12/admin_manual/installation/nginx.html documentation]. You will have to change the {{ic|root}} location, as the Arch package installs to {{ic|/usr/share/webapps/nextcloud}} instead of {{ic|/var/www/nextcloud}}.<br />
<br />
Addtitionally, you change the php-handler block so it looks like this one <br />
<br />
upstream php-handler {<br />
server unix:/run/php-fpm/php-fpm.sock;<br />
}<br />
<br />
in the {{ic|/etc/nginx/conf.d/nextcloud.conf}} file.<br />
<br />
From this point on, it is recommended to obtain a secure-certificates using [[Let's Encrypt]], see [[#Security Hardening]].<br />
<br />
===== PHP-FPM configuration =====<br />
<br />
Make sure PHP-FPM has been configured correctly as described in [[Nginx#FastCGI]].<br />
<br />
Uncomment {{ic|1=env[PATH] = /usr/local/bin:/usr/bin:/bin}} in {{ic|/etc/php/php-fpm.d/www.conf}} and [[restart]] {{ic|php-fpm.service}} to apply the changes.<br />
<br />
== Initialize ==<br />
<br />
Open the address where you have installed Nextcloud in a web browser (e.g., https://www.examples.com/nextcloud).<br />
<br />
=== Create storage directories ===<br />
<br />
You will now see "Cannot write into "apps" directory". This is because Arch packages Nextcloud in a way where the ''apps'' folder only has the webserver as a group without web permissions and the ''data'' folder is nonexistent.<br />
<br />
The easiest non-conflicting way is to create a new writable folder for apps and also create a writable data folder. Replace the http group with the group your webserver uses if needed.<br />
<br />
# mkdir -p /usr/share/webapps/nextcloud/data<br />
# mkdir -p /usr/share/webapps/nextcloud/apps2<br />
# chown http:http /usr/share/webapps/nextcloud/data<br />
# chown http:http /usr/share/webapps/nextcloud/apps2<br />
# chmod 700 /usr/share/webapps/nextcloud/data<br />
# chmod 700 /usr/share/webapps/nextcloud/apps2<br />
<br />
Next edit the configuration file at {{ic|/etc/webapps/nextcloud/config/config.php}} and add following lines before the closing {{ic|);}} of the file.<br />
<br />
{{hc|/etc/webapps/nextcloud/config/config.php|2=<br />
'apps_paths' =><br />
array (<br />
0 =><br />
array (<br />
'path' => '/usr/share/webapps/nextcloud/apps',<br />
'url' => '/apps',<br />
'writable' => false,<br />
),<br />
1 =><br />
array (<br />
'path' => '/usr/share/webapps/nextcloud/apps2',<br />
'url' => '/apps2',<br />
'writable' => true,<br />
),<br />
),<br />
'datadirectory' => '/usr/share/webapps/nextcloud/data'<br />
}}<br />
<br />
Refresh the page and the error should be gone.<br />
<br />
From there follow the instructions in adding an administrator account as well as selecting the database you created earlier.<br />
<br />
== Security Hardening ==<br />
<br />
The [https://docs.nextcloud.com/server/12/admin_manual/configuration_server/harden_server.html Nextcloud Hardening and Security]<br />
article guides along generic topics. See also the project's [https://scan.nextcloud.com/ Security scanner].<br />
<br />
=== Let's Encrypt ===<br />
==== nginx ====<br />
{{Move|Let's Encrypt|A simple example of configuring an 'empty' domain for Let's Encrypt should be created. The example file given is too big and could be far easier.}}<br />
1. Create the cloud configuration {{ic|/etc/nginx/conf.d/cloud-initial.conf}} using [https://github.com/graysky2/configs/blob/master/nginx/nextcloud-initial.conf this initial file] as a template. Substitute the literal "@@FQDN@@" in the template file with the actual [[wikipedia:Fully_qualified_domain_name|FQDN]] to be used. The certs for the server need to be generated using this unencrypted configuration initially. Follow the steps outlined on [[Let’s Encrypt]] to generate the server encryption certificates.<br />
<br />
2. Upon successfully generating certificates, replace {{ic|/etc/nginx/conf.d/cloud-initial.conf}} (it may be safely renamed so long as it does not end in ".conf" or simply deleted) with a new file, {{ic|/etc/nginx/conf.d/cloud.conf}} using [https://github.com/graysky2/configs/blob/master/nginx/nextcloud.conf this file] as a template. Again, substitute the literal "@@FQDN@@" in the template file with the actual [[wikipedia:Fully_qualified_domain_name|FQDN]] to be used. [[Start]] and optionally [[enable]] {{ic|nginx.service}}.<br />
<br />
=== uWSGI ===<br />
<br />
You can run Nextcloud in its own process and service by using the [[uWSGI]] application server with {{pkg|uwsgi-plugin-php}}. This allows you to define a [[PHP#Configuration|PHP configuration]] only for this instance of PHP, without the need to edit the global {{ic|php.ini}} and thus keeping your web application configurations compartmentalized. ''uWSGI'' itself has a wealth of features to limit the resource use and to harden the security of the application, and by being a separate process it can run under its own user.<br />
<br />
The only part that differs from [[#php-fpm configuration]]{{Broken section link}} is the {{ic|<nowiki>location ~ \.php(?:$|/) {}</nowiki>}} block:<br />
{{bc|<nowiki><br />
location ~ \.php(?:$|/) {<br />
include uwsgi_params;<br />
uwsgi_modifier1 14;<br />
# Avoid duplicate headers confusing OC checks<br />
uwsgi_hide_header X-Frame-Options;<br />
uwsgi_hide_header X-XSS-Protection;<br />
uwsgi_hide_header X-Content-Type-Options;<br />
uwsgi_hide_header X-Robots-Tag;<br />
uwsgi_pass unix:/run/uwsgi/nextcloud.sock;<br />
}<br />
</nowiki>}}<br />
<br />
Then create a config file for ''uWSGI'':<br />
<br />
{{hc|/etc/uwsgi/nextcloud.ini|<nowiki><br />
[uwsgi]<br />
; load the required plugins<br />
plugins = php<br />
; force the sapi name to 'apache', this will enable the opcode cache <br />
php-sapi-name = apache<br />
<br />
; set master process name and socket<br />
; '%n' refers to the name of this configuration file without extension<br />
procname-master = uwsgi %n<br />
master = true<br />
socket = /run/uwsgi/%n.sock<br />
<br />
; drop privileges<br />
uid = http<br />
gid = http<br />
umask = 027<br />
<br />
; run with at least 1 process but increase up to 4 when needed<br />
processes = 4<br />
cheaper = 1<br />
<br />
; reload whenever this config file changes<br />
; %p is the full path of the current config file<br />
touch-reload = %p<br />
<br />
; disable uWSGI request logging<br />
;disable-logging = true<br />
<br />
; enforce a DOCUMENT_ROOT<br />
php-docroot = /usr/share/webapps/%n<br />
; limit allowed extensions<br />
php-allowed-ext = .php<br />
; and search for index.php if required<br />
php-index = index.php<br />
<br />
; set php configuration for this instance of php, no need to edit global php.ini<br />
php-set = date.timezone=Etc/UTC<br />
;php-set = open_basedir=/tmp/:/usr/share/webapps/nextcloud:/etc/webapps/nextcloud:/dev/urandom<br />
php-set = expose_php=false<br />
; avoid security risk of leaving sessions in world-readable /tmp<br />
php-set = session.save_path=/usr/share/webapps/nextcloud/data<br />
<br />
; port of php directives set upstream in /usr/share/webapps/nextcloud/.user.ini for use with PHP-FPM<br />
php-set = upload_max_filesize=513M<br />
php-set = post_max_size=513M<br />
php-set = memory_limit=512M<br />
php-set = output_buffering=off<br />
<br />
; load all extensions only in this instance of php, no need to edit global php.ini<br />
;; required core modules<br />
php-set = extension=gd<br />
php-set = extension=iconv<br />
;php-set = extension=zip # enabled by default in global php.ini<br />
<br />
;; database connectors<br />
;; uncomment your selected driver<br />
;php-set = extension=pdo_sqlite<br />
;php-set = extension=pdo_mysql<br />
;php-set = extension=pdo_pgsql<br />
<br />
;; recommended extensions<br />
;php-set = extension=curl # enabled by default in global php.ini<br />
php-set = extension=bz2<br />
php-set = extension=intl<br />
<br />
;; required for specific apps<br />
;php-set = extension=ldap # for LDAP integration<br />
;php-set = extension=ftp # for FTP storage / external user authentication<br />
;php-set = extension=imap # for external user authentication, requires php-imap<br />
<br />
;; recommended for specific apps<br />
;php-set = extension=exif # for image rotation in pictures app, requires exiv2<br />
;php-set = extension=gmp # for SFTP storage<br />
<br />
;; for preview generation<br />
;; provided by packages in AUR<br />
; php-set = extension=imagick<br />
<br />
; opcache<br />
php-set = zend_extension=opcache<br />
<br />
; user cache<br />
; provided by php-acpu, to be enabled **either** here **or** in /etc/php/conf.d/apcu.ini<br />
php-set = extension=apcu<br />
; per https://github.com/krakjoe/apcu/blob/simplify/INSTALL<br />
php-set = apc.ttl=7200<br />
php-set = apc.enable_cli=1<br />
<br />
cron2 = minute=-15,unique=1 /usr/bin/php -f /usr/share/webapps/nextcloud/cron.php 1>/dev/null<br />
</nowiki>}}<br />
<br />
{{Note|1=<br />
* Do not forget to set your timezone and uncomment the required database connector in the uWSGI config file<br />
* The [[PHP#Configuration|open_basedir]] directive is optional and commented out. You can uncomment to harden security. Be aware that it may [https://github.com/owncloud/core/search?q=open_basedir&type=Issues occasionally break things].<br />
* Use {{ic|1=php-docroot = /usr/share/webapps}} if placing nextcloud in /nextcloud subdirectory. }}<br />
<br />
{{Warning|The way the [https://docs.nextcloud.com/server/12/admin_manual/configuration_server/background_jobs_configuration.html Nextcloud background job] is currently set up with [https://uwsgi-docs.readthedocs.org/en/latest/Cron.html uWSGI cron] will make use of the default global configuration from {{ic|/etc/php/php.ini}}. This means that none of the specific parameters defined (e.g. required modules) will be enabled, [https://github.com/owncloud/core/issues/12678#issuecomment-66114448 leading to various issues]. One solution is to copy {{ic|/etc/php/php.ini}} to e.g. {{ic|/etc/uwsgi/cron-php.ini}}, make the required modifications there (mirroring {{ic|/etc/uwsgi/nextcloud.ini}} parameters) and referencing it in the cron directive by adding the {{ic|-c /etc/uwsgi/cron-php.ini}} option to ''php'' invocation.}}<br />
<br />
==== Activation ====<br />
<br />
[[uWSGI]] provides a [[Systemd#Using_units|template unit]] that allows to start and enable application using their configuration file name as instance identifier. For example:<br />
# systemctl start uwsgi@nextcloud.socket<br />
would start it on demand referencing the configuration file {{ic|/etc/uwsgi/nextcloud.ini}}. <br />
<br />
To enable the uwsgi service by default at start-up, run:<br />
# systemctl enable uwsgi@nextcloud.socket<br />
<br />
{{Note|Here we make use of [http://0pointer.de/blog/projects/socket-activation.html systemd socket activation] to prevent unnecessary resources consumption when no connections are made to the instance. If you would rather have it constantly active, simply remove the {{ic|.socket}} part to start and enable the service instead.}}<br />
<br />
See also [[UWSGI#Starting service]]{{Broken section link}}.<br />
<br />
=== Setting strong permissions for the filesystem ===<br />
<br />
You should set the permissions for {{ic|config/}}, {{ic|data/}} and {{ic|apps/}} as strict as possible. That means that your HTTP user (''http'' in case of {{pkg|apache}}) should own them, and the should have {{ic|700}} permissions. You can use the following script to achieve this.<br />
<br />
{{Style|Complex script for trivial one-time task.}}<br />
<br />
{{hc|oc-perms|2=<nowiki><br />
#!/bin/bash<br />
ocpath='/usr/share/webapps/nextcloud'<br />
htuser='http'<br />
htgroup='http'<br />
rootuser='root'<br />
<br />
printf "Creating possible missing Directories\n"<br />
mkdir -p $ocpath/data<br />
mkdir -p $ocpath/assets<br />
<br />
printf "chmod Files and Directories\n"<br />
find ${ocpath}/ -type f -print0 | xargs -0 chmod 0640<br />
find ${ocpath}/ -type d -print0 | xargs -0 chmod 0750<br />
<br />
printf "chown Directories\n"<br />
chown -R ${rootuser}:${htgroup} ${ocpath}/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/apps/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/assets/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/config/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/data/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/themes/<br />
chown -R ${htuser}:${htgroup} ${ocpath}/updater/<br />
<br />
chmod +x ${ocpath}/occ<br />
<br />
printf "chmod/chown .htaccess\n"<br />
if [ -f ${ocpath}/.htaccess ]<br />
then<br />
chmod 0644 ${ocpath}/.htaccess<br />
chown ${rootuser}:${htgroup} ${ocpath}/.htaccess<br />
fi<br />
if [ -f ${ocpath}/data/.htaccess ]<br />
then<br />
chmod 0644 ${ocpath}/data/.htaccess<br />
chown ${rootuser}:${htgroup} ${ocpath}/data/.htaccess<br />
fi<br />
</nowiki>}}<br />
<br />
If you have customized your Nextcloud installation and your filepaths are different than the standard installation, then modify this script accordingly.<br />
<br />
== Synchronization ==<br />
<br />
=== Desktop ===<br />
<br />
The official client can be installed with the {{Pkg|owncloud-client}}, {{aur|nextcloud-client}} or {{Flatpak|org.nextcloud.Nextcloud|Nextcloud}} package.<br />
Alternative versions are available in the [[AUR]]: {{AUR|owncloud-client-git}}.<br />
<br />
==== Calendar ====<br />
<br />
To access your Nextcloud calendars using Mozilla [[Thunderbird]]'s Lightning calendar you would use the following URL:<br />
<br />
<nowiki>https://ADDRESS/remote.php/caldav/calendars/USERNAME/CALENDARNAME</nowiki><br />
<br />
To access your Nextcloud calendars using CalDAV-compatible programs like Kontact or [[Evolution]], you would use the following URL:<br />
<br />
<nowiki>https://ADDRESS/remote.php/caldav</nowiki><br />
<br />
For details see the [https://docs.nextcloud.com/server/12/user_manual/pim/index.html official documentation].<br />
<br />
==== Contacts ====<br />
<br />
To sync contacts with [[Thunderbird]], see [https://docs.nextcloud.com/server/12/user_manual/pim/sync_thunderbird.html these instructions] from the official doc.<br />
<br />
==== Mounting files with davfs2 ====<br />
<br />
If you want to mount your ownCloud permanently install {{Pkg|davfs2}} (as described in [[davfs2]]) first.<br />
<br />
Considering your ownCloud were at {{ic|https://own.example.com}}, your WebDAV URL would be {{ic|https://own.example.com/remote.php/webdav}} (as of ownCloud 6.0).<br />
<br />
To mount your ownCloud, use:<br />
<br />
# mount -t davfs https://own.example.com/remote.php/webdav /path/to/mount<br />
<br />
You can also create an entry for this in {{ic|/etc/fstab}}<br />
<br />
{{hc|/etc/fstab|<br />
https://own.example.com/remote.php/webdav /path/to/mount davfs rw,user,noauto 0 0<br />
}}<br />
<br />
{{Tip|In order to allow automount you can also store your username (and password if you like) in a file as described in [[davfs2#Storing credentials]].}}<br />
<br />
{{Note|If creating/copying files is not possible, while the same operations work on directories, see [[davfs2#Creating/copying files not possible and/or freezes]].}}<br />
<br />
=== Mounting files in GNOME Files (Nautilus) ===<br />
<br />
You can access the files directly in Nautilus ('+ Other Locations') through WebDAV protocol - use the link as shown in your Nextcloud installation Web GUI (typically: https://example.org/remote.php/webdav/) but replace the protocol name from 'https' to 'davs'. Nautilus will ask for user name and password when trying to connect.<br />
<br />
=== Android ===<br />
<br />
Download the official Nextcloud app from [https://play.google.com/store/apps/details?id=com.nextcloud.client Google Play].<br />
<br />
To enable contacts and calendar sync (Android 4+):<br />
# download [https://davdroid.bitfire.at/ DAVdroid] ([https://play.google.com/store/apps/details?id=at.bitfire.davdroid Play Store], [https://f-droid.org/app/at.bitfire.davdroid F-Droid])<br />
# Enable mod_rewrite.so in httpd.conf<br />
# create a new DAVdroid account in the ''Account'' settings, and specify your "short" server address and login/password couple, e.g. {{ic|<nowiki>https://cloud.example.com</nowiki>}} (there is no need for the {{ic|<nowiki>/remote.php/{carddav,webdav}</nowiki>}} part if you configured your web server with the proper redirections, as illustrated previously in the article; ''DAVdroid'' will find itself the right URLs)<br />
<br />
=== iOS ===<br />
<br />
Download the official Nextcloud app from the [https://itunes.apple.com/us/app/nextcloud/id1125420102 App Store].<br />
<br />
=== SABnzbd ===<br />
<br />
When using [[SABnzbd]], you might want to set<br />
folder_rename 0<br />
in your sabnzbd.ini file, because ownCloud will scan the files as soon as they get uploaded, preventing SABnzbd from removing UNPACKING prefixes etc.<br />
<br />
== Troubleshooting ==<br />
{{Out of date|A lot of references to OwnCloud, are these still valid with Nextcloud?}}<br />
<br />
=== Self-signed certificate not accepted ===<br />
<br />
ownCloud uses [[Wikipedia:cURL]] and [[Wikipedia:SabreDAV]] to check if WebDAV is enabled.<br />
If you use SSL/TLS with a self-signed certificate, e.g. as shown in [[LAMP]], and access ownCloud's admin panel, you will see the following error message:<br />
<br />
Your web server is not yet properly setup to allow files synchronization because the WebDAV interface seems to be broken.<br />
<br />
Assuming that you followed the [[LAMP]] tutorial, execute the following steps:<br />
<br />
Create a local directory for non-distribution certificates and copy [[LAMP]]s certificate there. This will prevent {{ic|ca-certificates}}-updates from overwriting it.<br />
<br />
# cp /etc/httpd/conf/server.crt /usr/share/ca-certificates/''WWW.EXAMPLE.COM.crt''<br />
<br />
Add ''WWW.EXAMPLE.COM.crt'' to {{ic|/etc/ca-certificates.conf}}:<br />
<br />
''WWW.EXAMPLE.COM.crt''<br />
<br />
Now, regenerate your certificate store:<br />
<br />
# update-ca-certificates<br />
<br />
Restart the httpd service to activate your certificate.<br />
<br />
=== Self-signed certificate for Android devices ===<br />
<br />
Once you have followed the setup for SSL, as on [[LAMP#TLS.2FSSL|LAMP]]{{Broken section link}} for example, early versions of DAVdroid will<br />
reject the connection because the certificate is not trusted. A certificate can be made as follows on your server:<br />
<br />
# openssl x509 -req -days 365 -in /etc/httpd/conf/server.csr -signkey /etc/httpd/conf/server.key -extfile android.txt -out CA.crt<br />
# openssl x509 -inform PEM -outform DER -in CA.crt -out CA.der.crt <br />
<br />
The file {{ic|android.txt}} should contain the following:<br />
<br />
basicConstraints=CA:true<br />
<br />
Then import {{ic|CA.der.crt}} to your Android device:<br />
<br />
Put the {{ic|CA.der.crt}} file onto the sdcard of your Android device (usually to the internal one, e.g. save from a mail attachment).<br />
It should be in the root directory. Go to ''Settings > Security > Credential storage'' and select ''Install from device storage''.<br />
The {{ic|.crt}} file will be detected and you will be prompted to enter a certificate name. After importing the certificate,<br />
you will find it in ''Settings > Security > Credential storage > Trusted credentials > User''.<br />
<br />
Thanks to: [http://www.leftbrainthings.com/2013/10/13/creating-and-importing-self-signed-certificate-to-android-device/]<br />
<br />
Another way is to import the certificate directly from your server via [https://play.google.com/store/apps/details?id=at.bitfire.cadroid CAdroid] and follow the instructions there.<br />
<br />
=== Cannot write into config directory! ===<br />
<br />
If you have set {{ic|open_basedir}} in your PHP/web server configuration file (e.g. {{ic|/etc/httpd/conf/extra/nextcloud.conf}}), make sure that it includes {{ic|/etc/webapps}}.<br />
<br />
Restart the web server to apply the change.<br />
<br />
=== Cannot create data directory ===<br />
<br />
If you have set {{ic|open_basedir}} in your PHP/web server configuration file (e.g. {{ic|/etc/httpd/conf/extra/nextcloud.conf}}), make sure that it includes the data directory.<br />
<br />
Restart the web server to apply the change.<br />
<br />
=== CSync failed to find a specific file. ===<br />
<br />
This is most likely a certificate issue. Recreate it, and do not leave the common name empty or you will see the error again.<br />
<br />
# openssl req -new -x509 -nodes -newkey rsa:4096 -keyout server.key -out server.crt<br />
<br />
=== Seeing white page after login ===<br />
<br />
The cause is probably a new app that you installed. To fix that, you can use the occ command as described<br />
[https://docs.nextcloud.com/server/12/admin_manual/configuration_server/occ_command.html here]. So with<br />
sudo -u http php /usr/share/webapps/nextcloud/occ app:list<br />
you can list all apps (if you installed nextcloud in the standard directory), and with <br />
sudo -u http php /usr/share/webapps/nextcloud/occ app:disable <nameOfExtension><br />
you can disable the troubling app.<br />
<br />
Alternatively, you can either use [[phpMyAdmin]] to edit the {{ic|oc_appconfig}} table (if you got lucky and the table has an edit option), or do it by hand with mysql:<br />
<br />
mysql -u root -p owncloud<br />
MariaDB [owncloud]> '''delete from''' oc_appconfig '''where''' appid='<nameOfExtension>' '''and''' configkey='enabled' '''and''' configvalue='yes';<br />
MariaDB [owncloud]> '''insert into''' oc_appconfig (appid,configkey,configvalue) '''values''' ('<nameOfExtension>','enabled','no');<br />
<br />
This should delete the relevant configuration from the table and add it again.<br />
<br />
=== GUI sync client fails to connect ===<br />
<br />
If using HTTP basic authentication, make sure to exclude "status.php", which must be publicly accessible. [https://github.com/owncloud/mirall/issues/734]<br />
<br />
=== Some files upload, but give an error 'Integrity constraint violation...' ===<br />
<br />
You may see the following error in the ownCloud sync client:<br />
<br />
SQLSTATE[23000]: Integrity constraint violation: ... Duplicate entry '...' for key 'fs_storage_path_hash')...<br />
<br />
This is caused by an issue with the File Locking app, which is often not sufficient to keep conflicts from occurring on some webserver configurations.<br />
A more complete [https://docs.nextcloud.com/server/12/admin_manual/configuration_files/files_locking_transactional.html Transactional File Locking]<br />
is available that rids these errors, but you must be using the Redis php-caching method. Install {{Pkg|redis}} and {{AUR|php-redis}}, comment out<br />
your current php-cache mechanism, and then in {{ic|/etc/php/conf.d/redis.ini}} uncomment {{ic|1=extension=redis}}.<br />
Then in {{ic|config.php}} make the following changes:<br />
<br />
'memcache.local' => '\OC\Memcache\Redis',<br />
'filelocking.enabled' => 'true',<br />
'memcache.locking' => '\OC\Memcache\Redis',<br />
'redis' => array(<br />
'host' => 'localhost',<br />
'port' => 6379,<br />
'timeout' => 0.0,<br />
),<br />
<br />
and [[start/enable]] {{ic|redis.service}}.<br />
<br />
Finally, disable the File Locking App, as the Transational File Locking will take care of it (and would conflict).<br />
<br />
If everything is working, you should see 'Transactional File Locking Enabled' under Server Status on the Admin page, and syncs should no longer cause issues.<br />
<br />
=== "Cannot write into apps directory" ===<br />
<br />
As mentioned in the [https://docs.nextcloud.com/server/12/admin_manual/installation/apps_management_installation.html official admin manual],<br />
either you need an apps directory that is writable by the http user, or you need to set {{ic|appstoreenabled}} to {{ic|false}}. <br />
<br />
=== Installed apps get blocked because of MIME type error ===<br />
<br />
If you're putting your apps folder outside of the nextcloud installation directory make sure your webserver serves it properly.<br />
<br />
In nginx this is accomplished by adding a location block to the nginx configuration as the folder will not be included in it by default.<br />
<br />
location ~ /apps2/(.*)$ {<br />
alias /var/www/nextcloud/apps/$1;<br />
}<br />
<br />
=== Security warnings even though the recommended settings have been included in nginx.conf ===<br />
<br />
At the top of the admin page there might be a warning to set the {{ic|Strict-Transport-Security}}, {{ic|X-Content-Type-Options}},<br />
{{ic|X-Frame-Options}}, {{ic|X-XSS-Protection}} and {{ic|X-Robots-Tag}} according to https://docs.nextcloud.com/server/12/admin_manual/configuration_server/harden_server.html<br />
even though they are already set like that.<br />
<br />
A possible cause could be that because owncloud sets those settings, uwsgi passed them along and nginx added them again:<br />
<br />
{{hc|$ curl -I https://domain.tld|<br />
<nowiki>...<br />
X-XSS-Protection: 1; mode=block<br />
X-Content-Type-Options: nosniff<br />
X-Frame-Options: Sameorigin<br />
X-Robots-Tag: none<br />
Strict-Transport-Security: max-age=15768000; includeSubDomains; preload;<br />
X-Content-Type-Options: nosniff<br />
X-Frame-Options: SAMEORIGIN<br />
X-XSS-Protection: 1; mode=block<br />
X-Robots-Tag: none</nowiki>}}<br />
<br />
While the fast_cgi sample config has a parameter to avoid that ( {{ic|fastcgi_param modHeadersAvailable true; #Avoid sending the security headers twice}} ), when using uwsgi and nginx the following modification of the uwsgi part in nginx.conf could help:<br />
<br />
{{hc| /etc/nginx/nginx.conf|<br />
<nowiki>...<br />
# pass all .php or .php/path urls to uWSGI<br />
location ~ ^(.+\.php)(.*)$ {<br />
include uwsgi_params;<br />
uwsgi_modifier1 14;<br />
# hode following headers received from uwsgi, because otherwise we would send them twice since we already add them in nginx itself<br />
uwsgi_hide_header X-Frame-Options;<br />
uwsgi_hide_header X-XSS-Protection;<br />
uwsgi_hide_header X-Content-Type-Options;<br />
uwsgi_hide_header X-Robots-Tag;<br />
uwsgi_hide_header X-Frame-Options;<br />
#Uncomment line below if you get connection refused error. Remember to commet out line with "uwsgi_pass 127.0.0.1:3001;" below<br />
uwsgi_pass unix:/run/uwsgi/owncloud.sock;<br />
#uwsgi_pass 127.0.0.1:3001;<br />
}<br />
...</nowiki>}}<br />
<br />
=== "Reading from keychain failed with error: 'No keychain service available'" ===<br />
<br />
Can be fixed for Gnome by installing the following 2 packages, {{Pkg|libgnome-keyring}} and {{Pkg|gnome-keyring}}.<br />
Or the following for KDE, {{Pkg|libgnome-keyring}} and {{Pkg|qtkeychain}}.<br />
<br />
=== FolderSync: "Method Not Allowed" ===<br />
<br />
FolderSync needs access to {{ic|/owncloud/remote.php/webdav}}, so you could create another alias for owncloud in your {{ic|/etc/httpd/conf/extra/nextcloud.conf}}<br />
<IfModule mod_alias.c><br />
Alias /nextcloud /usr/share/webapps/nextcloud/<br />
Alias /owncloud /usr/share/webapps/nextcloud/<br />
</IfModule><br />
<br />
== Tips and tricks ==<br />
<br />
=== Running ownCloud in a subdirectory ===<br />
<br />
By including the default {{ic|owncloud.conf}} in {{ic|httpd.conf}}, ownCloud will take control of port 80 and your localhost domain. <br />
<br />
If you would like to have ownCloud run in a subdirectory, then edit the {{ic|/etc/httpd/conf/extra/owncloud.conf}}<br />
you included and comment out the {{ic|<nowiki><VirtualHost *:80> ... </VirtualHost></nowiki>}} part of the include file.<br />
<br />
You can use the following nginx config when using owncloud with uwsgi:<br />
{{hc|head=/etc/nginx/conf.d/owncloud.conf|output=<nowiki><br />
location = /.well-known/carddav {<br />
return 301 $scheme://$host/owncloud/remote.php/dav;<br />
}<br />
<br />
location = /.well-known/caldav {<br />
return 301 $scheme://$host/owncloud/remote.php/dav;<br />
}<br />
<br />
location /.well-known/acme-challenge { }<br />
<br />
location ^~ /owncloud {<br />
<br />
root /usr/share/webapps;<br />
<br />
# set max upload size<br />
client_max_body_size 512M;<br />
fastcgi_buffers 64 4K;<br />
<br />
# Disable gzip to avoid the removal of the ETag header<br />
gzip off;<br />
<br />
# Uncomment if your server is build with the ngx_pagespeed module<br />
# This module is currently not supported.<br />
#pagespeed off;<br />
<br />
location /owncloud {<br />
rewrite ^ /owncloud/index.php$uri;<br />
}<br />
<br />
location ~ ^/owncloud/(?:build|tests|config|lib|3rdparty|templates|data)/ {<br />
deny all;<br />
}<br />
<br />
location ~ ^/owncloud/(?:\.|autotest|occ|issue|indie|db_|console) {<br />
deny all;<br />
}<br />
<br />
location ~ ^/owncloud/(?:updater|ocs-provider)(?:$|/) {<br />
try_files $uri/ =404;<br />
index index.php;<br />
}<br />
<br />
location ~ ^/owncloud/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|core/templates/40[34])\.php(?:$|/) {<br />
include uwsgi_params;<br />
uwsgi_modifier1 14;<br />
# Avoid duplicate headers confusing OC checks<br />
uwsgi_hide_header X-Frame-Options;<br />
uwsgi_hide_header X-XSS-Protection;<br />
uwsgi_hide_header X-Content-Type-Options;<br />
uwsgi_hide_header X-Robots-Tag;<br />
uwsgi_pass unix:/run/uwsgi/owncloud.sock;<br />
}<br />
<br />
# Adding the cache control header for js and css files<br />
# Make sure it is BELOW the PHP block<br />
location ~* \.(?:css|js) {<br />
try_files $uri /owncloud/index.php$uri$is_args$args;<br />
add_header Cache-Control "public, max-age=7200";<br />
# Add headers to serve security related headers (It is intended<br />
# to have those duplicated to the ones above)<br />
# Before enabling Strict-Transport-Security headers please read<br />
# into this topic first.<br />
# add_header Strict-Transport-Security "max-age=15768000;<br />
# includeSubDomains; preload;";<br />
add_header X-Content-Type-Options nosniff;<br />
add_header X-Frame-Options "SAMEORIGIN";<br />
add_header X-XSS-Protection "1; mode=block";<br />
add_header X-Robots-Tag none;<br />
add_header X-Download-Options noopen;<br />
add_header X-Permitted-Cross-Domain-Policies none;<br />
# Optional: Don't log access to assets<br />
access_log off;<br />
}<br />
<br />
location ~* \.(?:svg|gif|png|html|ttf|woff|ico|jpg|jpeg) {<br />
try_files $uri /owncloud/index.php$uri$is_args$args;<br />
# Optional: Don't log access to other assets<br />
access_log off;<br />
}<br />
}<br />
</nowiki>}}<br />
<br />
=== Docker ===<br />
See the [https://hub.docker.com/_/owncloud/ ownCloud] or [https://github.com/nextcloud/docker Nextcloud] repository for [[Docker]].<br />
<br />
=== Upload and share from File Manager ===<br />
[https://github.com/schiesbn/shareLinkCreator shareLinkCreator] provides the ability to upload a file to<br />
OwnCloud via a supported file manager and receive a link to the uploaded file which can then be emailed or shared in another way.<br />
<br />
=== Defining Background Jobs ===<br />
<br />
Nextcloud requires scheduled execution of some tasks, and by default it achieves this by using AJAX,<br />
however AJAX is the least reliable method, and it is recommended to use [[Cron]] instead. However,<br />
ArchLinux ships with {{Pkg|systemd}}, so the preferred way of executing scheduled tasks is a [[Systemd#Timers|systemd timer]]. <br />
<br />
First create a service:<br />
<br />
{{hc|/etc/systemd/system/nextcloudcron.service|<nowiki><br />
[Unit]<br />
Description=Nextcloud cron.php job<br />
<br />
[Service]<br />
User=http<br />
ExecStart=/usr/bin/php -f /usr/share/webapps/nextcloud/cron.php<br />
<br />
[Install]<br />
WantedBy=basic.target<br />
</nowiki>}}<br />
<br />
Then create a timer for that service:<br />
<br />
{{hc|/etc/systemd/system/nextcloudcron.timer|<nowiki><br />
[Unit]<br />
Description=Run Nextcloud cron.php every 15 minutes<br />
<br />
[Timer]<br />
OnBootSec=5min<br />
OnUnitActiveSec=15min<br />
Unit=nextcloudcron.service<br />
<br />
[Install]<br />
WantedBy=timers.target<br />
</nowiki>}}<br />
<br />
[[Start/enable]] {{ic|nextcloudcron.timer}}.<br />
<br />
Confirm that it is running by running<br />
# systemctl list-timers<br />
<br />
=== Collabora Online Office integration ===<br />
'''Solution with Docker: ''CODE backend using the official Docker image'''''<br />
<br />
The first, install a {{Pkg|docker}} package to provide collabora files and setup a Collabora server.<br />
<br />
[[Start/enable]] docker.service<br />
<br />
# systemctl enable docker.service<br />
# systemctl start docker.service<br />
<br />
Then, download the required binares :<br />
<br />
# docker pull collabora/code<br />
<br />
And, installing a Collabora server. Make sure {{ic|cloud//.example//.com}} is your nextcloud's domain, not a collabora :<br />
<br />
# docker run -t -d -p 127.0.0.1:9980:9980 -e 'domain=cloud//.example//.com' --restart always --cap-add MKNOD collabora/code<br />
<br />
Also make sure to escape all dots with double backslashes (\), since this string will be evaluated as a regular expression (and your bash 'eats' the first backslash.) If you want to use the docker container with more than one Nextcloud, you'll need to use 'domain=cloud\\.example\\.com\|second\\.example\\.com' instead. (All hosts are separated by \|.) When using `localhost` as domain for testing you need to add {ic|--net host}} to ensure the docker container can access your Nextcloud server. <br />
<br />
If you need to delete or reinstall Collabora server use:<br />
<br />
For recognition CONTAINER_ID of server<br />
# docker ps<br />
<br />
Stop and delete<br />
# docker stop CONTAINER_ID<br />
# docker rm CONTAINER_ID<br />
<br />
Futher, follow the instruction of webserver you are using:<br />
<br />
'''Nginx setup example:'''<br />
<br />
Add following to your nextcloud domain config or add new config file in /etc/nginx/conf.d/ directory, (Don't forget to change {{ic|office.example.com}} and {{ic|ssl_certificate}} to the right values:<br />
<br />
{{hc|/etc/nginx/conf.d/example.conf|<nowiki><br />
upstream office.example.com {<br />
server 127.0.0.1:9980;<br />
}<br />
<br />
server {<br />
listen 443 ssl;<br />
server_name office.example.com;<br />
<br />
ssl_certificate /etc/letsencrypt/live/office.example.com/fullchain.pem;<br />
ssl_certificate_key /etc/letsencrypt/live/office.example.com/privkey.pem;<br />
<br />
# static files<br />
location ^~ /loleaflet {<br />
proxy_pass http://127.0.0.1:9980;<br />
proxy_set_header Host $host;<br />
}<br />
<br />
# WOPI discovery URL<br />
location ^~ /hosting/discovery {<br />
proxy_pass http://127.0.0.1:9980;<br />
proxy_set_header Host $host;<br />
}<br />
<br />
# Main websocket<br />
location ~ /lool/(.*)/ws$ {<br />
proxy_pass http://127.0.0.1:9980;<br />
proxy_set_header Upgrade $http_upgrade;<br />
proxy_set_header Connection "Upgrade";<br />
proxy_set_header Host $host;<br />
proxy_read_timeout 36000s;<br />
}<br />
<br />
# Admin Console websocket<br />
location ^~ /lool/adminws {<br />
proxy_buffering off;<br />
proxy_pass http://127.0.0.1:9980;<br />
proxy_set_header Upgrade $http_upgrade;<br />
proxy_set_header Connection "Upgrade";<br />
proxy_set_header Host $host;<br />
proxy_read_timeout 36000s;<br />
}<br />
<br />
# download, presentation and image upload<br />
location ~ /lool {<br />
proxy_pass http://127.0.0.1:9980;<br />
proxy_set_header Host $host;<br />
}<br />
}<br />
</nowiki>}}<br />
<br />
Restart a nginx:<br />
<br />
# nginx -s reload<br />
or<br />
# systemctl restart nginx.service<br />
<br />
'''Apache setup example:'''<br />
<br />
Add following to nextcloud config file. Don't forget to change to the right values<br />
<br />
{{hc|/etc/httpd/conf/extra/nextcloud.conf|<nowiki><br />
<VirtualHost *:443><br />
ServerName office.nextcloud.com:443<br />
<br />
# SSL configuration, you may want to take the easy route instead and use Lets Encrypt!<br />
SSLEngine on<br />
SSLCertificateFile /path/to/signed_certificate<br />
SSLCertificateChainFile /path/to/intermediate_certificate<br />
SSLCertificateKeyFile /path/to/private/key<br />
SSLProtocol all -SSLv2 -SSLv3<br />
SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS<br />
SSLHonorCipherOrder on<br />
<br />
# Encoded slashes need to be allowed<br />
AllowEncodedSlashes NoDecode<br />
<br />
# Container uses a unique non-signed certificate<br />
SSLProxyEngine On<br />
SSLProxyVerify None<br />
SSLProxyCheckPeerCN Off<br />
SSLProxyCheckPeerName Off<br />
<br />
# keep the host<br />
ProxyPreserveHost On<br />
<br />
# static html, js, images, etc. served from loolwsd<br />
# loleaflet is the client part of LibreOffice Online<br />
ProxyPass /loleaflet https://127.0.0.1:9980/loleaflet retry=0<br />
ProxyPassReverse /loleaflet https://127.0.0.1:9980/loleaflet<br />
<br />
# WOPI discovery URL<br />
ProxyPass /hosting/discovery https://127.0.0.1:9980/hosting/discovery retry=0<br />
ProxyPassReverse /hosting/discovery https://127.0.0.1:9980/hosting/discovery<br />
<br />
# Main websocket<br />
ProxyPassMatch "/lool/(.*)/ws$" wss://127.0.0.1:9980/lool/$1/ws nocanon<br />
<br />
# Admin Console websocket<br />
ProxyPass /lool/adminws wss://127.0.0.1:9980/lool/adminws<br />
<br />
# Download as, Fullscreen presentation and Image upload operations<br />
ProxyPass /lool https://127.0.0.1:9980/lool<br />
ProxyPassReverse /lool https://127.0.0.1:9980/lool<br />
</VirtualHost><br />
</nowiki>}}<br />
<br />
After configuring these do restart your apache:<br />
# systemctl restart httpd<br />
<br />
'''Install the Nextcloud app'''<br />
<br />
Go to the Apps section and choose “Office & Text”, install the “Collabora Online” app. In admin panel select Collabora Online tab and specific the server's domain you have setup before.<br />
<br />
'''Solution without Docker: ''CODE backend using an Archlinux package'''''<br />
<br />
The {{AUR|collabora-online-server-nodocker}} package brings to your Archlinux installation 1º Collabora Office (the desktop suite), and 2º the “CODE” (Collabora Online Development Edition) server, which is based on “lool” (LibreOffice OnLine).<br />
<br />
Alter the `/etc/loolwsd/loolwsd.xml` file, so that:<br />
* `config > server_name` contains the host and port of the public Nextcloud address, separated by a colon (eg. `example.org:443`),<br />
* `config > ssl > enable` is false (ie. web browser —HTTPS→ proxy —HTTP→ loolwsd),<br />
* `config > ssl > termination` is true (I suppose you’ll manage TLS at the proxy level),<br />
* `config > storage > wopi > host` reflects the actual hostname (or pattern) of the proxy server (eg. `(?:.*\.)?example\.org`),<br />
* `config > admin_console > username` and `config > admin_console > password` are set to values of your choice.<br />
<br />
Then:<br />
* start and enable `loolwsd.service`;<br />
* configure Nginx as showed in /usr/share/doc/loolwsd/example.nginx.conf, and restart it.<br />
<br />
== See also ==<br />
* [https://docs.nextcloud.com/ nextcloud official website]<br />
* [https://docs.nextcloud.com/server/12/admin_manual/ nextcloud 12.0 Admin Documentation]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=User:NeoTheFox/Flatpak&diff=508572User:NeoTheFox/Flatpak2018-01-26T18:54:07Z<p>NeoTheFox: Fist version</p>
<hr />
<div><noinclude>{{Template:Template}}<br />
<br />
'''An inline link to search for Flatpak'd packages and to [[Flatpak]] wiki page.'''<br />
<br />
==Usage==<br />
<br />
{{Ic|<nowiki>{{Flatpak|org.developer.Package|Name}}</nowiki>}}<br />
<br />
==Example==<br />
<br />
{{Flatpak|com.play0ad.zeroad|0ad}}<br />
<br />
==See also==<br />
<br />
* [[Template:Pkg]]<br />
* [[Template:Grp]]<br />
* [[Template:AUR]]<br />
<includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{{2}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508571Template:Sandbox2018-01-26T18:52:56Z<p>NeoTheFox: edition 14 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad|0ad}}<br />
<br />
<includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{{2}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508570Template:Sandbox2018-01-26T18:51:48Z<p>NeoTheFox: edition 13 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
<includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{#invoke:String|len|{{{1}}}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508569Template:Sandbox2018-01-26T18:50:59Z<p>NeoTheFox: edition 12 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
<includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{#len:{{{1}}}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508568Template:Sandbox2018-01-26T18:47:17Z<p>NeoTheFox: edition 11 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
<includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{#len:{{1}}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508567Template:Sandbox2018-01-26T18:45:40Z<p>NeoTheFox: edition 10 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
<includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{#invoke:String|sub|{{1}}|2|{{#invoke:String|len|s={{1}} }}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508566Template:Sandbox2018-01-26T18:44:58Z<p>NeoTheFox: edition 9 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
<includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{#invoke:String|sub|{{1}}|2|{{#invoke:String|len|s={{1}} }}}}</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508565Template:Sandbox2018-01-26T18:41:48Z<p>NeoTheFox: edition 8 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
<includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref }}</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508553Template:Sandbox2018-01-26T14:38:24Z<p>NeoTheFox: edition 7 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
<includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{{#invoke:String|replace|source={{{1}}}|pattern=.*{{{sep|\.}}}|replace=|plain=false}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508552Template:Sandbox2018-01-26T14:33:40Z<p>NeoTheFox: edition 6 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
<includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{{#invoke:String|replace|source={{{1}}}|pattern=^[^{{{sep|\.}}}]*{{{sep|\.}}}*|replace=|plain=false}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508551Template:Sandbox2018-01-26T14:32:53Z<p>NeoTheFox: edition 5 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
<includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{{#invoke:String|replace|source={{{1}}}|pattern=^[^{{{sep|.}}}]*{{{sep|.}}}*|replace=|plain=false}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508550Template:Sandbox2018-01-26T14:32:23Z<p>NeoTheFox: edition 4 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
</noinclude><includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{{{#invoke:String|replace|source={{{1}}}|pattern=^[^{{{sep|.}}}]*{{{sep|.}}}*|replace=|plain=false}}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508549Template:Sandbox2018-01-26T14:31:58Z<p>NeoTheFox: edition 4 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
</noinclude><includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{{{{#invoke:String|replace|source={{{1}}}|pattern=^[^{{{sep|.}}}]*{{{sep|.}}}*|replace=|plain=false}}}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508548Template:Sandbox2018-01-26T14:31:19Z<p>NeoTheFox: edition 3 of flatpak template</p>
<hr />
<div><br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
</noinclude><includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{{#invoke:String|replace|source={{{1}}}|pattern=^[^{{{sep|.}}}]*{{{sep|.}}}*|replace=|plain=false}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508546Template:Sandbox2018-01-26T14:22:46Z<p>NeoTheFox: edition 2 of flatpak template</p>
<hr />
<div><noinclude>{{Template:Template}}<br />
<br />
'''An inline link to search for Flatpak'd packages and to [[Flatpak]] wiki page.'''<br />
<br />
==Usage==<br />
<br />
{{Ic|<nowiki>{{Flatpak|org.developer.Package}}</nowiki>}}<br />
<br />
==Example==<br />
<br />
{{Sandbox|com.play0ad.zeroad|0ad}}<br />
<br />
==See also==<br />
<br />
* [[Template:Pkg]]<br />
* [[Template:Grp]]<br />
* [[Template:AUR]]<br />
</noinclude><includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}.flatpakref {{{2}}}]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Template:Sandbox&diff=508545Template:Sandbox2018-01-26T14:21:15Z<p>NeoTheFox: testing flatpak template</p>
<hr />
<div><noinclude>{{Template:Template}}<br />
<br />
'''An inline link to search for Flatpak'd packages and to [[Flatpak]] wiki page.'''<br />
<br />
==Usage==<br />
<br />
{{Ic|<nowiki>{{Flatpak|org.developer.Package}}</nowiki>}}<br />
<br />
==Example==<br />
<br />
{{Sandbox|com.play0ad.zeroad}}<br />
<br />
==See also==<br />
<br />
* [[Template:Pkg]]<br />
* [[Template:Grp]]<br />
* [[Template:AUR]]<br />
</noinclude><includeonly><span class="plainlinks archwiki-template-pkg">[https://flathub.org/repo/appstream/{{urlencode:{{{1}}}}}/ {{{1}}}.flatpakref]</span><sup><small>[[Flatpak]]</small></sup></includeonly></div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=508518PCI passthrough via OVMF2018-01-25T23:13:25Z<p>NeoTheFox: Undo revision 508517 by NeoTheFox (talk) dodm</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=processors&VTD=true List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Merge|QEMU#Enabling IOMMU (Intel VT-d/AMD-Vi) support|Same topic.}}<br />
<br />
IOMMU is a system specific IO mapping mechanism and can be used with most devices. IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
<br />
Before you enable IOMMU, you might have to first enable (non-IOMMU) virtualisation (Intel VT-x/"Vanderpool" or AMD-V/"Pacifica") in your BIOS settings.<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is enabled in your BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
You will also have to enable iommu support in the kernel itself through a [[Kernel parameters|bootloader kernel option]]. Depending on your type of CPU, use either {{ic|1=intel_iommu=on}} for Intel CPUs (VT-d) or {{ic|1=amd_iommu=on}} for AMD CPUs (AMD-Vi).<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for d in /sys/kernel/iommu_groups/*/devices/*; do <br />
n=${d#*/iommu_groups/*}; n=${n%%/*}<br />
printf 'IOMMU Group %s ' "$n"<br />
lspci -nns "${d##*/}"<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0104] (rev 09)<br />
IOMMU Group 1 00:16.0 Communication controller [0780]: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 [8086:1c3a] (rev 04)<br />
IOMMU Group 2 00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 04)<br />
IOMMU Group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev <br />
...<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will be appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the prefered method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this.}}<br />
<br />
=== Using vfio-pci ===<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use. If your system supports it, which you can try by running the following command, you should use it. If it returns an error, use pci-stub instead.<br />
<br />
{{hc|$ modinfo vfio-pci|<br />
filename: /lib/modules/4.4.5-1-ARCH/kernel/drivers/vfio/pci/vfio-pci.ko.gz<br />
description: VFIO PCI - User Level meta-driver<br />
author: Alex Williamson <alex.williamson@redhat.com><br />
...<br />
}}<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
You can then add those vendor-device ID pairs to the default parameters passed to vfio-pci whenever it is inserted into the kernel.<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.}}<br />
<br />
This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, {{ic|vfio_pci}} and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]]. Should you change the IDs of the devices in {{ic|/etc/modprobe.d/vfio.conf}}, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
=== Using pci-stub (legacy method, pre-4.1 kernels) ===<br />
<br />
If your kernel does not support vfio-pci, you can use the pci-stub module instead, which is a dummy driver used to claim a device and prevent other drivers from using it.<br />
<br />
Pci-stub normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
{{Pkg|linux}} and {{Pkg|linux-lts}} have pci-stub built as a module. You will need to add it to MODULES array in {{ic|mkinitpcio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... pci-stub ...)<br />
}}<br />
<br />
If you did need to add this module to your kernel image configuration manually, you must also [[regenerate the initramfs]].<br />
<br />
Add the relevant PCI device IDs to the {{ic|pci-stubs.ids}} [[kernel parameter]], e.g. {{ic|1=pci-stub.ids=10de:13c2,10de:0fbb}}.<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|pci-stub}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|pci-stub}} to bind with.}}<br />
<br />
Check dmesg output for successful assignment of the device to pci-stub:<br />
<br />
{{hc|dmesg {{!}} grep pci-stub|<br />
[ 2.390128] pci-stub: add 10DE:13C2 sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390143] pci-stub 0000:06:00.0: claimed by stub<br />
[ 2.390150] pci-stub: add 10DE:0FBB sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390159] pci-stub 0000:06:00.1: claimed by stub<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it's possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
After installing {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|ovmf}} and {{Pkg|virt-manager}}, add the path to your OVMF firmware image and runtime variables template to your libvirt config so {{ic|virt-install}} or {{ic|virt-manager}} can find those later on.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
nvram = [<br />
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"<br />
]<br />
}}<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self explainatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the libvirt [[group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps :<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that you have correctly specified the location of your firmware in {{ic|/etc/libvirt/qemu.conf}} and restart {{ic|libvirtd.service}}.<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, go into "Add Hardware" and add a Controller for SCSI drives of the "VirtIO SCSI" model. You can then change the default IDE disk for a SCSI disk, which will bind to said controller.<br />
** Windows VMs will not recognize those drives by default, so you need to download the ISO containing the drivers from [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ here] and add an IDE (or SATA for Windows 8.1 and newer) CD-ROM storage device linking to said ISO, otherwise you will not be able to get Windows to recognize it during the installation process. When prompted to select a disk to install windows on, load the drivers contained on the CD-ROM under ''vioscsi''.<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it's now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate linux/windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. Since switching between threads adds a bit of overhead (because context switching forces the core to change its cache between operations), this can noticeably harm performance on the guest. CPU pinning aims to resolve this as it overrides process scheduling and ensures that the VM threads will always run and only run on those specific cores. Here, for instance, the guest cores 0, 1, 2 and 3 are mapped to the host cores 4, 5, 6 and 7 respectively.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
==== The case of Hyper-threading ====<br />
<br />
If your CPU supports hardware multitasking, also known as Hyper-threading on Intel chips, there are two ways you can go with your CPU pinning. That is, Hyper-threading is simply a very efficient way of running two threads on one CPU at any given time, so while it may give you 8 logical cores on what would otherwise be a quad-core CPU, if the physical core is overloaded, the logical core will not be of any use. One could pin their VM threads on 2 physical cores and their 2 respective threads, but any task overloading those two cores will not be helped by the extra two logical cores, since in the end you are only passing through two cores out of four, not four out of eight. What you should do knowing this depends on what you intend to do with your host while your VM is running.<br />
<br />
This is the abridged content of {{ic|/proc/cpuinfo}} on a quad-core machine with hyper-threading.<br />
<br />
{{hc|$ grep -e "processor" -e "core id" -e "^$" /proc/cpuinfo|<br />
processor : 0<br />
core id : 0<br />
<br />
processor : 1<br />
core id : 1<br />
<br />
processor : 2<br />
core id : 2<br />
<br />
processor : 3<br />
core id : 3<br />
<br />
processor : 4<br />
core id : 0<br />
<br />
processor : 5<br />
core id : 1<br />
<br />
processor : 6<br />
core id : 2<br />
<br />
processor : 7<br />
core id : 3<br />
}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, it would probably be better to pin your VM threads across all of your logical cores, so that the VM can fully take advantage of the spare CPU time on all your cores.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='4' threads='1'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
If you would instead prefer to have the host and guest running intensive tasks at the same time, it would then be preferable to pin a limited amount of physical cores and their respective threads on the guest and leave the rest to the host to avoid the two competing for CPU time.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='6'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='2' threads='2'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
=== Static huge pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information accross multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4GB of memory divided into 4kB pages (which is the default size for normal pages), meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession. This is normally handeled with transparent huge pages, which dynamically manages hugepages to keep up with the demand.<br />
<br />
On a VM with a PCI passthrough, however, it is '''not possible''' to benefit from transparent huge pages, as IOMMU requires that the guest's memory be allocated and pinned as soon as the VM starts. It is therefore required to allocate huge pages statically in order to benefit from them. <br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4GBs worth of huge pages on a machine with 8GBs of memory will only leave you with 4GBs of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel comand line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048kB per huge page creates 2GBs worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GB huge page support could be verified by {{ic|<nowiki>grep pdpe1gb /proc/cpuinfo</nowiki>}}. Setting 1 GB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X transparent_hugepage=never}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== High DPC Latency ===<br />
<br />
{{Accuracy|As far as I can tell all virtio modules listed here are for virtual devices used when Linux runs as a guest. Loading them on the host serves no purpose.}}<br />
<br />
If you are experiencing high DPC and/or interrupt latency in your Guest VM, ensure you have [[Kernel modules#Manual module handling|loaded the needed virtio kernel modules]] on the host kernel. Loadable virtio kernel modules include: {{ic|virtio-pci}}, {{ic|virtio-net}}, {{ic|virtio-blk}}, {{ic|virtio-balloon}}, {{ic|virtio-ring}} and {{ic|virtio}}. <br />
<br />
After loading one or more of these modules, {{ic|lsmod {{!}} grep virtio}} executed on the host should not return empty.<br />
<br />
==== CPU pinning with isolcpus ====<br />
<br />
Alternatively, make sure that you have isolated CPUs properly. In this example, let us assume you are using CPUs 4-7.<br />
Use the kernel parameters {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. <br />
<br />
{{hc|sudo vim /etc/defaults/grub|<nowiki><br />
...<br />
GRUB_CMDLINE_LINUX="..your other params.. isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"<br />
...<br />
</nowiki>}}<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this reddit comment] for more info.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you're running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you're on the latest). If you're running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
=== Further Tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide]. This guide may be especially helpful if you are experiencing:<br />
<br />
* Moderate CPU load on the host during downloads/uploads from within the guest: See ''Bridge Zero Copy Transmit'' for a potential fix.<br />
* Guests capping out at certain network speeds during downloads/uploads despite virtio-net being used: See ''Multi-queue virtio-net'' for a potential fix.<br />
* Guests "stuttering" under high I/O, despite the same workload not affecting hosts to the same degree: See ''Multi-queue virtio-scsi'' for a potential fix.<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how both pci-stub and vfio-pci use your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/devices/pci*/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
GROUP="0000:00:03.0"<br />
DEVS="0000:03:00.0 0000:03:00.1 ."<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Create {{ic|/etc/modprobe.d/vfio.conf}} with the following:<br />
<br />
install vfio-pci /usr/bin/vfio-pci-override.sh<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}<br />
<br />
Remove any video drivers from MODULES, and add {{ic|vfio-pci}}, and {{ic|vfio_iommu_type1}}<br />
<br />
MODULES=(ext4 vfat vfio-pci vfio_iommu_type1)<br />
<br />
Add {{ic|/etc/modprobe.d/vfio.conf}} and {{ic|/usr/bin/vfio-pci-override.sh}} to FILES:<br />
<br />
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)<br />
<br />
[[Regenerate the initramfs]] and reboot:<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|Not possible at the time as far as I'm aware, but a common issue on various forums.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that's not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.hostfission.com/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total megabytes + 2<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MB + 2 = 17.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 17.82 is bigger than 16 we should choose 32<br />
<br />
Next create a script to create a shared memory file.<br />
<br />
{{hc|/usr/local/bin/looking-glass-init.sh|2=<br />
#!/bin/sh<br />
<br />
touch /dev/shm/looking-glass<br />
chown '''user''':kvm /dev/shm/looking-glass<br />
chmod 660 /dev/shm/looking-glass<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Create a [[systemd]] unit to execute this script during boot<br />
<br />
{{hc|/etc/systemd/system/looking-glass-init.service|2=<br />
[Unit]<br />
Description=Create shared memory for looking glass<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/local/bin/looking-glass-init.sh<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
}}<br />
<br />
[[Start]] and [[enable]] {{ic|looking-glass-init.service}}<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download a signed driver [https://github.com/virtio-win/kvm-guest-drivers-windows/issues/217 from issue 217], as it is not yet available elsewhere.<br />
<br />
Once the driver is installed you must download the latest [https://looking-glass.hostfission.com/downloads looking-glass-host] from the looking glass website and start it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]<br />
It is also possible to make it start automatically on VM boot by editing the {{ic|HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run}} registry and adding a path to the downloaded executable. <br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you don't want to use Spice to control the guest mouse and keyboard you can disable the Spice server<br />
<br />
$ looking-glass-client -s<br />
<br />
Refer to <br />
<br />
$ looking-glass-client --help<br />
<br />
for more info<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream}} is typically sufficient.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it's not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It's recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing though other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting :[[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio environment variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
<qemu:commandline><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_AUDIO_DRV' value<nowiki>=</nowiki>'pa'/><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_PA_SERVER' value<nowiki>=</nowiki>'/run/user/1000/pulse/native'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
Change 1000 under the user directory to your user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and [[systemd/User|user service]] {{ic|pulseaudio.service}}.<br />
<br />
Virtual Machine audio will now be routed through the host as an application. The application {{Pkg|pavucontrol}} can be used to control the output device. Be aware that on Windows guests, this can cause audio crackling without [[#Slowed down audio pumped through HDMI on the video card|using Message-Signaled Interrupts.]]<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functionnal and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
If you have trouble configuring a certain mechanism in your setup, you might want to look up [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]]. A few users have described their setups and you might want to look up certain tricks from their configuration files.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the cpu parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. It may help for the ID to be set to a 12-character alphanumeric (e.g. '123456789ab') as opposed to longer or shorter strings.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
==== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ====<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
=== UEFI (OVMF) Compatability in VBIOS ===<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it's not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it's stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it's pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
Some tools named {{ic|MSI_util}} or similar are available on the Internet, but they do not work on Windows 10 64bit.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read {{ic|Graphics Problems?}} in [https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt Intel-IOMMU.txt] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X doesn't start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user account is added to the kvm and libvirt [[groups]].<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script from https://www.youtube.com/watch?v=37D2bRsthfI]<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://webchat.freenode.net/?channels=vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=508517PCI passthrough via OVMF2018-01-25T23:07:51Z<p>NeoTheFox: /* Adding IVSHMEM Device to VM */ Make "32" bold so it would stand out</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=processors&VTD=true List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Merge|QEMU#Enabling IOMMU (Intel VT-d/AMD-Vi) support|Same topic.}}<br />
<br />
IOMMU is a system specific IO mapping mechanism and can be used with most devices. IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
<br />
Before you enable IOMMU, you might have to first enable (non-IOMMU) virtualisation (Intel VT-x/"Vanderpool" or AMD-V/"Pacifica") in your BIOS settings.<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is enabled in your BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
You will also have to enable iommu support in the kernel itself through a [[Kernel parameters|bootloader kernel option]]. Depending on your type of CPU, use either {{ic|1=intel_iommu=on}} for Intel CPUs (VT-d) or {{ic|1=amd_iommu=on}} for AMD CPUs (AMD-Vi).<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for d in /sys/kernel/iommu_groups/*/devices/*; do <br />
n=${d#*/iommu_groups/*}; n=${n%%/*}<br />
printf 'IOMMU Group %s ' "$n"<br />
lspci -nns "${d##*/}"<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0104] (rev 09)<br />
IOMMU Group 1 00:16.0 Communication controller [0780]: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 [8086:1c3a] (rev 04)<br />
IOMMU Group 2 00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 04)<br />
IOMMU Group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev <br />
...<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will be appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the prefered method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this.}}<br />
<br />
=== Using vfio-pci ===<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use. If your system supports it, which you can try by running the following command, you should use it. If it returns an error, use pci-stub instead.<br />
<br />
{{hc|$ modinfo vfio-pci|<br />
filename: /lib/modules/4.4.5-1-ARCH/kernel/drivers/vfio/pci/vfio-pci.ko.gz<br />
description: VFIO PCI - User Level meta-driver<br />
author: Alex Williamson <alex.williamson@redhat.com><br />
...<br />
}}<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
You can then add those vendor-device ID pairs to the default parameters passed to vfio-pci whenever it is inserted into the kernel.<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.}}<br />
<br />
This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, {{ic|vfio_pci}} and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]]. Should you change the IDs of the devices in {{ic|/etc/modprobe.d/vfio.conf}}, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
=== Using pci-stub (legacy method, pre-4.1 kernels) ===<br />
<br />
If your kernel does not support vfio-pci, you can use the pci-stub module instead, which is a dummy driver used to claim a device and prevent other drivers from using it.<br />
<br />
Pci-stub normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
{{Pkg|linux}} and {{Pkg|linux-lts}} have pci-stub built as a module. You will need to add it to MODULES array in {{ic|mkinitpcio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... pci-stub ...)<br />
}}<br />
<br />
If you did need to add this module to your kernel image configuration manually, you must also [[regenerate the initramfs]].<br />
<br />
Add the relevant PCI device IDs to the {{ic|pci-stubs.ids}} [[kernel parameter]], e.g. {{ic|1=pci-stub.ids=10de:13c2,10de:0fbb}}.<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|pci-stub}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|pci-stub}} to bind with.}}<br />
<br />
Check dmesg output for successful assignment of the device to pci-stub:<br />
<br />
{{hc|dmesg {{!}} grep pci-stub|<br />
[ 2.390128] pci-stub: add 10DE:13C2 sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390143] pci-stub 0000:06:00.0: claimed by stub<br />
[ 2.390150] pci-stub: add 10DE:0FBB sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390159] pci-stub 0000:06:00.1: claimed by stub<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it's possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
After installing {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|ovmf}} and {{Pkg|virt-manager}}, add the path to your OVMF firmware image and runtime variables template to your libvirt config so {{ic|virt-install}} or {{ic|virt-manager}} can find those later on.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
nvram = [<br />
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"<br />
]<br />
}}<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self explainatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the libvirt [[group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps :<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that you have correctly specified the location of your firmware in {{ic|/etc/libvirt/qemu.conf}} and restart {{ic|libvirtd.service}}.<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, go into "Add Hardware" and add a Controller for SCSI drives of the "VirtIO SCSI" model. You can then change the default IDE disk for a SCSI disk, which will bind to said controller.<br />
** Windows VMs will not recognize those drives by default, so you need to download the ISO containing the drivers from [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ here] and add an IDE (or SATA for Windows 8.1 and newer) CD-ROM storage device linking to said ISO, otherwise you will not be able to get Windows to recognize it during the installation process. When prompted to select a disk to install windows on, load the drivers contained on the CD-ROM under ''vioscsi''.<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it's now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate linux/windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. Since switching between threads adds a bit of overhead (because context switching forces the core to change its cache between operations), this can noticeably harm performance on the guest. CPU pinning aims to resolve this as it overrides process scheduling and ensures that the VM threads will always run and only run on those specific cores. Here, for instance, the guest cores 0, 1, 2 and 3 are mapped to the host cores 4, 5, 6 and 7 respectively.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
==== The case of Hyper-threading ====<br />
<br />
If your CPU supports hardware multitasking, also known as Hyper-threading on Intel chips, there are two ways you can go with your CPU pinning. That is, Hyper-threading is simply a very efficient way of running two threads on one CPU at any given time, so while it may give you 8 logical cores on what would otherwise be a quad-core CPU, if the physical core is overloaded, the logical core will not be of any use. One could pin their VM threads on 2 physical cores and their 2 respective threads, but any task overloading those two cores will not be helped by the extra two logical cores, since in the end you are only passing through two cores out of four, not four out of eight. What you should do knowing this depends on what you intend to do with your host while your VM is running.<br />
<br />
This is the abridged content of {{ic|/proc/cpuinfo}} on a quad-core machine with hyper-threading.<br />
<br />
{{hc|$ grep -e "processor" -e "core id" -e "^$" /proc/cpuinfo|<br />
processor : 0<br />
core id : 0<br />
<br />
processor : 1<br />
core id : 1<br />
<br />
processor : 2<br />
core id : 2<br />
<br />
processor : 3<br />
core id : 3<br />
<br />
processor : 4<br />
core id : 0<br />
<br />
processor : 5<br />
core id : 1<br />
<br />
processor : 6<br />
core id : 2<br />
<br />
processor : 7<br />
core id : 3<br />
}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, it would probably be better to pin your VM threads across all of your logical cores, so that the VM can fully take advantage of the spare CPU time on all your cores.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='4' threads='1'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
If you would instead prefer to have the host and guest running intensive tasks at the same time, it would then be preferable to pin a limited amount of physical cores and their respective threads on the guest and leave the rest to the host to avoid the two competing for CPU time.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='6'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='2' threads='2'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
=== Static huge pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information accross multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4GB of memory divided into 4kB pages (which is the default size for normal pages), meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession. This is normally handeled with transparent huge pages, which dynamically manages hugepages to keep up with the demand.<br />
<br />
On a VM with a PCI passthrough, however, it is '''not possible''' to benefit from transparent huge pages, as IOMMU requires that the guest's memory be allocated and pinned as soon as the VM starts. It is therefore required to allocate huge pages statically in order to benefit from them. <br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4GBs worth of huge pages on a machine with 8GBs of memory will only leave you with 4GBs of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel comand line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048kB per huge page creates 2GBs worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GB huge page support could be verified by {{ic|<nowiki>grep pdpe1gb /proc/cpuinfo</nowiki>}}. Setting 1 GB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X transparent_hugepage=never}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== High DPC Latency ===<br />
<br />
{{Accuracy|As far as I can tell all virtio modules listed here are for virtual devices used when Linux runs as a guest. Loading them on the host serves no purpose.}}<br />
<br />
If you are experiencing high DPC and/or interrupt latency in your Guest VM, ensure you have [[Kernel modules#Manual module handling|loaded the needed virtio kernel modules]] on the host kernel. Loadable virtio kernel modules include: {{ic|virtio-pci}}, {{ic|virtio-net}}, {{ic|virtio-blk}}, {{ic|virtio-balloon}}, {{ic|virtio-ring}} and {{ic|virtio}}. <br />
<br />
After loading one or more of these modules, {{ic|lsmod {{!}} grep virtio}} executed on the host should not return empty.<br />
<br />
==== CPU pinning with isolcpus ====<br />
<br />
Alternatively, make sure that you have isolated CPUs properly. In this example, let us assume you are using CPUs 4-7.<br />
Use the kernel parameters {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. <br />
<br />
{{hc|sudo vim /etc/defaults/grub|<nowiki><br />
...<br />
GRUB_CMDLINE_LINUX="..your other params.. isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"<br />
...<br />
</nowiki>}}<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this reddit comment] for more info.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you're running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you're on the latest). If you're running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
=== Further Tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide]. This guide may be especially helpful if you are experiencing:<br />
<br />
* Moderate CPU load on the host during downloads/uploads from within the guest: See ''Bridge Zero Copy Transmit'' for a potential fix.<br />
* Guests capping out at certain network speeds during downloads/uploads despite virtio-net being used: See ''Multi-queue virtio-net'' for a potential fix.<br />
* Guests "stuttering" under high I/O, despite the same workload not affecting hosts to the same degree: See ''Multi-queue virtio-scsi'' for a potential fix.<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how both pci-stub and vfio-pci use your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/devices/pci*/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
GROUP="0000:00:03.0"<br />
DEVS="0000:03:00.0 0000:03:00.1 ."<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Create {{ic|/etc/modprobe.d/vfio.conf}} with the following:<br />
<br />
install vfio-pci /usr/bin/vfio-pci-override.sh<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}<br />
<br />
Remove any video drivers from MODULES, and add {{ic|vfio-pci}}, and {{ic|vfio_iommu_type1}}<br />
<br />
MODULES=(ext4 vfat vfio-pci vfio_iommu_type1)<br />
<br />
Add {{ic|/etc/modprobe.d/vfio.conf}} and {{ic|/usr/bin/vfio-pci-override.sh}} to FILES:<br />
<br />
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)<br />
<br />
[[Regenerate the initramfs]] and reboot:<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|Not possible at the time as far as I'm aware, but a common issue on various forums.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that's not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.hostfission.com/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>'''32'''</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total megabytes + 2<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MB + 2 = 17.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 17.82 is bigger than 16 we should choose 32<br />
<br />
Next create a script to create a shared memory file.<br />
<br />
{{hc|/usr/local/bin/looking-glass-init.sh|2=<br />
#!/bin/sh<br />
<br />
touch /dev/shm/looking-glass<br />
chown '''user''':kvm /dev/shm/looking-glass<br />
chmod 660 /dev/shm/looking-glass<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Create a [[systemd]] unit to execute this script during boot<br />
<br />
{{hc|/etc/systemd/system/looking-glass-init.service|2=<br />
[Unit]<br />
Description=Create shared memory for looking glass<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/local/bin/looking-glass-init.sh<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
}}<br />
<br />
[[Start]] and [[enable]] {{ic|looking-glass-init.service}}<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download a signed driver [https://github.com/virtio-win/kvm-guest-drivers-windows/issues/217 from issue 217], as it is not yet available elsewhere.<br />
<br />
Once the driver is installed you must download the latest [https://looking-glass.hostfission.com/downloads looking-glass-host] from the looking glass website and start it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]<br />
It is also possible to make it start automatically on VM boot by editing the {{ic|HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run}} registry and adding a path to the downloaded executable. <br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you don't want to use Spice to control the guest mouse and keyboard you can disable the Spice server<br />
<br />
$ looking-glass-client -s<br />
<br />
Refer to <br />
<br />
$ looking-glass-client --help<br />
<br />
for more info<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream}} is typically sufficient.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it's not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It's recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing though other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting :[[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio environment variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
<qemu:commandline><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_AUDIO_DRV' value<nowiki>=</nowiki>'pa'/><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_PA_SERVER' value<nowiki>=</nowiki>'/run/user/1000/pulse/native'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
Change 1000 under the user directory to your user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and [[systemd/User|user service]] {{ic|pulseaudio.service}}.<br />
<br />
Virtual Machine audio will now be routed through the host as an application. The application {{Pkg|pavucontrol}} can be used to control the output device. Be aware that on Windows guests, this can cause audio crackling without [[#Slowed down audio pumped through HDMI on the video card|using Message-Signaled Interrupts.]]<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functionnal and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
If you have trouble configuring a certain mechanism in your setup, you might want to look up [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]]. A few users have described their setups and you might want to look up certain tricks from their configuration files.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the cpu parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. It may help for the ID to be set to a 12-character alphanumeric (e.g. '123456789ab') as opposed to longer or shorter strings.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
==== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ====<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
=== UEFI (OVMF) Compatability in VBIOS ===<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it's not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it's stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it's pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
Some tools named {{ic|MSI_util}} or similar are available on the Internet, but they do not work on Windows 10 64bit.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read {{ic|Graphics Problems?}} in [https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt Intel-IOMMU.txt] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X doesn't start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user account is added to the kvm and libvirt [[groups]].<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script from https://www.youtube.com/watch?v=37D2bRsthfI]<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://webchat.freenode.net/?channels=vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=508516PCI passthrough via OVMF2018-01-25T23:07:21Z<p>NeoTheFox: /* Adding IVSHMEM Device to VM */ Clarified how IVSHMEM works</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=processors&VTD=true List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Merge|QEMU#Enabling IOMMU (Intel VT-d/AMD-Vi) support|Same topic.}}<br />
<br />
IOMMU is a system specific IO mapping mechanism and can be used with most devices. IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
<br />
Before you enable IOMMU, you might have to first enable (non-IOMMU) virtualisation (Intel VT-x/"Vanderpool" or AMD-V/"Pacifica") in your BIOS settings.<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is enabled in your BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
You will also have to enable iommu support in the kernel itself through a [[Kernel parameters|bootloader kernel option]]. Depending on your type of CPU, use either {{ic|1=intel_iommu=on}} for Intel CPUs (VT-d) or {{ic|1=amd_iommu=on}} for AMD CPUs (AMD-Vi).<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for d in /sys/kernel/iommu_groups/*/devices/*; do <br />
n=${d#*/iommu_groups/*}; n=${n%%/*}<br />
printf 'IOMMU Group %s ' "$n"<br />
lspci -nns "${d##*/}"<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0104] (rev 09)<br />
IOMMU Group 1 00:16.0 Communication controller [0780]: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 [8086:1c3a] (rev 04)<br />
IOMMU Group 2 00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 04)<br />
IOMMU Group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev <br />
...<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will be appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the prefered method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this.}}<br />
<br />
=== Using vfio-pci ===<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use. If your system supports it, which you can try by running the following command, you should use it. If it returns an error, use pci-stub instead.<br />
<br />
{{hc|$ modinfo vfio-pci|<br />
filename: /lib/modules/4.4.5-1-ARCH/kernel/drivers/vfio/pci/vfio-pci.ko.gz<br />
description: VFIO PCI - User Level meta-driver<br />
author: Alex Williamson <alex.williamson@redhat.com><br />
...<br />
}}<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
You can then add those vendor-device ID pairs to the default parameters passed to vfio-pci whenever it is inserted into the kernel.<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.}}<br />
<br />
This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, {{ic|vfio_pci}} and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]]. Should you change the IDs of the devices in {{ic|/etc/modprobe.d/vfio.conf}}, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
=== Using pci-stub (legacy method, pre-4.1 kernels) ===<br />
<br />
If your kernel does not support vfio-pci, you can use the pci-stub module instead, which is a dummy driver used to claim a device and prevent other drivers from using it.<br />
<br />
Pci-stub normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
{{Pkg|linux}} and {{Pkg|linux-lts}} have pci-stub built as a module. You will need to add it to MODULES array in {{ic|mkinitpcio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... pci-stub ...)<br />
}}<br />
<br />
If you did need to add this module to your kernel image configuration manually, you must also [[regenerate the initramfs]].<br />
<br />
Add the relevant PCI device IDs to the {{ic|pci-stubs.ids}} [[kernel parameter]], e.g. {{ic|1=pci-stub.ids=10de:13c2,10de:0fbb}}.<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|pci-stub}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|pci-stub}} to bind with.}}<br />
<br />
Check dmesg output for successful assignment of the device to pci-stub:<br />
<br />
{{hc|dmesg {{!}} grep pci-stub|<br />
[ 2.390128] pci-stub: add 10DE:13C2 sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390143] pci-stub 0000:06:00.0: claimed by stub<br />
[ 2.390150] pci-stub: add 10DE:0FBB sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390159] pci-stub 0000:06:00.1: claimed by stub<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it's possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
After installing {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|ovmf}} and {{Pkg|virt-manager}}, add the path to your OVMF firmware image and runtime variables template to your libvirt config so {{ic|virt-install}} or {{ic|virt-manager}} can find those later on.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
nvram = [<br />
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"<br />
]<br />
}}<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self explainatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the libvirt [[group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps :<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that you have correctly specified the location of your firmware in {{ic|/etc/libvirt/qemu.conf}} and restart {{ic|libvirtd.service}}.<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, go into "Add Hardware" and add a Controller for SCSI drives of the "VirtIO SCSI" model. You can then change the default IDE disk for a SCSI disk, which will bind to said controller.<br />
** Windows VMs will not recognize those drives by default, so you need to download the ISO containing the drivers from [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ here] and add an IDE (or SATA for Windows 8.1 and newer) CD-ROM storage device linking to said ISO, otherwise you will not be able to get Windows to recognize it during the installation process. When prompted to select a disk to install windows on, load the drivers contained on the CD-ROM under ''vioscsi''.<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it's now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate linux/windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. Since switching between threads adds a bit of overhead (because context switching forces the core to change its cache between operations), this can noticeably harm performance on the guest. CPU pinning aims to resolve this as it overrides process scheduling and ensures that the VM threads will always run and only run on those specific cores. Here, for instance, the guest cores 0, 1, 2 and 3 are mapped to the host cores 4, 5, 6 and 7 respectively.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
==== The case of Hyper-threading ====<br />
<br />
If your CPU supports hardware multitasking, also known as Hyper-threading on Intel chips, there are two ways you can go with your CPU pinning. That is, Hyper-threading is simply a very efficient way of running two threads on one CPU at any given time, so while it may give you 8 logical cores on what would otherwise be a quad-core CPU, if the physical core is overloaded, the logical core will not be of any use. One could pin their VM threads on 2 physical cores and their 2 respective threads, but any task overloading those two cores will not be helped by the extra two logical cores, since in the end you are only passing through two cores out of four, not four out of eight. What you should do knowing this depends on what you intend to do with your host while your VM is running.<br />
<br />
This is the abridged content of {{ic|/proc/cpuinfo}} on a quad-core machine with hyper-threading.<br />
<br />
{{hc|$ grep -e "processor" -e "core id" -e "^$" /proc/cpuinfo|<br />
processor : 0<br />
core id : 0<br />
<br />
processor : 1<br />
core id : 1<br />
<br />
processor : 2<br />
core id : 2<br />
<br />
processor : 3<br />
core id : 3<br />
<br />
processor : 4<br />
core id : 0<br />
<br />
processor : 5<br />
core id : 1<br />
<br />
processor : 6<br />
core id : 2<br />
<br />
processor : 7<br />
core id : 3<br />
}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, it would probably be better to pin your VM threads across all of your logical cores, so that the VM can fully take advantage of the spare CPU time on all your cores.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='4' threads='1'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
If you would instead prefer to have the host and guest running intensive tasks at the same time, it would then be preferable to pin a limited amount of physical cores and their respective threads on the guest and leave the rest to the host to avoid the two competing for CPU time.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='6'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='2' threads='2'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
=== Static huge pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information accross multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4GB of memory divided into 4kB pages (which is the default size for normal pages), meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession. This is normally handeled with transparent huge pages, which dynamically manages hugepages to keep up with the demand.<br />
<br />
On a VM with a PCI passthrough, however, it is '''not possible''' to benefit from transparent huge pages, as IOMMU requires that the guest's memory be allocated and pinned as soon as the VM starts. It is therefore required to allocate huge pages statically in order to benefit from them. <br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4GBs worth of huge pages on a machine with 8GBs of memory will only leave you with 4GBs of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel comand line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048kB per huge page creates 2GBs worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GB huge page support could be verified by {{ic|<nowiki>grep pdpe1gb /proc/cpuinfo</nowiki>}}. Setting 1 GB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X transparent_hugepage=never}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== High DPC Latency ===<br />
<br />
{{Accuracy|As far as I can tell all virtio modules listed here are for virtual devices used when Linux runs as a guest. Loading them on the host serves no purpose.}}<br />
<br />
If you are experiencing high DPC and/or interrupt latency in your Guest VM, ensure you have [[Kernel modules#Manual module handling|loaded the needed virtio kernel modules]] on the host kernel. Loadable virtio kernel modules include: {{ic|virtio-pci}}, {{ic|virtio-net}}, {{ic|virtio-blk}}, {{ic|virtio-balloon}}, {{ic|virtio-ring}} and {{ic|virtio}}. <br />
<br />
After loading one or more of these modules, {{ic|lsmod {{!}} grep virtio}} executed on the host should not return empty.<br />
<br />
==== CPU pinning with isolcpus ====<br />
<br />
Alternatively, make sure that you have isolated CPUs properly. In this example, let us assume you are using CPUs 4-7.<br />
Use the kernel parameters {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. <br />
<br />
{{hc|sudo vim /etc/defaults/grub|<nowiki><br />
...<br />
GRUB_CMDLINE_LINUX="..your other params.. isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"<br />
...<br />
</nowiki>}}<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this reddit comment] for more info.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you're running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you're on the latest). If you're running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
=== Further Tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide]. This guide may be especially helpful if you are experiencing:<br />
<br />
* Moderate CPU load on the host during downloads/uploads from within the guest: See ''Bridge Zero Copy Transmit'' for a potential fix.<br />
* Guests capping out at certain network speeds during downloads/uploads despite virtio-net being used: See ''Multi-queue virtio-net'' for a potential fix.<br />
* Guests "stuttering" under high I/O, despite the same workload not affecting hosts to the same degree: See ''Multi-queue virtio-scsi'' for a potential fix.<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how both pci-stub and vfio-pci use your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/devices/pci*/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
GROUP="0000:00:03.0"<br />
DEVS="0000:03:00.0 0000:03:00.1 ."<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Create {{ic|/etc/modprobe.d/vfio.conf}} with the following:<br />
<br />
install vfio-pci /usr/bin/vfio-pci-override.sh<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}<br />
<br />
Remove any video drivers from MODULES, and add {{ic|vfio-pci}}, and {{ic|vfio_iommu_type1}}<br />
<br />
MODULES=(ext4 vfat vfio-pci vfio_iommu_type1)<br />
<br />
Add {{ic|/etc/modprobe.d/vfio.conf}} and {{ic|/usr/bin/vfio-pci-override.sh}} to FILES:<br />
<br />
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)<br />
<br />
[[Regenerate the initramfs]] and reboot:<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|Not possible at the time as far as I'm aware, but a common issue on various forums.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that's not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.hostfission.com/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
Looking glass works by creating a shared memory buffer between a host and a guest. This is a lot faster than streaming frames via localhost, but requires additional setup. <br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total megabytes + 2<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MB + 2 = 17.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 17.82 is bigger than 16 we should choose 32<br />
<br />
Next create a script to create a shared memory file.<br />
<br />
{{hc|/usr/local/bin/looking-glass-init.sh|2=<br />
#!/bin/sh<br />
<br />
touch /dev/shm/looking-glass<br />
chown '''user''':kvm /dev/shm/looking-glass<br />
chmod 660 /dev/shm/looking-glass<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Create a [[systemd]] unit to execute this script during boot<br />
<br />
{{hc|/etc/systemd/system/looking-glass-init.service|2=<br />
[Unit]<br />
Description=Create shared memory for looking glass<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/local/bin/looking-glass-init.sh<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
}}<br />
<br />
[[Start]] and [[enable]] {{ic|looking-glass-init.service}}<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download a signed driver [https://github.com/virtio-win/kvm-guest-drivers-windows/issues/217 from issue 217], as it is not yet available elsewhere.<br />
<br />
Once the driver is installed you must download the latest [https://looking-glass.hostfission.com/downloads looking-glass-host] from the looking glass website and start it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]<br />
It is also possible to make it start automatically on VM boot by editing the {{ic|HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run}} registry and adding a path to the downloaded executable. <br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you don't want to use Spice to control the guest mouse and keyboard you can disable the Spice server<br />
<br />
$ looking-glass-client -s<br />
<br />
Refer to <br />
<br />
$ looking-glass-client --help<br />
<br />
for more info<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream}} is typically sufficient.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it's not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It's recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing though other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting :[[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio environment variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
<qemu:commandline><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_AUDIO_DRV' value<nowiki>=</nowiki>'pa'/><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_PA_SERVER' value<nowiki>=</nowiki>'/run/user/1000/pulse/native'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
Change 1000 under the user directory to your user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and [[systemd/User|user service]] {{ic|pulseaudio.service}}.<br />
<br />
Virtual Machine audio will now be routed through the host as an application. The application {{Pkg|pavucontrol}} can be used to control the output device. Be aware that on Windows guests, this can cause audio crackling without [[#Slowed down audio pumped through HDMI on the video card|using Message-Signaled Interrupts.]]<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functionnal and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
If you have trouble configuring a certain mechanism in your setup, you might want to look up [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]]. A few users have described their setups and you might want to look up certain tricks from their configuration files.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the cpu parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. It may help for the ID to be set to a 12-character alphanumeric (e.g. '123456789ab') as opposed to longer or shorter strings.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
==== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ====<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
=== UEFI (OVMF) Compatability in VBIOS ===<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it's not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it's stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it's pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
Some tools named {{ic|MSI_util}} or similar are available on the Internet, but they do not work on Windows 10 64bit.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read {{ic|Graphics Problems?}} in [https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt Intel-IOMMU.txt] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X doesn't start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user account is added to the kvm and libvirt [[groups]].<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script from https://www.youtube.com/watch?v=37D2bRsthfI]<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://webchat.freenode.net/?channels=vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=508515PCI passthrough via OVMF2018-01-25T22:56:41Z<p>NeoTheFox: /* Special procedures */ Added Looking Glass setup procedure</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=processors&VTD=true List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
{{Merge|QEMU#Enabling IOMMU (Intel VT-d/AMD-Vi) support|Same topic.}}<br />
<br />
IOMMU is a system specific IO mapping mechanism and can be used with most devices. IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
<br />
Before you enable IOMMU, you might have to first enable (non-IOMMU) virtualisation (Intel VT-x/"Vanderpool" or AMD-V/"Pacifica") in your BIOS settings.<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is enabled in your BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
You will also have to enable iommu support in the kernel itself through a [[Kernel parameters|bootloader kernel option]]. Depending on your type of CPU, use either {{ic|1=intel_iommu=on}} for Intel CPUs (VT-d) or {{ic|1=amd_iommu=on}} for AMD CPUs (AMD-Vi).<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for d in /sys/kernel/iommu_groups/*/devices/*; do <br />
n=${d#*/iommu_groups/*}; n=${n%%/*}<br />
printf 'IOMMU Group %s ' "$n"<br />
lspci -nns "${d##*/}"<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0104] (rev 09)<br />
IOMMU Group 1 00:16.0 Communication controller [0780]: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 [8086:1c3a] (rev 04)<br />
IOMMU Group 2 00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 04)<br />
IOMMU Group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev <br />
...<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will be appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the prefered method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this.}}<br />
<br />
=== Using vfio-pci ===<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use. If your system supports it, which you can try by running the following command, you should use it. If it returns an error, use pci-stub instead.<br />
<br />
{{hc|$ modinfo vfio-pci|<br />
filename: /lib/modules/4.4.5-1-ARCH/kernel/drivers/vfio/pci/vfio-pci.ko.gz<br />
description: VFIO PCI - User Level meta-driver<br />
author: Alex Williamson <alex.williamson@redhat.com><br />
...<br />
}}<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
You can then add those vendor-device ID pairs to the default parameters passed to vfio-pci whenever it is inserted into the kernel.<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.}}<br />
<br />
This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, {{ic|vfio_pci}} and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]]. Should you change the IDs of the devices in {{ic|/etc/modprobe.d/vfio.conf}}, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
=== Using pci-stub (legacy method, pre-4.1 kernels) ===<br />
<br />
If your kernel does not support vfio-pci, you can use the pci-stub module instead, which is a dummy driver used to claim a device and prevent other drivers from using it.<br />
<br />
Pci-stub normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
{{Pkg|linux}} and {{Pkg|linux-lts}} have pci-stub built as a module. You will need to add it to MODULES array in {{ic|mkinitpcio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... pci-stub ...)<br />
}}<br />
<br />
If you did need to add this module to your kernel image configuration manually, you must also [[regenerate the initramfs]].<br />
<br />
Add the relevant PCI device IDs to the {{ic|pci-stubs.ids}} [[kernel parameter]], e.g. {{ic|1=pci-stub.ids=10de:13c2,10de:0fbb}}.<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|pci-stub}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|pci-stub}} to bind with.}}<br />
<br />
Check dmesg output for successful assignment of the device to pci-stub:<br />
<br />
{{hc|dmesg {{!}} grep pci-stub|<br />
[ 2.390128] pci-stub: add 10DE:13C2 sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390143] pci-stub 0000:06:00.0: claimed by stub<br />
[ 2.390150] pci-stub: add 10DE:0FBB sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390159] pci-stub 0000:06:00.1: claimed by stub<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it's possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
After installing {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|ovmf}} and {{Pkg|virt-manager}}, add the path to your OVMF firmware image and runtime variables template to your libvirt config so {{ic|virt-install}} or {{ic|virt-manager}} can find those later on.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
nvram = [<br />
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"<br />
]<br />
}}<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self explainatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the libvirt [[group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps :<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that you have correctly specified the location of your firmware in {{ic|/etc/libvirt/qemu.conf}} and restart {{ic|libvirtd.service}}.<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, go into "Add Hardware" and add a Controller for SCSI drives of the "VirtIO SCSI" model. You can then change the default IDE disk for a SCSI disk, which will bind to said controller.<br />
** Windows VMs will not recognize those drives by default, so you need to download the ISO containing the drivers from [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ here] and add an IDE (or SATA for Windows 8.1 and newer) CD-ROM storage device linking to said ISO, otherwise you will not be able to get Windows to recognize it during the installation process. When prompted to select a disk to install windows on, load the drivers contained on the CD-ROM under ''vioscsi''.<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it's now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate linux/windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. Since switching between threads adds a bit of overhead (because context switching forces the core to change its cache between operations), this can noticeably harm performance on the guest. CPU pinning aims to resolve this as it overrides process scheduling and ensures that the VM threads will always run and only run on those specific cores. Here, for instance, the guest cores 0, 1, 2 and 3 are mapped to the host cores 4, 5, 6 and 7 respectively.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
==== The case of Hyper-threading ====<br />
<br />
If your CPU supports hardware multitasking, also known as Hyper-threading on Intel chips, there are two ways you can go with your CPU pinning. That is, Hyper-threading is simply a very efficient way of running two threads on one CPU at any given time, so while it may give you 8 logical cores on what would otherwise be a quad-core CPU, if the physical core is overloaded, the logical core will not be of any use. One could pin their VM threads on 2 physical cores and their 2 respective threads, but any task overloading those two cores will not be helped by the extra two logical cores, since in the end you are only passing through two cores out of four, not four out of eight. What you should do knowing this depends on what you intend to do with your host while your VM is running.<br />
<br />
This is the abridged content of {{ic|/proc/cpuinfo}} on a quad-core machine with hyper-threading.<br />
<br />
{{hc|$ grep -e "processor" -e "core id" -e "^$" /proc/cpuinfo|<br />
processor : 0<br />
core id : 0<br />
<br />
processor : 1<br />
core id : 1<br />
<br />
processor : 2<br />
core id : 2<br />
<br />
processor : 3<br />
core id : 3<br />
<br />
processor : 4<br />
core id : 0<br />
<br />
processor : 5<br />
core id : 1<br />
<br />
processor : 6<br />
core id : 2<br />
<br />
processor : 7<br />
core id : 3<br />
}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, it would probably be better to pin your VM threads across all of your logical cores, so that the VM can fully take advantage of the spare CPU time on all your cores.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='4' threads='1'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
If you would instead prefer to have the host and guest running intensive tasks at the same time, it would then be preferable to pin a limited amount of physical cores and their respective threads on the guest and leave the rest to the host to avoid the two competing for CPU time.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='6'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='2' threads='2'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
=== Static huge pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information accross multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4GB of memory divided into 4kB pages (which is the default size for normal pages), meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession. This is normally handeled with transparent huge pages, which dynamically manages hugepages to keep up with the demand.<br />
<br />
On a VM with a PCI passthrough, however, it is '''not possible''' to benefit from transparent huge pages, as IOMMU requires that the guest's memory be allocated and pinned as soon as the VM starts. It is therefore required to allocate huge pages statically in order to benefit from them. <br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4GBs worth of huge pages on a machine with 8GBs of memory will only leave you with 4GBs of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel comand line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048kB per huge page creates 2GBs worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GB huge page support could be verified by {{ic|<nowiki>grep pdpe1gb /proc/cpuinfo</nowiki>}}. Setting 1 GB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X transparent_hugepage=never}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== High DPC Latency ===<br />
<br />
{{Accuracy|As far as I can tell all virtio modules listed here are for virtual devices used when Linux runs as a guest. Loading them on the host serves no purpose.}}<br />
<br />
If you are experiencing high DPC and/or interrupt latency in your Guest VM, ensure you have [[Kernel modules#Manual module handling|loaded the needed virtio kernel modules]] on the host kernel. Loadable virtio kernel modules include: {{ic|virtio-pci}}, {{ic|virtio-net}}, {{ic|virtio-blk}}, {{ic|virtio-balloon}}, {{ic|virtio-ring}} and {{ic|virtio}}. <br />
<br />
After loading one or more of these modules, {{ic|lsmod {{!}} grep virtio}} executed on the host should not return empty.<br />
<br />
==== CPU pinning with isolcpus ====<br />
<br />
Alternatively, make sure that you have isolated CPUs properly. In this example, let us assume you are using CPUs 4-7.<br />
Use the kernel parameters {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. <br />
<br />
{{hc|sudo vim /etc/defaults/grub|<nowiki><br />
...<br />
GRUB_CMDLINE_LINUX="..your other params.. isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"<br />
...<br />
</nowiki>}}<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this reddit comment] for more info.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you're running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you're on the latest). If you're running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
=== Further Tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide]. This guide may be especially helpful if you are experiencing:<br />
<br />
* Moderate CPU load on the host during downloads/uploads from within the guest: See ''Bridge Zero Copy Transmit'' for a potential fix.<br />
* Guests capping out at certain network speeds during downloads/uploads despite virtio-net being used: See ''Multi-queue virtio-net'' for a potential fix.<br />
* Guests "stuttering" under high I/O, despite the same workload not affecting hosts to the same degree: See ''Multi-queue virtio-scsi'' for a potential fix.<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how both pci-stub and vfio-pci use your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/devices/pci*/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
GROUP="0000:00:03.0"<br />
DEVS="0000:03:00.0 0000:03:00.1 ."<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Create {{ic|/etc/modprobe.d/vfio.conf}} with the following:<br />
<br />
install vfio-pci /usr/bin/vfio-pci-override.sh<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}<br />
<br />
Remove any video drivers from MODULES, and add {{ic|vfio-pci}}, and {{ic|vfio_iommu_type1}}<br />
<br />
MODULES=(ext4 vfat vfio-pci vfio_iommu_type1)<br />
<br />
Add {{ic|/etc/modprobe.d/vfio.conf}} and {{ic|/usr/bin/vfio-pci-override.sh}} to FILES:<br />
<br />
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)<br />
<br />
[[Regenerate the initramfs]] and reboot:<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|Not possible at the time as far as I'm aware, but a common issue on various forums.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that's not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Using Looking Glass to stream guest screen to the host ===<br />
<br />
It is possible to make VM share the monitor, and optionally a keyboard and a mouse with a help of [https://looking-glass.hostfission.com/ Looking Glass].<br />
<br />
==== Adding IVSHMEM Device to VM ====<br />
<br />
With your VM turned off open the machine configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<devices><br />
...<br />
<shmem name='looking-glass'><br />
<model type='ivshmem-plain'/><br />
<size unit='M'>32</size><br />
</shmem><br />
</devices><br />
...<br />
</nowiki>}}<br />
<br />
You should replace 32 with your own calculated value based on what resolution you are going to pass through. It can be calculated like this:<br />
<br />
width x height x 4 x 2 = total bytes<br />
total bytes / 1024 / 1024 = total megabytes + 2<br />
<br />
For example, in case of 1920x1080<br />
<br />
1920 x 1080 x 4 x 2 = 16,588,800 bytes<br />
16,588,800 / 1024 / 1024 = 15.82 MB + 2 = 17.82<br />
<br />
The result must be '''rounded up''' to the nearest power of two, and since 17.82 is bigger than 16 we should choose 32<br />
<br />
Next create a script to create a shared memory file.<br />
<br />
{{hc|/usr/local/bin/looking-glass-init.sh|2=<br />
#!/bin/sh<br />
<br />
touch /dev/shm/looking-glass<br />
chown '''user''':kvm /dev/shm/looking-glass<br />
chmod 660 /dev/shm/looking-glass<br />
}}<br />
<br />
Replace user with your username.<br />
<br />
Create a [[systemd]] unit to execute this script during boot<br />
<br />
{{hc|/etc/systemd/system/looking-glass-init.service|2=<br />
[Unit]<br />
Description=Create shared memory for looking glass<br />
<br />
[Service]<br />
Type=oneshot<br />
ExecStart=/usr/local/bin/looking-glass-init.sh<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
}}<br />
<br />
[[Start]] and [[enable]] {{ic|looking-glass-init.service}}<br />
<br />
==== Installing the IVSHMEM Host to Windows guest ====<br />
<br />
Currently Windows would not notify users about a new IVSHMEM device, it would silently install a dummy driver. To actually enable the device you have to into device manager and update the driver for the device under the "System Devices" node for '''"PCI standard RAM Controller"'''. Download a signed driver [https://github.com/virtio-win/kvm-guest-drivers-windows/issues/217 from issue 217], as it is not yet available elsewhere.<br />
<br />
Once the driver is installed you must download the latest [https://looking-glass.hostfission.com/downloads looking-glass-host] from the looking glass website and start it on your guest. In order to run it you would also need to install Microsoft Visual C++ Redistributable from [https://www.visualstudio.com/downloads/ Microsoft]<br />
It is also possible to make it start automatically on VM boot by editing the {{ic|HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run}} registry and adding a path to the downloaded executable. <br />
<br />
==== Getting a client ====<br />
<br />
Looking glass client can be installed from AUR using {{AUR|looking-glass}} or {{AUR|looking-glass-git}} packages.<br />
You can start it once the VM is set up and running<br />
<br />
$ looking-glass-client<br />
<br />
If you don't want to use Spice to control the guest mouse and keyboard you can disable the Spice server<br />
<br />
$ looking-glass-client -s<br />
<br />
Refer to <br />
<br />
$ looking-glass-client --help<br />
<br />
for more info<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream}} is typically sufficient.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it's not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It's recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing though other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting :[[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio environment variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
<qemu:commandline><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_AUDIO_DRV' value<nowiki>=</nowiki>'pa'/><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_PA_SERVER' value<nowiki>=</nowiki>'/run/user/1000/pulse/native'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
Change 1000 under the user directory to your user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and [[systemd/User|user service]] {{ic|pulseaudio.service}}.<br />
<br />
Virtual Machine audio will now be routed through the host as an application. The application {{Pkg|pavucontrol}} can be used to control the output device. Be aware that on Windows guests, this can cause audio crackling without [[#Slowed down audio pumped through HDMI on the video card|using Message-Signaled Interrupts.]]<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functionnal and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
If you have trouble configuring a certain mechanism in your setup, you might want to look up [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]]. A few users have described their setups and you might want to look up certain tricks from their configuration files.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the cpu parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. It may help for the ID to be set to a 12-character alphanumeric (e.g. '123456789ab') as opposed to longer or shorter strings.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
==== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ====<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
=== UEFI (OVMF) Compatability in VBIOS ===<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it's not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it's stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it's pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
Some tools named {{ic|MSI_util}} or similar are available on the Internet, but they do not work on Windows 10 64bit.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read {{ic|Graphics Problems?}} in [https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt Intel-IOMMU.txt] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X doesn't start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user account is added to the kvm and libvirt [[groups]].<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script from https://www.youtube.com/watch?v=37D2bRsthfI]<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://webchat.freenode.net/?channels=vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=Talk:PCI_passthrough_via_OVMF&diff=508241Talk:PCI passthrough via OVMF2018-01-22T13:00:05Z<p>NeoTheFox: /* Supported Hardware Table */ Added my config</p>
<hr />
<div>== Supported Hardware Table ==<br />
<br />
I think it would be a good idea to add a table of working / not working hardware like this:<br />
<br />
{| class="wikitable"<br />
! Motherboard<br />
! CPU<br />
! Host GPU<br />
! Client GPU<br />
! uname -r<br />
! ACS Override needed<br />
! Notes<br />
|-<br />
| [https://www.asus.com/Motherboards/Z170-A/ ASUS Z170-A]<br />
| Intel i7-6700K<br />
| Integrated<br />
| [https://www.asus.com/Graphics-Cards/STRIXGTX980TIDC3OC6GD5GAMING/ ASUS Strix GTX 980 Ti]<br />
| 4.3.3-3-ARCH<br />
| no<br />
| No problems<br />
|-<br />
| [http://asrock.com/mb/AMD/Fatal1ty%20X370%20Gaming%20X/index.asp Asrock Fatal1ty X370 Gaming X]<br />
| Ryzen 5 1600X<br />
| Nvidia GTX 480<br />
| Nvidia GTX 960<br />
| 4.14.14-1-ck<br />
| no<br />
| Couldn't get it to work with GTX 480 as guest GPU<br />
|-<br />
| <br />
| <br />
| <br />
| <br />
| <br />
| <br />
| <br />
|}<br />
<br />
--[[User:Phiresky|Phiresky]] ([[User talk:Phiresky|talk]]) 12:28, 29 January 2016 (UTC)<br />
<br />
: Well, there's: https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit#gid=2 and it has (if I read it properly) 208 unique motherboards. I'm not sure if Wiki can handle such a long list. [[User:Annisar|Annisar]] ([[User talk:Annisar|talk]])<br />
<br />
: Huh, looks like I missed that. Yeah, that list is probably too long. Kind of unclean / hard to read and can't edit it though. [[User:Phiresky|Phiresky]] ([[User talk:Phiresky|talk]]) 13:20, 29 January 2016 (UTC)<br />
<br />
== Page rewrite ==<br />
<br />
I've fiddled a lot with vfio in the last few months and I've been thinking about restructuring this page based on the information I've gathered over time. Considering large chunks of the page date all the way back from wen it was first written, and that the structure of the page isn't as researched as what you'd find on the rest of the wiki, I think a restructuration could greatly improve the overall understability of the page. Here's the overall structure I had in mind :<br />
; Prerequisites<br />
; Setting up IOMMU<br />
: Enabling IOMMU<br />
: Ensuring that the groups are valid<br />
: Common mistakes/Gotchas<br />
; Isolating the GPU<br />
: Using vfio-pci (Reccomended, Linux 4.1+)<br />
: Using pci-stub (Legacy)<br />
: Common mistakes/Gotchas<br />
; Setting up QEMU-kvm<br />
: Using libvirt<br />
: Using the qemu command line<br />
; Troubleshooting<br />
: Error 43 in Windows on Nvidia cards<br />
: "Invalid ROM content" in dmesg when trying to claim the guest GPU<br />
: ...<br />
; Extras<br />
: ACS override patch<br />
: Improving performance on the VM<br />
<br />
I've already written a good chunk of what's left, but I'd like some feedback on the proposed structure and what's already there before I proceed.<br />
[[User:Victoire|Victoire]] ([[User talk:Victoire|talk]]) 20:55, 8 April 2016 (UTC)<br />
<br />
:Due to a lack of technical background, I can't really comment on completeness of your draft TOC compared to what the article covers, but it reads very structured. I had a look over the last edits you already committed to the article. Well done, I only want to give a couple of hints: <br />
:# Please take care when moving sections and editing, best split the move into its own commit. It is very difficult for someone else (or yourself later) to follow-up on what has been done (see [[ArchWiki:Contributing#Do not make complex edits at once]]). <br />
:# Point 1 is particularly important if you decide content is outdated and remove it during the restructuring. Usually we put [[Template:Deletion]] to the part first. Maybe you can do that as well. However, during a restructure the content might be in the way. You can also use a temporary section e.g. "Obsoleted sections" to move the sections to for the time being. If you delete parts, please do so only per subsection (edit button next to the header). This way it is easier to figure via history what went where. <br />
:# For the language, simple one: We avoid contractions ([[Help:Style#Language register]]) <br />
:# Looking at the existing article, there are some sections (e.g. [[PCI passthrough via OVMF#Complete example for QEMU .28CLI-based.29 without libvirtd]]) with very very long code blocks. Too long for good reading. Two alternatives for those (if you need them in the anticipated structure): Either move the long example code blocks to the end of the article (e.g. the Extras section) and crosslink them from above, quoting only the required excerpts. Or quote as well, but move the actual full code to a non-ad-spoiled gist (e.g. gist.github.com). If you move them outside the wiki, please do the removal/replacement with the gist link in one edit (same reason as for deletions, thanks).<br />
:# In the other talk items there are a few suggestions, e.g. [[#Article_Specificity]] and [[#Supported_Hardware_Table]] that might be useful to consider going forward. <br />
:That said, I hope your push to restructure and refresh the article gets input and help from other contributors to this article. --[[User:Indigo|Indigo]] ([[User talk:Indigo|talk]]) 18:49, 18 April 2016 (UTC)<br />
<br />
:Thanks for all of the contributions, please remember to keep the scope of the page inside it's original design: to achieve PCI passthrough via OVMF. I would like to have the long blocks of code removed/relocated especially those that do not necessarily explain the actual process of getting passthrough to work. Thanks!<br />
:{{unsigned|20:42, 24 April 2016|Naruni}}<br />
<br />
::I think it may have lost some focus because OVMF was not explicitly pointed to in the intro. As suggested in [[#Article_Specificity]], it '''may''' be useful to split content unrelated to OVMF into a separate article, e.g. [[PCI passthrough]] and this article may stay single or become a subpage for it (e.g. [[PCI passthrough/OVMF]]), whatever is best to keep focus, while not duplicating instructions or loose contributions which are not directly OVMF related. <br />
::If you look at the above structure Victoire has worked out, does it contain sections you would consider out-of-scope for OVMF? <br />
::--[[User:Indigo|Indigo]] ([[User talk:Indigo|talk]]) 13:04, 26 April 2016 (UTC)<br />
<br />
:So I have reached the point where most of the structure is now in place and most of the work left is either related to adressing specific issues people are likely to encounter or rephrasing some parts to make them more readable, such as the part on setting up the guest OS, which could use some fleshing out.<br />
:I have also taken the liberty of adding a ''Performance Tuning'' section, which may or may not be out of place considering the scope of the article. I would appreciate some feedback on whether or this belongs here.<br />
:Also, both for the sake of readability and because I couldn't test the instructions myself, I removed most of the QEMU-related (without libvirt) instructions. While it seemed like a good decision at the time, I would like to get some feedback on this, as it is still a removal of potentially useful instructions (although some of them did seem dubtious).<br />
:[[User:Victoire|Victoire]] ([[User talk:Victoire|talk]]) 14:07, 9 May 2016 (UTC)<br />
<br />
::I was hoping others with topic experience chime in to feedback, perhaps that happens still. Arguably the whole article is about performance tuning, so I would see the section you added in scope. It is an interesting read even for someone like me, who has not used any of this. For the QEMU point I'm unable to give feedback. --[[User:Indigo|Indigo]] ([[User talk:Indigo|talk]]) 08:02, 30 June 2016 (UTC)<br />
<br />
::I agree that some of the QEMU scripts removed from the article were too detailed and still not explained well enough, but originally when I was setting up my passthrough system with the help of this article, I preferred the scripted way instead of libvirt and still do after a long good experience with the method. At the time the scripts in the article pointed me to the right direction. I'd really like to contribute to the article with smaller and simpler script examples with decent explanations for using every core parameter needed for the VM started with QEMU commands, so anyone with an opinion about this please comment before I start working on it. The scripted way is great for many use cases and I really think it needs to be addressed better on this article. [[User:Nd43|Nd43]] ([[User talk:Nd43|talk]]) 07:28, 20 April 2017 (UTC)<br />
<br />
==Additionnal sections==<br />
In case I forget, I made a list of things I wanted to add to this article some time ago, I just haven't found the time to write those parts yet.<br />
<br />
* Performance tuning<br />
** Kernel config ([https://www.redhat.com/archives/vfio-users/2016-March/msg00081.html Volountary preemption], 1000 Hz timer frequency, [https://www.youtube.com/watch?v=G3jHP9kNjwc dynticks], halt_poll_ns, etc.)<br />
** Advanced CPUs features<br />
*** 1GB hugepages<br />
*** NUMA node assignment<br />
*** Hardware virtual interrupt management (AVIC/APICv)<br />
<s><br />
* Special Procedures<br />
** Using identical guest and host GPUs<br />
** Passing the boot GPU ([https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html see here])<br />
** Bypassing the IOMMU groups (ACS override patch)<br />
* Additionnal devices<br />
** GPU soundcard (MSI interrupts)</s><br />
<br />
As of now, I don't have the sort of hardware that would allow me to test these special cases, so it's a bit hard for me to justify writing those sections, but it might be interesting to add those someday.<br />
<br />
[[User:Victoire|Victoire]] ([[User talk:Victoire|talk]]) 02:16, 5 August 2016 (UTC)<br />
<br />
EDIT1 : [[User:Victoire|Victoire]] ([[User talk:Victoire|talk]]) 03:10, 9 August 2016 (UTC)<br />
<br />
EDIT2 (Might need to revisit some of those now that Ryzen comes with most of these features) : [[User:Victoire|Victoire]] ([[User talk:Victoire|talk]]) 00:09, 7 August 2017 (UTC)<br />
<br />
== hotplug gpu ==<br />
<br />
found something interresting for hotpluging the gpu without to restart x:<br />
<br />
http://arseniyshestakov.com/2016/03/31/how-to-pass-gpu-to-vm-and-back-without-x-restart/<br />
repo: https://gist.github.com/ArseniyShestakov/dc152d080c65ebaa6781<br />
<br />
It removes the card from the graphics driver module (here radeon) and add's it to the vfio-module.<br />
<br />
I'm currently testing this. <br />
{{unsigned| 22:25, 12 August 2016|Xtf}}<br />
<br />
== Necessity of ovmf-git ==<br />
<br />
Right now, the article recommends using a package from the AUR, {{AUR|ovmf-git}}. The only reason for that is because this version comes with the EFI var template files while the {{Pkg|ovmf}} package from the Extra repos doesn't. I'm not entirely sure how complex those templates are and if there is a way to install OVMF properly without them, though, and it would definitely be best if we could do without the use of an AUR package.<br />
[[User:Victoire|Victoire]] ([[User talk:Victoire|talk]]) 17:40, 22 November 2016 (UTC)<br />
<br />
:I aggree, but has the current maintainer of {{Pkg|ovmf}} been made aware of this issue yet?<br />
:--[[User:Lineks|Lineks]] ([[User talk:Lineks|talk]]) 17:33, 9 April 2017 (UTC)<br />
<br />
:I think whis was fixed on the latest update in march. See https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/ovmf&id=af39dbbc3037734379416016e55d41548a2c2968<br />
:Bug report closed as implemented: https://bugs.archlinux.org/task/47101<br />
:I'll test and confirm.<br />
:--[[User:WeissJT|WeissJT]] ([[User talk:WeissJT|talk]]) 23:15, 8 May 2017 (UTC)<br />
<br />
:Confirmed, {{Pkg|ovmf}} works fine. {{AUR|ovmf-git}} is not needed anymore.<br />
:--[[User:WeissJT|WeissJT]] ([[User talk:WeissJT|talk]]) 23:17, 12 May 2017 (UTC)<br />
<br />
:: Thanks for the confirmation, made the edit on the plain QEMU section too. [[User:Nd43|Nd43]] ([[User talk:Nd43|talk]]) 07:30, 13 May 2017 (UTC)<br />
<br />
== Inaccurate hugepage advice ==<br />
<br />
The static hugepage section says:<br />
"On a VM with a PCI passthrough, however, it is not possible to benefit from transparent huge pages, as IOMMU requires that the guest's memory be allocated and pinned as soon as the VM starts. It is therefore required to allocate huge pages statically in order to benefit from them."<br />
<br />
This may have been true previously but in my experience (currently verified on an Ubuntu Trusty box with kernel 4.4) this is no longer accurate:<br />
<nowiki>gpu-hypervisor:~$ uname -a<br />
Linux rcgpudc1r54-07 4.4.0-59-generic #80~14.04.1-Ubuntu SMP Fri Jan 6 18:02:02 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux<br />
gpu-hypervisor:~$ ps auxww | grep qemu | grep -v qemu<br />
libvirt+ 105915 297 95.3 325650548 251763980 ? SLl Apr10 9725:30 /usr/bin/qemu-system-x86_64 -name instance-00000167 -S -machine pc-i440fx-xenial,accel=kvm,usb=off -cpu host -m 245760 -realtime mlock=off -smp 24,sockets=2,cores=12,threads=1 -object memory-backend-ram,id=ram-node0,size=128849018880,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-11,memdev=ram-node0 -object memory-backend-ram,id=ram-node1,size=128849018880,host-nodes=1,policy=bind -numa node,nodeid=1,cpus=12-23,memdev=ram-node1 -uuid 87681aae-2bc7-4b2e-b17b-f407cf23701e -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=12.0.4,serial=4c4c4544-0059-4710-8036-c3c04f483832,uuid=87681aae-2bc7-4b2e-b17b-f407cf23701e,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-00000167/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/87681aae-2bc7-4b2e-b17b-f407cf23701e/disk,format=raw,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/nova/instances/87681aae-2bc7-4b2e-b17b-f407cf23701e/disk.eph0,format=raw,if=none,id=drive-virtio-disk1,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=31 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:5e:21:1c,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/87681aae-2bc7-4b2e-b17b-f407cf23701e/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device vfio-pci,host=05:00.0,id=hostdev0,bus=pci.0,addr=0x6 -device vfio-pci,host=06:00.0,id=hostdev1,bus=pci.0,addr=0x7 -device vfio-pci,host=85:00.0,id=hostdev2,bus=pci.0,addr=0x8 -device vfio-pci,host=86:01.1,id=hostdev3,bus=pci.0,addr=0x9 -device vfio-pci,host=84:00.0,id=hostdev4,bus=pci.0,addr=0xa -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xb -msg timestamp=on<br />
gpu-hypervisor:~$ grep AnonHuge /proc/meminfo<br />
AnonHugePages: 246108160 kB</nowiki><br />
<br />
I don't have an Arch system to verify this on, but I don't expect it to be distro specific. Any objection to removing this?<br />
<br />
:Have you tested memory performance inside the VM? As in, statically allocated hugepages vs transparent, with VFIO devices bound and actively used inside the VM? [[User:DragoonAethis|DragoonAethis]] ([[User talk:DragoonAethis|talk]]) 11:18, 13 April 2017 (UTC)<br />
<br />
== Using identical guest and host GPUs - did not work for me. ==<br />
<br />
I had to create the script vfio-pci-override.sh in bin, not sbin, or it would not find the file. (and update the modprobe and mkinitcpio accordingly)<br />
<br />
furthermore, I had to change the sys/devices path to search a directory deeper. The script /bin/vfio-pci-override.sh looks like this then:<br />
<br />
#!/bin/sh<br />
<br />
for i in /sys/devices/pci*/*/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
<br />
<br />
This situation is the only one that worked for me.<br />
[[User:Eggz|Eggz]] ([[User talk:Eggz|talk]]) 19:12, 14 April 2017 (UTC)<br />
<br />
== Passthrough via OVMF without libvirt ==<br />
<br />
I don't have a graphic card with UEFI fw to test this but it should work (with the ovmf package and not ovmf-git), please test and add it to the page if it does work.<br />
<br />
qemu-system-x86_64 \<br />
-enable-kvm \<br />
-m 8192 \<br />
-M q35 \<br />
-cpu host \<br />
-smp 4,sockets=1,cores=4,threads=1 \<br />
-vga none \<br />
-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/ovmf_code_x64.bin \<br />
-drive if=pflash,format=raw,file=/tmp/ovmf/qemu-vm-01/ovmf_vars_x64.bin \<br />
-device vfio-pci,host=05:00.0,addr=09.0,multifunction=on,x-vga=on \<br />
-device vfio-pci,host=05:00.1,addr=09.1 \<br />
-drive file=diskimg.qcow2,format=qcow2<br />
[[User:Dhead|Dhead]] ([[User talk:Dhead|talk]]) 15:58, 8 May 2017 (UTC)<br />
<br />
: And you should copy the vars first<br />
mkdir -p /tmp/ovmf/qemu_vm-01<br />
cp /usr/share/ovmf/ovmf_vars_x64.bin /tmp/ovmf/qemu_vm-01/ovmf_vars_x64.bin<br />
: [[User:Dhead|Dhead]] ([[User talk:Dhead|talk]]) 16:08, 8 May 2017 (UTC)<br />
<br />
: Oh, indeed the official ovmf package seems to be updated after a long while to include the variable files, at least I was told they were missing from it before. I'll test this out and do a writeup about it as I'm actually writing a draft for adding the scripted method back to the artictle, check here: [[User:Nd43/Scripted QEMU setup without libvirt]]. It's still incomplete but already it's also getting pretty big and detailed, any comments welcomed. Also I'm wondering if it should be a subpage or not when I finally start merging it back to this actual article. [[User:Nd43|Nd43]] ([[User talk:Nd43|talk]]) 08:19, 9 May 2017 (UTC)<br />
<br />
:: Nice guide but I don't think the ArchWiki is the right place for it, it's too specific and too detailed to be incorporated into this article and adding it as a subpage will end with duplicated content which will never be in sync.<br>The starting point of this article is that the user (an Arch user, not a copy-paste monkey) already gone through the QEMU article so everything not directly related to pci passing like systemd service, audio, putting all in a script should not be added to this article. So instead we should just add the very basic command switches related to using OVMF.<br><br>I might be a little digressing, but I also think this page should be torn down and reworked. All the performance tuning should be in its own subpage of the main QEMU article. Using OVMF instead of the default SeaBIOS also should be in a subpage (of QEMU). So we will be left with just the PCI passthrough stuff here. [[User:Dhead|Dhead]] ([[User talk:Dhead|talk]])<br />
<br />
::: How about mentioning just the passthrough switches for QEMU here? The <code>-device vfio-pci,host=...</code> ones, even OVMF isn't *required* for the passthrough to work (although it is easier to work with). Other than that, passthrough can be thrown in for just about any QEMU VM, and it's up to the guest OS to handle these devices somehow. Other QEMU args are specific to that VM. [[User:DragoonAethis|DragoonAethis]] ([[User talk:DragoonAethis|talk]]) 10:09, 9 May 2017 (UTC)<br />
<br />
::::I agree, just adding the related QEMU command switches should be more than enough. p.s. OVMF might be preferred for VGA PCI passthrough as at least on my system SeaBIOS seems to have some issues with USB input devices on boot/in bootloader (not an issue when the OS/kernel is running).[[User:Dhead|Dhead]] ([[User talk:Dhead|talk]]) 12:05, 9 May 2017 (UTC)<br />
<br />
::: You're probably right, but I still think many details of my writeup should be easily available for Arch users looking just to deploy a QEMU VM with PCI passthrough. I also agree that the whole article needs a rewrite on some parts but I really don't want to see the actually relevant information get removed temporarily or buried behind too many internal links to other articles. I've already seen how the article has suffered during the few years and currently its structure is partly very unclear. Does anyone have additional suggestions for the information to be added or should I just start editing with small additions to see how they're received? [[User:Nd43|Nd43]] ([[User talk:Nd43|talk]]) 18:44, 9 May 2017 (UTC)<br />
<br />
:::: Passing through the GPUs to a ready VM is just about adding those two switches and making sure you have some "extras" in if you're using a specific setup (GeForces requiring the spoofed vendor ID, etc). It might make more sense to let people configure their VM first (= installing OS under UEFI, installing VirtIO drivers, configuring their base environment etc) and THEN explain how to pass through the GPU to that ready VM - this might fit pretty well in this article.<br />
<br />
:::: Most of these copy-n-paste guides assume a very specific setup (memory, CPU cores, disks, etc), and there is value in those, since you can peek at complete solutions that are known to work under specific hardware combinations. However I'd rather create a separate VFIO "examples" page, where a brief description of that user's hardware, software (kernel + cmdline, additional VM configuration steps, etc) and the QEMU script/libvirt domain file would be posted. Exact specifics (additional scripts, configuration files for Synergy, PulseAudio and such) would have to be posted on GitHub/GitLab/somewhere else as to not clutter the wiki beyond reason (older examples would have to be removed or updated over time, since QEMU/libvirt break things once in a while). Does that sound fine? [[User:DragoonAethis|DragoonAethis]] ([[User talk:DragoonAethis|talk]]) 21:55, 10 May 2017 (UTC)<br />
<br />
::::: I like your thinking. I made an initial revision edit at [[PCI passthrough via OVMF#Plain QEMU without libvirt]] describing the bare minimum for achieving a practical QEMU setup with links to other relevant articles. I'm probably going to continue editing the section later as I want to pay attention to some details and terminology a bit more, but for now I think it will do for provoking people to make additions/changes to reach better guidance on the topic. So, comments and edits highly welcomed. I also removed the old links for the time being, even though I really liked Melvin's script examples, they indeed belong to the "See also" section or somewhere else totally. Feel free to relocate the links to a logical order. [[User:Nd43|Nd43]] ([[User talk:Nd43|talk]]) 15:32, 13 May 2017 (UTC)<br />
<br />
:::::: I've created the [[PCI_passthrough_via_OVMF/Examples]] page, feel free to contribute your working setups there. There's a template etc available to make adding new entries easier and more consistent. [[User:DragoonAethis|DragoonAethis]] ([[User talk:DragoonAethis|talk]]) 14:37, 19 May 2017 (UTC)<br />
<br />
== Moving the full tutorial elsewhere? ==<br />
<br />
A number of people on this talk page have already mentioned how having a complete tutorial like this on the Arch Wiki makes some parts of the article somewhat redundant with other pages on the wiki, and that it would probably be better if the article were split into multiple parts and spread across a number of pages. At the same time, having a full tutorial like this here means it can't really cover other distros without breaking the contribution guidelines, which limits the article's reach somewhat.<br />
<br />
However, I know a lot of people (myself included) have been convinced to try Arch after seeing a number of quality articles on the official wiki, and from what I've seen elsewhere, this very article has been this to some people. I've seen people on [https://www.reddit.com/r/VFIO/ /r/vfio] and elsewhere use this article as their primary reference for people who want to setup their machine this way. That's somewhat what I was aiming for when I started reworking this article to try and adapt [http://vfio.blogspot.com AW's blog] into something that's less intimidating to read, and it's great to see it has managed to evolve into what it is now thanks to all the contributions that have happened since then.<br />
<br />
I'd like to know, according to more experienced Arch Wiki contributors than me, whether or not this article actually belongs here, or if the current page shouldn't be moved to the VFIO subreddit and the Arch page torn down and its content split on other QEMU-related pages.<br />
<br />
[[User:Victoire|Victoire]] ([[User talk:Victoire|talk]]) 14:56, 10 July 2017 (UTC)<br />
<br />
:IMO this article is fine here - it's widely linked, kept mostly up-to-date, clearly explained and doesn't require two days of research to even attempt this setup. I would agree making an exception and allowing other distro-specific information to be posted here would be great, but moving the entire post to /r/vfio, where the wiki pages don't work on mobile and limit contributors to the subreddit's moderators or manually added people will limit the article's reach even more and cause it to go out-of-date in a few months. (If the wiki is open to all registered redditors, most more-popular wiki pages *will* be defaced.) [[User:DragoonAethis|DragoonAethis]] ([[User talk:DragoonAethis|talk]]) 12:24, 11 July 2017 (UTC)<br />
<br />
:Victoire, this is the Arch Wiki, users of other distros will be better served with full tutorials elsewhere, /r/VFIO has a wiki.<br>Regarding your choice of words, the page shouldn't be moved anywhere, copied somewhere else, yes, there you could just have a simple full tutorial from start to finish to set a host and guest QEMU machines with GPU passthrough, but the contents of this page should stay here so Arch Wiki users and maintainers could change it as needed, which I gather it's what you meant by "moved". In fact, from the little I've been seeing in /r/VFIO there are nice examples and tutorials that are buried and not being mentioned and linked in the /r/VFIO wiki.<br>Regards the needed updates to this page, first start, I would say the Performance tuning needs to go into the Tips and tricks in the QEMU page (which deserve its own subpage), the article name should be changed to PCI passthrough so it would reflect the actual content of the page which isn't UEFI specific.<br>Also, the complete examples page seems superfluous, why not have a wiki page in /r/VFIO with known compatible MB, or MB+GPU setups as all rest of the devices (recent CPU, RAM, storage) shouldn't matter, I wouldn't be surprised if the moderators would remove it.[[User:Dhead|Dhead]] ([[User talk:Dhead|talk]]) 14:05, 11 July 2017 (UTC)<br />
<br />
== Set firmware to UEFI ==<br />
<br />
>"In the "Overview" section, set your firmware to "UEFI". If the option is grayed out, make sure that you have correctly specified the location of your firmware in /etc/libvirt/qemu.conf and restart libvirtd.service."<br />
<br />
I see the Overview section, but I do not see "firmware" or "UEFI" anywhere on that section. There is nothing "grayed out". I only see:<br />
*Name<br />
*UUID<br />
*Status<br />
*Title<br />
*Description<br />
*Hypervisor<br />
*Architecture<br />
*Emulator<br />
*Chipset (i440FX or Q35 are the options)<br />
<br />
This makes me think the information could be outdated. I am not able to proceed and I suspect this to be the cause. [[User:Henstepl|Henstepl]] ([[User talk:Henstepl|talk]]) 19:57, 3 August 2017 (UTC)<br />
<br />
:This can be set only during the VM creation - check "Customize installation" on the final creation wizard screen and [https://i.imgur.com/73r2ctM.png the relevant option will show up]. If you can't select this during the VM creation, install `ovmf-git` or `ovmf`. [[User:DragoonAethis|DragoonAethis]] ([[User talk:DragoonAethis|talk]]) 20:05, 3 August 2017 (UTC)<br />
::Yes, I am looking at exactly that, but there is no "Firmware" option at all. It just skips directly from "Emulator" to "Chipset". Nothing greyed out. Simply not there. [[User:Henstepl|Henstepl]] ([[User talk:Henstepl|talk]]) 23:07, 3 August 2017 (UTC)<br />
:::Are you sure you've got <code>ovmf</code> or <code>ovmf-git</code> installed? If you've built it yourself, make sure <code>/etc/libvirt/qemu.conf</code> contains the proper OVMF binary path: <code>nvram = [ "/usr/share/ovmf/x64/ovmf_x64.bin:/usr/share/ovmf/x64/ovmf_vars_x64.bin" ]</code> - there should be a section like this in that file, but commented and possibly with different paths. Point it at your OVMF binaries (<code>pacman -Ql ovmf or ovmf-git</code> will show all files in that package). [[User:DragoonAethis|DragoonAethis]] ([[User talk:DragoonAethis|talk]]) 10:20, 4 August 2017 (UTC)<br />
::::Yes, ovmf has been installed from the start. <code>nvram = ["/usr/share/ovmf/ovmf_code_x64.bin:/usr/share/ovmf/ovmf_vars_x64.bin"]</code>, as contains the paths listed by <code>pacman -Ql ovfm</code>, has been in <code>/etc/libvirt/qemu.conf</code>. Thanks for your advice but it hasn't elucidated anything I've done wrong, and there still is no Firmware option. [[User:Henstepl|Henstepl]] ([[User talk:Henstepl|talk]]) 04:38, 5 August 2017 (UTC)<br />
:::::Okay, not sure why this option doesn't show up, but there's also a way to manually manipulate the domain XML file to use OVMF. [https://github.com/DragoonAethis/VFIO/blob/master/Windows8.1.xml#L18-L23 Here are the relevant lines], once you've created the VM, open your terminal and enter <code>sudo EDITOR=your-editor-like-nano-or-vim-or-somethign virsh edit your-domain-name</code> and make the <os> section there contain the <loader> and <nvram> entries pointing to proper OVMF files. If you're missing the vars file, [https://stuff.dragonic.eu/Windows8.1_VARS.fd here it is] from my VM. [[User:DragoonAethis|DragoonAethis]] ([[User talk:DragoonAethis|talk]]) 11:39, 5 August 2017 (UTC)<br />
::::::I was hoping that a solution to this would help me avoid the error message I have been getting when I try to start my VM:<pre>Unable to complete install: 'internal error: process exited while connecting to monitor: 2017-08-03T03:20:54.793536Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/2 (label charserial0)<br />
Could not access KVM kernel module: Permission denied<br />
failed to initialize KVM: Permission denied'<br />
<br />
Traceback (most recent call last):<br />
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 88, in cb_wrapper<br />
callback(asyncjob, *args, **kwargs)<br />
File "/usr/share/virt-manager/virtManager/create.py", line 2288, in _do_async_install<br />
guest.start_install(meter=meter)<br />
File "/usr/share/virt-manager/virtinst/guest.py", line 477, in start_install<br />
doboot, transient)<br />
File "/usr/share/virt-manager/virtinst/guest.py", line 405, in _create_guest<br />
self.domain.create()<br />
File "/usr/lib/python2.7/site-packages/libvirt.py", line 1062, in create<br />
if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)<br />
libvirtError: internal error: process exited while connecting to monitor: 2017-08-03T03:20:54.793536Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/2 (label charserial0)<br />
Could not access KVM kernel module: Permission denied<br />
failed to initialize KVM: Permission denied</pre><br />
::::::Which prevents me from creating a VM with those settings... Welp. Maybe I will just reinstall everything from the start. [[User:Henstepl|Henstepl]] ([[User talk:Henstepl|talk]]) 05:48, 7 August 2017 (UTC)<br />
<br />
I had the same issue, fixed it by forcing the kvm gid to 78, an update of systemd give kvm group a dynamic id while libvirtd expect to be 78. <code>groupmod -g 78 kvm</code> and a reboot fixed it (https://bbs.archlinux.org/viewtopic.php?id=228936) [[User:Makz|Makz]] ([[User talk:Makz|talk]]) 22:15, 9 September 2017 (UTC)<br />
<br />
== Boot GPU Passthrough ==<br />
<br />
I managed to pass the boot gpu (the only one in my system) by:<br />
* setting the GRUB option {{ic|<nowiki>GRUB_GFXPAYLOAD_LINUX=text</nowiki>}} (for example in /etc/default/grub)<br />
* and adding the kernel parameter {{ic|<nowiki>earlymodules=vfio-pci</nowiki>}} (although I'm not sure if that is necessary at all)<br />
<br />
That prevents the initramfs from initializing a graphical console on the boot gpu, allowing the VM to take over it directly after GRUB. However, you can't see your boot status and will have to type your LUKS password blind if you encrypted your root partition.<br />
<br />
The Hardware I used: AMD Ryzen 7 1700 CPU, Mainboard: Biostar X370GTN, Passthrough GPU: Nvidia GeForce GTX 970<br />
<br />
-- [[User:Dasisdormax|dasisdormax]] ([[User talk:Dasisdormax|talk]]) 11:33, 8 August 2017 (UTC)<br />
<br />
== Slow boot linked to the RAM amount ==<br />
<br />
> Windows 10 VM with hugepages, 16Gb ram = 1m40 from the start to login screen, same VM reduced to 1Gb ram = 10 seconds from start to login screen (i7-6900K + 64Gb DDR4-2133 non-ECC) brand new installation<br />
<br />
Same setup on a bi-Xeon + 64Gb DDR3-1600 ECC ram, 16Gb ram = 13 seconds, 1Gb = 13 seconds<br />
<br />
While the VM is booting, all assigned CPU are working at 100%<br />
<br />
Is the ECC ram help to boot faster ? Is it a OVMF related issue ? Maybe the virtual bios is checking the ram before starting the OS ?<br />
<br />
While it's loading, i have a black screen, i don't think the GPU is handled by the VM at this moment.<br />
<br />
I saw some users on forums having this issue and tried latest ovmf build but no fix so far.<br />
<br />
[[User:Makz|Makz]] ([[User talk:Makz|talk]]) 09:10, 10 September 2017 (UTC)<br />
<br />
== Question about virtlogd and iommu=pt ==<br />
<br />
Is enabling virtlogd.socket optional? If so, that should be stated.<br />
<br />
Is iommu=pt only for AMD? It looks like that on this page https://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM<br />
<br />
Their instructions says:<br />
<br />
intel_iommu=on # Intel only<br />
<br />
iommu=pt iommu=1 # AMD only<br />
<br />
{{Unsigned|19:59, 16 January 2018|Limero}}<br />
<br />
:virtlogd.socket must be enabled, libvirt will otherwise fail to start. iommu=pt is separate from the intel_iommu=on and amd_iommu=on - iommu=pt enables the IOMMU hardware (on any platform) only for the passthrough devices and lets all the other devices (not passed to the VMs) do whatever they want. The "normal" IOMMU mode is to isolate all devices and make sure all devices behave nicely (no writes out of their address space, DMA only to their assigned section of memory, etc), but this may cause problems on some older platforms. [[User:DragoonAethis|DragoonAethis]] ([[User talk:DragoonAethis|talk]]) 20:27, 16 January 2018 (UTC)</div>NeoTheFoxhttps://wiki.archlinux.org/index.php?title=PCI_passthrough_via_OVMF&diff=508239PCI passthrough via OVMF2018-01-22T12:18:58Z<p>NeoTheFox: /* UEFI (OVMF) Compatability in VBIOS */ Added GOPUpd link</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:OVMF による PCI パススルー]]<br />
The Open Virtual Machine Firmware ([https://github.com/tianocore/tianocore.github.io/wiki/OVMF OVMF]) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the VM native graphics performance which is useful for graphic-intensive tasks.<br />
<br />
Provided you have a desktop computer with a spare GPU you can dedicate to the host (be it an integrated GPU or an old OEM card, the brands do not even need to match) and that your hardware supports it (see [[#Prerequisites]]), it is possible to have a VM of any OS with its own dedicated GPU and near-native performance. For more information on techniques see the background [https://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf presentation (pdf)]. <br />
<br />
== Prerequisites ==<br />
<br />
A VGA Passthrough relies on a number of technologies that are not ubiquitous as of today and might not be available on your hardware. You will not be able to do this on your machine unless the following requirements are met :<br />
<br />
* Your CPU must support hardware virtualization (for kvm) and IOMMU (for the passthrough itself)<br />
** [https://ark.intel.com/Search/FeatureFilter?productType=processors&VTD=true List of compatible Intel CPUs (Intel VT-x and Intel VT-d)]<br />
** All AMD CPUs from the Bulldozer generation and up (including Zen) should be compatible.<br />
*** CPUs from the K10 generation (2007) do not have an IOMMU, so you '''need''' to have a motherboard with a [https://support.amd.com/TechDocs/43403.pdf#page=18 890FX] or [https://support.amd.com/TechDocs/48691.pdf#page=21 990FX] chipset to make it work, as those have their own IOMMU.<br />
* Your motherboard must also support IOMMU<br />
** Both the chipset and the BIOS must support it. It is not always easy to tell at a glance whether or not this is the case, but there is a fairly comprehensive list on the matter on the [https://wiki.xen.org/wiki/VTd_HowTo Xen wiki] as well as [[Wikipedia:List of IOMMU-supporting hardware]].<br />
* Your guest GPU ROM must support UEFI.<br />
** If you can find [https://www.techpowerup.com/vgabios/ any ROM in this list] that applies to your specific GPU and is said to support UEFI, you are generally in the clear. All GPUs from 2012 and later should support this, as Microsoft made UEFI a requirement for devices to be marketed as compatible with Windows 8.<br />
<br />
You will probably want to have a spare monitor or one with multiple input ports connected to different GPUs (the passthrough GPU will not display anything if there is no screen plugged in and using a VNC or Spice connection will not help your performance), as well as a mouse and a keyboard you can pass to your VM. If anything goes wrong, you will at least have a way to control your host machine this way.<br />
<br />
== Setting up IOMMU ==<br />
<br />
IOMMU is a system specific IO mapping mechanism and can be used with most devices. IOMMU is a generic name for Intel VT-d and AMD-Vi.<br />
<br />
Before you enable IOMMU, you might have to first enable (non-IOMMU) virtualisation (Intel VT-x/"Vanderpool" or AMD-V/"Pacifica") in your BIOS settings.<br />
<br />
=== Enabling IOMMU ===<br />
<br />
Ensure that AMD-Vi/Intel VT-d is enabled in your BIOS settings. Both normally show up alongside other CPU features (meaning they could be in an overclocking-related menu) either with their actual names ("VT-d" or "AMD-Vi") or in more ambiguous terms such as "Virtualization technology", which may or may not be explained in the manual.<br />
<br />
You will also have to enable iommu support in the kernel itself through a [[Kernel parameters|bootloader kernel option]]. Depending on your type of CPU, use either {{ic|1=intel_iommu=on}} for Intel CPUs (VT-d) or {{ic|1=amd_iommu=on}} for AMD CPUs (AMD-Vi).<br />
<br />
You should also append the {{ic|1=iommu=pt}} parameter. This will prevent Linux from touching devices which cannot be passed through.<br />
<br />
After rebooting, check dmesg to confirm that IOMMU has been correctly enabled:<br />
<br />
{{hc|dmesg {{!}} grep -e DMAR -e IOMMU|<br />
[ 0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001)<br />
[ 0.000000] Intel-IOMMU: enabled<br />
[ 0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a<br />
[ 0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da<br />
[ 0.028950] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1<br />
[ 0.536212] DMAR: No ATSR found<br />
[ 0.536229] IOMMU 0 0xfed90000: using Queued invalidation<br />
[ 0.536230] IOMMU 1 0xfed91000: using Queued invalidation<br />
[ 0.536231] IOMMU: Setting RMRR:<br />
[ 0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]<br />
[ 0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]<br />
[ 0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC<br />
[ 0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]<br />
[ 2.182790] [drm] DMAR active, disabling use of stolen memory<br />
}}<br />
<br />
=== Ensuring that the groups are valid ===<br />
<br />
The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.<br />
<br />
#!/bin/bash<br />
shopt -s nullglob<br />
for d in /sys/kernel/iommu_groups/*/devices/*; do <br />
n=${d#*/iommu_groups/*}; n=${n%%/*}<br />
printf 'IOMMU Group %s ' "$n"<br />
lspci -nns "${d##*/}"<br />
done;<br />
<br />
Example output:<br />
<br />
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0104] (rev 09)<br />
IOMMU Group 1 00:16.0 Communication controller [0780]: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 [8086:1c3a] (rev 04)<br />
IOMMU Group 2 00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 04)<br />
IOMMU Group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev <br />
...<br />
<br />
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that [[#USB controller|any of them could be passed to a VM without affecting the others]].<br />
<br />
=== Gotchas ===<br />
<br />
==== Plugging your guest GPU in an unisolated CPU-based PCIe slot ====<br />
<br />
Not all PCI-E slots are the same. Most motherboards have PCIe slots provided by both the CPU and the PCH. Depending on your CPU, it is possible that your processor-based PCIe slot does not support isolation properly, in which case the PCI slot itself will be appear to be grouped with the device that is connected to it.<br />
<br />
IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)<br />
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)<br />
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)<br />
<br />
This is fine so long as only your guest GPU is included in here, such as above. Depending on what is plugged in your other PCIe slots and whether they are allocated to your CPU or your PCH, you may find yourself with additional devices within the same group, which would force you to pass those as well. If you are ok with passing everything that is in there to your VM, you are free to continue. Otherwise, you will either need to try and plug your GPU in your other PCIe slots (if you have any) and see if those provide isolation from the rest or to install the ACS override patch, which comes with its own drawbacks. See [[#Bypassing the IOMMU groups (ACS override patch)]] for more information.<br />
<br />
{{Note|If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.}}<br />
<br />
== Isolating the GPU ==<br />
<br />
In order to assign a device to a virtual machine, this device and all those sharing the same IOMMU group must have their driver replaced by a stub driver or a VFIO driver in order to prevent the host machine from interacting with them. In the case of most devices, this can be done on the fly right before the VM starts.<br />
<br />
However, due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a VM without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the VM, in order to stop other drivers from attempting to claim it.<br />
<br />
The following section details how to configure a GPU so those placeholder drivers are bound early during the boot process, which makes said device inactive until a VM claims it or the driver is switched back. This is the prefered method, considering it has less caveats than switching drivers once the system is fully online.<br />
<br />
{{warning|Once you reboot after this procedure, whatever GPU you have configured will no longer be usable on the host until you reverse the manipulation. Make sure the GPU you intend to use on the host is properly configured before doing this.}}<br />
<br />
=== Using vfio-pci ===<br />
<br />
Starting with Linux 4.1, the kernel includes vfio-pci. This is a VFIO driver, meaning it fulfills the same role as pci-stub, but it can also control devices to an extent, such as by switching them into their D3 state when they are not in use. If your system supports it, which you can try by running the following command, you should use it. If it returns an error, use pci-stub instead.<br />
<br />
{{hc|$ modinfo vfio-pci|<br />
filename: /lib/modules/4.4.5-1-ARCH/kernel/drivers/vfio/pci/vfio-pci.ko.gz<br />
description: VFIO PCI - User Level meta-driver<br />
author: Alex Williamson <alex.williamson@redhat.com><br />
...<br />
}}<br />
<br />
Vfio-pci normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
You can then add those vendor-device ID pairs to the default parameters passed to vfio-pci whenever it is inserted into the kernel.<br />
<br />
{{hc|/etc/modprobe.d/vfio.conf|2=<br />
options vfio-pci ids=10de:13c2,10de:0fbb<br />
}}<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|vfio-pci}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|vfio-pci}} to bind with.}}<br />
<br />
This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, {{ic|vfio}}, {{ic|vfio_iommu_type1}}, {{ic|vfio_pci}} and {{ic|vfio_virqfd}} to [[mkinitcpio]]:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)<br />
}}<br />
<br />
{{Note|If you also have another driver loaded this way for [[Kernel mode setting#Early KMS start|early modesetting]] (such as {{ic|nouveau}}, {{ic|radeon}}, {{ic|amdgpu}}, {{ic|i915}}, etc.), all of the aforementioned VFIO modules must precede it.}}<br />
<br />
Also, ensure that the modconf hook is included in the HOOKS list of {{ic|mkinitcpio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
HOOKS=(... modconf ...)<br />
}}<br />
<br />
Since new modules have been added to the initramfs configuration, you must [[regenerate the initramfs]]. Should you change the IDs of the devices in {{ic|/etc/modprobe.d/vfio.conf}}, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.<br />
<br />
Reboot and verify that vfio-pci has loaded properly and that it is now bound to the right devices.<br />
<br />
{{hc|$ dmesg {{!}} grep -i vfio|<br />
[ 0.329224] VFIO - User Level meta-driver version: 0.3<br />
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000<br />
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000<br />
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)<br />
}}<br />
<br />
It is not necessary for all devices (or even expected device) from {{ic|vfio.conf}} to be in dmesg output. Sometimes a device does not appear in output at boot but actually is able to be visible and operatable in guest VM.<br />
<br />
{{hc|$ lspci -nnk -d 10de:13c2|<br />
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: nouveau nvidia<br />
}}<br />
<br />
{{hc|$ lspci -nnk -d 10de:0fbb|<br />
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
Kernel driver in use: vfio-pci<br />
Kernel modules: snd_hda_intel<br />
}}<br />
<br />
=== Using pci-stub (legacy method, pre-4.1 kernels) ===<br />
<br />
If your kernel does not support vfio-pci, you can use the pci-stub module instead, which is a dummy driver used to claim a device and prevent other drivers from using it.<br />
<br />
Pci-stub normally targets PCI devices by ID, meaning you only need to specify the IDs of the devices you intend to passthrough. For the following IOMMU group, you would want to bind vfio-pci with {{ic|10de:13c2}} and {{ic|10de:0fbb}}, which will be used as example values for the rest of this section.<br />
<br />
IOMMU group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
IOMMU group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}<br />
<br />
{{Note|You cannot specify which device to isolate using vendor-device ID pairs if the host GPU and the guest GPU share the same pair (i.e : if both are the same model). If this is your case, read [[#Using identical guest and host GPUs]] instead.}}<br />
<br />
{{Pkg|linux}} and {{Pkg|linux-lts}} have pci-stub built as a module. You will need to add it to MODULES array in {{ic|mkinitpcio.conf}}:<br />
<br />
{{hc|/etc/mkinitcpio.conf|2=<br />
MODULES=(... pci-stub ...)<br />
}}<br />
<br />
If you did need to add this module to your kernel image configuration manually, you must also [[regenerate the initramfs]].<br />
<br />
Add the relevant PCI device IDs to the {{ic|pci-stubs.ids}} [[kernel parameter]], e.g. {{ic|1=pci-stub.ids=10de:13c2,10de:0fbb}}.<br />
<br />
{{Note|If, as noted in [[#Plugging your guest GPU in an unisolated CPU-based PCIe slot]], your pci root port is part of your IOMMU group, you '''must not''' pass its ID to {{ic|pci-stub}}, as it needs to remain attached to the host to function properly. Any other device within that group, however, should be left for {{ic|pci-stub}} to bind with.}}<br />
<br />
Check dmesg output for successful assignment of the device to pci-stub:<br />
<br />
{{hc|dmesg {{!}} grep pci-stub|<br />
[ 2.390128] pci-stub: add 10DE:13C2 sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390143] pci-stub 0000:06:00.0: claimed by stub<br />
[ 2.390150] pci-stub: add 10DE:0FBB sub<nowiki>=</nowiki>FFFFFFFF:FFFFFFFF cls<nowiki>=</nowiki>00000000/00000000<br />
[ 2.390159] pci-stub 0000:06:00.1: claimed by stub<br />
}}<br />
<br />
== Setting up an OVMF-based guest VM ==<br />
<br />
OVMF is an open-source UEFI firmware for QEMU virtual machines. While it's possible to use SeaBIOS to get similar results to an actual PCI passthough, the setup process is different and it is generally preferable to use the EFI method if your hardware supports it.<br />
<br />
=== Configuring libvirt ===<br />
<br />
[[Libvirt]] is a wrapper for a number of virtualization utilities that greatly simplifies the configuration and deployment process of virtual machines. In the case of KVM and QEMU, the frontend it provides allows us to avoid dealing with the permissions for QEMU and make it easier to add and remove various devices on a live VM. Its status as a wrapper, however, means that it might not always support all of the latest qemu features, which could end up requiring the use of a wrapper script to provide some extra arguments to QEMU.<br />
<br />
After installing {{Pkg|qemu}}, {{Pkg|libvirt}}, {{Pkg|ovmf}} and {{Pkg|virt-manager}}, add the path to your OVMF firmware image and runtime variables template to your libvirt config so {{ic|virt-install}} or {{ic|virt-manager}} can find those later on.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
nvram = [<br />
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"<br />
]<br />
}}<br />
<br />
You can now [[enable]] and [[start]] {{ic|libvirtd.service}} and its logging component {{ic|virtlogd.socket}}.<br />
<br />
=== Setting up the guest OS ===<br />
<br />
The process of setting up a VM using {{ic|virt-manager}} is mostly self explainatory, as most of the process comes with fairly comprehensive on-screen instructions.<br />
<br />
If using {{ic|virt-manager}}, you have to add your user to the libvirt [[group]] to ensure authentication.<br />
<br />
However, you should pay special attention to the following steps :<br />
<br />
* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.<br />
* In the "Overview" section, [https://i.imgur.com/73r2ctM.png set your firmware to "UEFI"]. If the option is grayed out, make sure that you have correctly specified the location of your firmware in {{ic|/etc/libvirt/qemu.conf}} and restart {{ic|libvirtd.service}}.<br />
* In the "CPUs" section, change your CPU model to "host-passthrough". If it is not in the list, you will have to type it by hand. This will ensure that your CPU is detected properly, since it causes libvirt to expose your CPU capabilities exactly as they are instead of only those it recognizes (which is the preferred default behavior to make CPU behavior easier to reproduce). Without it, some applications may complain about your CPU being of an unknown model.<br />
* If you want to minimize IO overhead, go into "Add Hardware" and add a Controller for SCSI drives of the "VirtIO SCSI" model. You can then change the default IDE disk for a SCSI disk, which will bind to said controller.<br />
** Windows VMs will not recognize those drives by default, so you need to download the ISO containing the drivers from [https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ here] and add an IDE (or SATA for Windows 8.1 and newer) CD-ROM storage device linking to said ISO, otherwise you will not be able to get Windows to recognize it during the installation process. When prompted to select a disk to install windows on, load the drivers contained on the CD-ROM under ''vioscsi''.<br />
<br />
The rest of the installation process will take place as normal using a standard QXL video adapter running in a window. At this point, there is no need to install additional drivers for the rest of the virtual devices, since most of them will be removed later on. Once the guest OS is done installing, simply turn off the virtual machine.<br />
<br />
=== Attaching the PCI devices ===<br />
<br />
With the installation done, it's now possible to edit the hardware details in libvirt and remove virtual integration devices, such as the spice channel and virtual display, the QXL video adapter, the emulated mouse and keyboard and the USB tablet device. Since that leaves you with no input devices, you may want to bind a few USB host devices to your VM as well, but remember to '''leave at least one mouse and/or keyboard assigned to your host''' in case something goes wrong with the guest. At this point, it also becomes possible to attach the PCI device that was isolated earlier; simply click on "Add Hardware" and select the PCI Host Devices you want to passthrough. If everything went well, the screen plugged into your GPU should show the OVMF splash screen and your VM should start up normally. From there, you can setup the drivers for the rest of your VM.<br />
<br />
=== Gotchas ===<br />
<br />
==== Using a non-EFI image on an OVMF-based VM ====<br />
<br />
The OVMF firmware does not support booting off non-EFI mediums. If the installation process drops you in a UEFI shell right after booting, you may have an invalid EFI boot media. Try using an alternate linux/windows image to determine if you have an invalid media.<br />
<br />
== Performance tuning ==<br />
<br />
Most use cases for PCI passthroughs relate to performance-intensive domains such as video games and GPU-accelerated tasks. While a PCI passthrough on its own is a step towards reaching native performance, there are still a few ajustments on the host and guest to get the most out of your VM.<br />
<br />
=== CPU pinning ===<br />
<br />
The default behavior for KVM guests is to run operations coming from the guest as a number of threads representing virtual processors. Those threads are managed by the Linux scheduler like any other thread and are dispatched to any available CPU cores based on niceness and priority queues. Since switching between threads adds a bit of overhead (because context switching forces the core to change its cache between operations), this can noticeably harm performance on the guest. CPU pinning aims to resolve this as it overrides process scheduling and ensures that the VM threads will always run and only run on those specific cores. Here, for instance, the guest cores 0, 1, 2 and 3 are mapped to the host cores 4, 5, 6 and 7 respectively.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
</nowiki>}}<br />
<br />
==== The case of Hyper-threading ====<br />
<br />
If your CPU supports hardware multitasking, also known as Hyper-threading on Intel chips, there are two ways you can go with your CPU pinning. That is, Hyper-threading is simply a very efficient way of running two threads on one CPU at any given time, so while it may give you 8 logical cores on what would otherwise be a quad-core CPU, if the physical core is overloaded, the logical core will not be of any use. One could pin their VM threads on 2 physical cores and their 2 respective threads, but any task overloading those two cores will not be helped by the extra two logical cores, since in the end you are only passing through two cores out of four, not four out of eight. What you should do knowing this depends on what you intend to do with your host while your VM is running.<br />
<br />
This is the abridged content of {{ic|/proc/cpuinfo}} on a quad-core machine with hyper-threading.<br />
<br />
{{hc|$ grep -e "processor" -e "core id" -e "^$" /proc/cpuinfo|<br />
processor : 0<br />
core id : 0<br />
<br />
processor : 1<br />
core id : 1<br />
<br />
processor : 2<br />
core id : 2<br />
<br />
processor : 3<br />
core id : 3<br />
<br />
processor : 4<br />
core id : 0<br />
<br />
processor : 5<br />
core id : 1<br />
<br />
processor : 6<br />
core id : 2<br />
<br />
processor : 7<br />
core id : 3<br />
}}<br />
<br />
If you do not intend to be doing any computation-heavy work on the host (or even anything at all) at the same time as you would on the VM, it would probably be better to pin your VM threads across all of your logical cores, so that the VM can fully take advantage of the spare CPU time on all your cores.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='4'/><br />
<vcpupin vcpu='1' cpuset='5'/><br />
<vcpupin vcpu='2' cpuset='6'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='4' threads='1'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
If you would instead prefer to have the host and guest running intensive tasks at the same time, it would then be preferable to pin a limited amount of physical cores and their respective threads on the guest and leave the rest to the host to avoid the two competing for CPU time.<br />
<br />
On the quad-core machine mentioned above, it would look like this :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<vcpu placement='static'>4</vcpu><br />
<cputune><br />
<vcpupin vcpu='0' cpuset='2'/><br />
<vcpupin vcpu='1' cpuset='6'/><br />
<vcpupin vcpu='2' cpuset='3'/><br />
<vcpupin vcpu='3' cpuset='7'/><br />
</cputune><br />
...<br />
<cpu mode='custom' match='exact'><br />
...<br />
<topology sockets='1' cores='2' threads='2'/><br />
...<br />
</cpu><br />
...<br />
</nowiki>}}<br />
<br />
=== Static huge pages ===<br />
<br />
When dealing with applications that require large amounts of memory, memory latency can become a problem since the more memory pages are being used, the more likely it is that this application will attempt to access information accross multiple memory "pages", which is the base unit for memory allocation. Resolving the actual address of the memory page takes multiple steps, and so CPUs normally cache information on recently used memory pages to make subsequent uses on the same pages faster. Applications using large amounts of memory run into a problem where, for instance, a virtual machine uses 4GB of memory divided into 4kB pages (which is the default size for normal pages), meaning that such cache misses can become extremely frequent and greatly increase memory latency. Huge pages exist to mitigate this issue by giving larger individual pages to those applications, increasing the odds that multiple operations will target the same page in succession. This is normally handeled with transparent huge pages, which dynamically manages hugepages to keep up with the demand.<br />
<br />
On a VM with a PCI passthrough, however, it is '''not possible''' to benefit from transparent huge pages, as IOMMU requires that the guest's memory be allocated and pinned as soon as the VM starts. It is therefore required to allocate huge pages statically in order to benefit from them. <br />
<br />
{{Warning|Static huge pages lock down the allocated amount of memory, making it unavailable for applications that are not configured to use them. Allocating 4GBs worth of huge pages on a machine with 8GBs of memory will only leave you with 4GBs of available memory on the host '''even when the VM is not running'''.}}<br />
<br />
To allocate huge pages at boot, one must simply specify the desired amount on their kernel comand line with {{ic|1=hugepages=''x''}}. For instance, reserving 1024 pages with {{ic|1=hugepages=1024}} and the default size of 2048kB per huge page creates 2GBs worth of memory for the virtual machine to use.<br />
<br />
If supported by CPU page size could be set manually. 1 GB huge page support could be verified by {{ic|<nowiki>grep pdpe1gb /proc/cpuinfo</nowiki>}}. Setting 1 GB huge page size via kernel parameters : {{ic|1=default_hugepagesz=1G hugepagesz=1G hugepages=X transparent_hugepage=never}}.<br />
<br />
Also, since static huge pages can only be used by applications that specifically request it, you must add this section in your libvirt domain configuration to allow kvm to benefit from them :<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<memoryBacking><br />
<hugepages/><br />
</memoryBacking><br />
...<br />
</nowiki>}}<br />
<br />
=== CPU frequency governor ===<br />
<br />
Depending on the way your [[CPU frequency scaling|CPU governor]] is configured, the VM threads may not hit the CPU load thresholds for the frequency to ramp up. Indeed, KVM cannot actually change the CPU frequency on its own, which can be a problem if it does not scale up with vCPU usage as it would result in underwhelming performance. An easy way to see if it behaves correctly is to check if the frequency reported by {{ic|watch lscpu}} goes up when running a CPU-intensive task on the guest. If you are indeed experiencing stutter and the frequency does not go up to reach its reported maximum, it may be due to [https://lime-technology.com/forum/index.php?topic=46664.msg447678#msg447678 cpu scaling being controlled by the host OS]. In this case, try setting all cores to maximum frequency to see if this improves performance. Note that if you are using a modern intel chip with the default pstate driver, cpupower commands will be [[CPU frequency scaling#CPU frequency driver|ineffective]], so monitor {{ic|/proc/cpuinfo}} to make sure your cpu is actually at max frequency.<br />
<br />
=== High DPC Latency ===<br />
<br />
{{Accuracy|As far as I can tell all virtio modules listed here are for virtual devices used when Linux runs as a guest. Loading them on the host serves no purpose.}}<br />
<br />
If you are experiencing high DPC and/or interrupt latency in your Guest VM, ensure you have [[Kernel modules#Manual module handling|loaded the needed virtio kernel modules]] on the host kernel. Loadable virtio kernel modules include: {{ic|virtio-pci}}, {{ic|virtio-net}}, {{ic|virtio-blk}}, {{ic|virtio-balloon}}, {{ic|virtio-ring}} and {{ic|virtio}}. <br />
<br />
After loading one or more of these modules, {{ic|lsmod {{!}} grep virtio}} executed on the host should not return empty.<br />
<br />
==== CPU pinning with isolcpus ====<br />
<br />
Alternatively, make sure that you have isolated CPUs properly. In this example, let us assume you are using CPUs 4-7.<br />
Use the kernel parameters {{ic|isolcpus nohz_full rcu_nocbs}} to completely isolate the CPUs from the kernel. <br />
<br />
{{hc|sudo vim /etc/defaults/grub|<nowiki><br />
...<br />
GRUB_CMDLINE_LINUX="..your other params.. isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"<br />
...<br />
</nowiki>}}<br />
<br />
Then, run {{ic|qemu-system-x86_64}} with taskset and chrt:<br />
<br />
# chrt -r 1 taskset -c 4-7 qemu-system-x86_64 ...<br />
<br />
The {{ic|chrt}} command will ensure that the task scheduler will round-robin distribute work (otherwise it will all stay on the first cpu). For {{ic|taskset}}, the CPU numbers can be comma- and/or dash-separated, like "0,1,2,3" or "0-4" or "1,7-8,10" etc.<br />
<br />
See [https://www.reddit.com/r/VFIO/comments/6vgtpx/high_dpc_latency_and_audio_stuttering_on_windows/dm0sfto/ this reddit comment] for more info.<br />
<br />
=== Improving performance on AMD CPUs ===<br />
<br />
Previously, Nested Page Tables (NPT) had to be disabled on AMD systems running KVM to improve GPU performance because of a [https://sourceforge.net/p/kvm/bugs/230/ very old bug], but the trade off was decreased CPU performance, including stuttering.<br />
<br />
There is a [https://patchwork.kernel.org/patch/10027525/ kernel patch] that resolves this issue, which was accepted into kernel 4.14-stable and 4.9-stable. If you're running the official {{Pkg|linux}} or {{Pkg|linux-lts}} kernel the patch has already been applied (make sure you're on the latest). If you're running another kernel you might need to manually patch yourself.<br />
<br />
{{Note|Several Ryzen users (see [https://www.reddit.com/r/VFIO/comments/78i3jx/possible_fix_for_the_npt_issue_discussed_on_iommu/ this Reddit thread]) have tested the patch, and can confirm that it works, bringing GPU passthrough performance up to near native quality.}}<br />
<br />
=== Further Tuning ===<br />
<br />
More specialized VM tuning tips are available at [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html Red Hat's Virtualization Tuning and Optimization Guide]. This guide may be especially helpful if you are experiencing:<br />
<br />
* Moderate CPU load on the host during downloads/uploads from within the guest: See ''Bridge Zero Copy Transmit'' for a potential fix.<br />
* Guests capping out at certain network speeds during downloads/uploads despite virtio-net being used: See ''Multi-queue virtio-net'' for a potential fix.<br />
* Guests "stuttering" under high I/O, despite the same workload not affecting hosts to the same degree: See ''Multi-queue virtio-scsi'' for a potential fix.<br />
<br />
== Special procedures ==<br />
<br />
Certain setups require specific configuration tweaks in order to work properly. If you are having problems getting your host or your VM to work properly, see if your system matches one of the cases below and try adjusting your configuration accordingly.<br />
<br />
=== Using identical guest and host GPUs ===<br />
<br />
{{Expansion|A number of users have been having issues with this, it should probably be adressed by the article.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
Due to how both pci-stub and vfio-pci use your vendor and device id pair to identify which device they need to bind to at boot, if you have two GPUs sharing such an ID pair you will not be able to get your passthough driver to bind with just one of them. This sort of setup makes it necessary to use a script, so that whichever driver you are using is instead assigned by pci bus address using the {{ic|driver_override}} mechanism.<br />
<br />
==== Script variants ====<br />
<br />
===== Passthrough all GPUs but the boot GPU =====<br />
<br />
Here, we will make a script to bind vfio-pci to all GPUs but the boot gpu. Create the script {{ic|/usr/bin/vfio-pci-override.sh}}:<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
for i in /sys/devices/pci*/*/boot_vga; do<br />
if [ $(cat "$i") -eq 0 ]; then<br />
GPU="${i%/boot_vga}"<br />
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"<br />
echo "vfio-pci" > "$GPU/driver_override"<br />
if [ -d "$AUDIO" ]; then<br />
echo "vfio-pci" > "$AUDIO/driver_override"<br />
fi<br />
fi<br />
done<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
===== Passthrough selected GPU =====<br />
<br />
In this case we manually specify the GPU to bind.<br />
<br />
{{bc|<nowiki><br />
#!/bin/sh<br />
<br />
GROUP="0000:00:03.0"<br />
DEVS="0000:03:00.0 0000:03:00.1 ."<br />
<br />
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then<br />
for DEV in $DEVS; do<br />
echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override<br />
done<br />
fi<br />
<br />
modprobe -i vfio-pci<br />
</nowiki>}}<br />
<br />
==== Script installation ====<br />
<br />
Create {{ic|/etc/modprobe.d/vfio.conf}} with the following:<br />
<br />
install vfio-pci /usr/bin/vfio-pci-override.sh<br />
<br />
Edit {{ic|/etc/mkinitcpio.conf}}<br />
<br />
Remove any video drivers from MODULES, and add {{ic|vfio-pci}}, and {{ic|vfio_iommu_type1}}<br />
<br />
MODULES=(ext4 vfat vfio-pci vfio_iommu_type1)<br />
<br />
Add {{ic|/etc/modprobe.d/vfio.conf}} and {{ic|/usr/bin/vfio-pci-override.sh}} to FILES:<br />
<br />
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)<br />
<br />
[[Regenerate the initramfs]] and reboot:<br />
<br />
=== Passing the boot GPU to the guest ===<br />
<br />
{{Expansion|Not possible at the time as far as I'm aware, but a common issue on various forums.|Talk:PCI passthrough via OVMF#Additionnal sections}}<br />
<br />
The GPU marked as {{ic|boot_vga}} is a special case when it comes to doing PCI passthroughs, since the BIOS needs to use it in order to display things like boot messages or the BIOS configuration menu. To do that, it makes [https://www.redhat.com/archives/vfio-users/2016-May/msg00224.html a copy of the VGA boot ROM which can then be freely modified]. This modified copy is the version the system gets to see, which the passthrough driver may reject as invalid. As such, it is generally recommended to change the boot GPU in the BIOS configuration so the host GPU is used instead or, if that's not possible, to swap the host and guest cards in the machine itself.<br />
<br />
=== Bypassing the IOMMU groups (ACS override patch) ===<br />
<br />
If you find your PCI devices grouped among others that you do not wish to pass through, you may be able to seperate them using Alex Williamson's ACS override patch. Make sure you understand [https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html the potential risk] of doing so. <br />
<br />
You will need a kernel with the patch applied. The easiest method to acquiring this is through the {{AUR|linux-vfio}} package. <br />
<br />
In addition, the ACS override patch needs to be enabled with kernel command line options. The patch file adds the following documentation:<br />
<br />
pcie_acs_override =<br />
[PCIE] Override missing PCIe ACS support for:<br />
downstream<br />
All downstream ports - full ACS capabilties<br />
multifunction<br />
All multifunction devices - multifunction ACS subset<br />
id:nnnn:nnnn<br />
Specfic device - full ACS capabilities<br />
Specified as vid:did (vendor/device ID) in hex<br />
<br />
The option {{ic|1=pcie_acs_override=downstream}} is typically sufficient.<br />
<br />
After installation and configuration, reconfigure your [[Kernel parameters|bootloader kernel parameters]] to load the new kernel with the {{ic|1=pcie_acs_override=}} option enabled.<br />
<br />
== Plain QEMU without libvirt ==<br />
<br />
Instead of setting up a virtual machine with the help of libvirt, plain QEMU commands with custom parameters can be used for running the VM intended to be used with PCI passthrough. This is desirable for some use cases like scripted setups, where the flexibility for usage with other scripts is needed.<br />
<br />
To achieve this after [[#Setting up IOMMU]] and [[#Isolating the GPU]], follow the [[QEMU]] article to setup the virtualized environment, [[QEMU#Enabling KVM|enable KVM]] on it and use the flag {{ic|1=-device vfio-pci,host=07:00.0}} replacing the identifier (07:00.0) with your actual device's ID that you used for the GPU isolation earlier.<br />
<br />
For utilizing the OVMF firmware, make sure the {{Pkg|ovmf}} package is installed, copy the UEFI variables from {{ic|/usr/share/ovmf/x64/OVMF_VARS.fd}} to temporary location like {{ic|/tmp/MY_VARS.fd}} and finally specify the OVMF paths by appending the following parameters to the QEMU command (order matters):<br />
<br />
* {{ic|1=-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd}} for the actual OVMF firmware binary, note the readonly option<br />
* {{ic|1=-drive if=pflash,format=raw,file=/tmp/MY_VARS.fd}} for the variables<br />
<br />
{{Note|QEMU's default SeaBIOS can be used instead of OVMF, but it's not recommended as it can cause issues with passthrough setups.}}<br />
<br />
It's recommended to study the QEMU article for ways to enhance the performance by using the [[QEMU#Installing virtio drivers|virtio drivers]] and other further configurations for the setup.<br />
<br />
You also might have to use the {{ic|1=-cpu host,kvm=off}} parameter to forward the host's CPU model info to the VM and fool the virtualization detection used by Nvidia's and possibly other manufacturers' device drivers trying to block the full hardware usage inside a virtualized system.<br />
<br />
== Passing though other devices ==<br />
<br />
=== USB controller ===<br />
<br />
If your motherboard has multiple USB controllers mapped to multiple groups, it is possible to pass those instead of USB devices. Passing an actual controller over an individual USB device provides the following advantages : <br />
<br />
* If a device disconnects or changes ID over the course of an given operation (such as a phone undergoing an update), the VM will not suddenly stop seeing it.<br />
* Any USB port managed by this controller is directly handled by the VM and can have its devices unplugged, replugged and changed without having to notify the hypervisor.<br />
* Libvirt will not complain if one of the USB devices you usually pass to the guest is missing when starting the VM.<br />
<br />
Unlike with GPUs, drivers for most USB controllers do not require any specific configuration to work on a VM and control can normally be passed back and forth between the host and guest systems with no side effects.<br />
<br />
{{Warning|Make sure your USB controller supports resetting :[[#Passing through a device that does not support resetting]]}}<br />
<br />
You can find out which PCI devices correspond to which controller and how various ports and devices are assigned to each one of them using this command :<br />
<br />
{{hc|$ <nowiki>for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done</nowiki>|<br />
Bus 1 --> 0000:00:1a.0 (IOMMU group 4)<br />
Bus 001 Device 004: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)<br />
Bus 001 Device 007: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]<br />
Bus 001 Device 008: ID 0781:5530 SanDisk Corp. Cruzer<br />
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
<br />
Bus 2 --> 0000:00:1d.0 (IOMMU group 9)<br />
Bus 002 Device 006: ID 0451:e012 Texas Instruments, Inc. TI-Nspire Calculator<br />
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub<br />
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub<br />
}}<br />
<br />
This laptop has 3 USB ports managed by 2 USB controllers, each with their own IOMMU group. In this example, Bus 001 manages a single USB port (with a SanDisk USB pendrive plugged into it so it appears on the list), but also a number of internal devices, such as the internal webcam and the bluetooth card. Bus 002, on the other hand, does not apprear to manage anything except for the calculator that is plugged into it. The third port is empty, which is why it does not show up on the list, but is actually managed by Bus 002.<br />
<br />
Once you have identified which controller manages which ports by plugging various devices into them and decided which one you want to passthrough, simply add it to the list of PCI host devices controlled by the VM in your guest configuration. No other configuration should be needed.<br />
<br />
=== Passing VM audio to host via PulseAudio ===<br />
<br />
It is possible to route the virtual machine's audio to the host as an application using libvirt. This has the advantage of multiple audio streams being routable to one host output, and working with audio output devices that do not support passthrough. [[PulseAudio]] is required for this to work.<br />
<br />
First, remove the comment from the {{ic|1=#user = ""}} line. Then add your username in the quotations. This tells QEMU which user's pulseaudio stream to route through.<br />
<br />
{{hc|/etc/libvirt/qemu.conf|2=<br />
user = "example"<br />
}}<br />
<br />
Next, modify the libvirt configuration<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm'><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
<domain type<nowiki>=</nowiki>'kvm' xmlns:qemu<nowiki>='http://libvirt.org/schemas/domain/qemu/1.0'</nowiki>><br />
}}<br />
<br />
Then set the QEMU PulseAudio environment variables at the bottom of the libvirt xml file<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
</domain><br />
}}<br />
<br />
to<br />
<br />
{{hc|$ virsh edit [vmname]|<br />
</devices><br />
<qemu:commandline><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_AUDIO_DRV' value<nowiki>=</nowiki>'pa'/><br />
<qemu:env name<nowiki>=</nowiki>'QEMU_PA_SERVER' value<nowiki>=</nowiki>'/run/user/1000/pulse/native'/><br />
</qemu:commandline><br />
</domain><br />
}}<br />
<br />
Change 1000 under the user directory to your user uid (which can be found by running the {{ic|id}} command. Remember to save the file and exit it without ending the process before continuing, otherwise the changes will not register. If you get the message {{ic|<nowiki>Domain [vmname] XML configuration edited.</nowiki>}} after exiting, it means that your changes have been applied.<br />
<br />
[[Restart]] {{ic|libvirtd.service}} and [[systemd/User|user service]] {{ic|pulseaudio.service}}.<br />
<br />
Virtual Machine audio will now be routed through the host as an application. The application {{Pkg|pavucontrol}} can be used to control the output device. Be aware that on Windows guests, this can cause audio crackling without [[#Slowed down audio pumped through HDMI on the video card|using Message-Signaled Interrupts.]]<br />
<br />
=== Gotchas ===<br />
<br />
==== Passing through a device that does not support resetting ====<br />
<br />
When the VM shuts down, all devices used by the guest are deinitialized by its OS in preparation for shutdown. In this state, those devices are no longer functionnal and must then be power-cycled before they can resume normal operation. Linux can handle this power-cycling on its own, but when a device has no known reset methods, it remains in this disabled state and becomes unavailable. Since Libvirt and Qemu both expect all host PCI devices to be ready to reattach to the host before completely stopping the VM, when encountering a device that will not reset, they will hang in a "Shutting down" state where they will not be able to be restarted until the host system has been rebooted. It is therefore reccomanded to only pass through PCI devices which the kernel is able to reset, as evidenced by the presence of a {{ic|reset}} file in the PCI device sysfs node, such as {{ic|/sys/bus/pci/devices/0000:00:1a.0/reset}}.<br />
<br />
The following bash command shows which devices can and cannot be reset.<br />
<br />
{{hc|<nowiki>for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done</nowiki>|<br />
IOMMU group 0<br />
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)<br />
IOMMU group 1<br />
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)<br />
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 720] [10de:1288] (rev a1)<br />
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)<br />
IOMMU group 2<br />
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)<br />
IOMMU group 4<br />
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)<br />
IOMMU group 5<br />
[RESET] 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)<br />
IOMMU group 10<br />
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)<br />
IOMMU group 13<br />
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br />
06:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)<br />
}}<br />
<br />
This signals that the xHCI USB controller in 00:14.0 cannot be reset and will therefore stop the VM from shutting down properly, while the integrated sound card in 00:1b.0 and the other two controllers in 00:1a.0 and 00:1d.0 do not share this problem and can be passed without issue.<br />
<br />
== Complete setups and examples ==<br />
<br />
If you have trouble configuring a certain mechanism in your setup, you might want to look up [[PCI_passthrough_via_OVMF/Examples|complete passthrough setup examples]]. A few users have described their setups and you might want to look up certain tricks from their configuration files.<br />
<br />
== Troubleshooting ==<br />
<br />
If your issue is not mentioned below, you may want to browse [[QEMU#Troubleshooting]].<br />
<br />
=== "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs ===<br />
<br />
{{Note|<br />
* This may also fix SYSTEM_THREAD_EXCEPTION_NOT_HANDLED boot crashes related to Nvidia drivers.<br />
* This may also fix problems under linux guests.<br />
}}<br />
<br />
Since version 337.88, Nvidia drivers on Windows check if an hypervisor is running and fail if it detects one, which results in an Error 43 in the Windows device manager. Starting with QEMU 2.5.0 and libvirt 1.3.3, the vendor_id for the hypervisor can be spoofed, which is enough to fool the Nvidia drivers into loading anyway. All one must do is add {{ic|<nowiki>hv_vendor_id=whatever</nowiki>}} to the cpu parameters in their QEMU command line, or by adding the following line to their libvirt domain configuration. It may help for the ID to be set to a 12-character alphanumeric (e.g. '123456789ab') as opposed to longer or shorter strings.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
...<br />
<vendor_id state='on' value='whatever'/><br />
...<br />
</hyperv><br />
...<br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
Users with older versions of QEMU and/or libvirt will instead have to disable a few hypervisor extensions, which can degrade performance substantially. If this is what you want to do, do the following replacement in your libvirt domain config file.<br />
<br />
{{hc|$ virsh edit [vmname]|<nowiki><br />
...<br />
<features><br />
<hyperv><br />
<relaxed state='on'/><br />
<vapic state='on'/><br />
<spinlocks state='on' retries='8191'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='yes'/><br />
</clock><br />
...<br />
</nowiki>}}<br />
<br />
{{bc|<nowiki><br />
...<br />
<clock offset='localtime'><br />
<timer name='hypervclock' present='no'/><br />
</clock><br />
...<br />
<features><br />
<kvm><br />
<hidden state='on'/><br />
</kvm><br />
...<br />
<hyperv><br />
<relaxed state='off'/><br />
<vapic state='off'/><br />
<spinlocks state='off'/><br />
</hyperv><br />
...<br />
</features><br />
...<br />
</nowiki>}}<br />
<br />
==== "BAR 3: cannot reserve [mem]" error in dmesg after starting VM ====<br />
<br />
With respect to [https://www.linuxquestions.org/questions/linux-kernel-70/kernel-fails-to-assign-memory-to-pcie-device-4175487043/ this article]:<br />
<br />
If you still have code 43 check dmesg for memory reservation errors after starting up VM, if you have similar it could be the case:<br />
<br />
vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]<br />
<br />
Find out a PCI Bridge your graphic card is connected to. This will give actual hierarchy of devices:<br />
<br />
$ lspci -t<br />
<br />
Before starting VM run following lines replacing IDs with actual from previous output.<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove<br />
# echo 1 > /sys/bus/pci/rescan<br />
<br />
{{Note|Probably setting [[kernel parameter]] {{ic|1=video=efifb:off}} is required as well. [https://pve.proxmox.com/wiki/Pci_passthrough#BAR_3:_can.27t_reserve_.5Bmem.5D_error Source]}}<br />
<br />
=== UEFI (OVMF) Compatability in VBIOS ===<br />
<br />
With respect to [https://pve.proxmox.com/wiki/Pci_passthrough#How_to_known_if_card_is_UEFI_.28ovmf.29_compatible this article]:<br />
<br />
Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use {{ic|rom-parser}}:<br />
<br />
$ git clone https://github.com/awilliam/rom-parser<br />
$ cd rom-parser && make<br />
<br />
Dump the GPU VBIOS:<br />
<br />
# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
# cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom<br />
# echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom<br />
<br />
And test it for compatibility:<br />
<br />
{{hc|$ ./rom-parser /tmp/image.rom|<br />
Valid ROM signature found @600h, PCIR offset 190h<br />
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 0, vendor revision: 1<br />
Valid ROM signature found @fa00h, PCIR offset 1ch<br />
PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000<br />
PCIR: revision 3, vendor revision: 0<br />
EFI: Signature Valid, Subsystem: Boot, Machine: X64<br />
Last image<br />
}}<br />
<br />
To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it's not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on [https://www.techpowerup.com/vgabios/ TechPowerUp].<br />
<br />
Updated VBIOS can be used in the VM without flashing. To load it in QEMU:<br />
<br />
-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \<br />
<br />
And in libvirt:<br />
<br />
{{bc|1=<br />
<hostdev><br />
...<br />
<rom file='/path/to/your/gpu/bios.bin'/><br />
...<br />
</hostdev><br />
}}<br />
<br />
One should compare VBIOS versions between host and guest systems using [https://www.techpowerup.com/download/nvidia-nvflash/ nvflash] (Linux versions under ''Show more versions'') or <br />
[https://www.techpowerup.com/download/techpowerup-gpu-z/ GPU-Z] (in Windows guest). To check the currently loaded VBIOS:<br />
<br />
{{hc|$ ./nvflash --version|<br />
...<br />
Version : 80.04.XX.00.97<br />
...<br />
UEFI Support : No<br />
UEFI Version : N/A<br />
UEFI Variant Id : N/A ( Unknown )<br />
UEFI Signer(s) : Unsigned<br />
...<br />
}}<br />
<br />
And to check a given VBIOS file:<br />
<br />
{{hc|$ ./nvflash --version NV299MH.rom|<br />
...<br />
Version : 80.04.XX.00.95<br />
...<br />
UEFI Support : Yes<br />
UEFI Version : 0x10022 (Jul 2 2013 @ 16377903 )<br />
UEFI Variant Id : 0x0000000000000004 ( GK1xx )<br />
UEFI Signer(s) : Microsoft Corporation UEFI CA 2011<br />
...<br />
}}<br />
<br />
If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using [https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html GOPUpd] tool, however this is risky and may result in GPU brick.<br />
<br />
{{Warning|Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. '''DO NOT''' flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.}}<br />
<br />
In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:<br />
<br />
# modprobe -r nvidia_modeset nvidia <br />
<br />
Flashing the VBIOS can be done with:<br />
<br />
# ./nvflash romfile.bin<br />
<br />
{{Warning|'''DO NOT''' interrupt the flashing process, even if it looks like it's stuck. Flashing should take about a minute on most GPUs, but may take longer.}}<br />
<br />
=== Slowed down audio pumped through HDMI on the video card ===<br />
<br />
For some users VM's audio slows down/starts stuttering/becomes demonic after a while when it's pumped through HDMI on the video card. This usually also slows down graphics.<br />
A possible solution consists of enabling MSI (Message Signaled-Based Interrupts) instead of the default (Line-Based Interrupts).<br />
<br />
In order to check whether MSI is supported or enabled, run the following command as root:<br />
<br />
# lspci -vs $device | grep 'MSI:'<br />
<br />
where `$device` is the card's address (e.g. `01:00.0`).<br />
<br />
The output should be similar to:<br />
<br />
Capabilities: [60] MSI: Enable'''-''' Count=1/1 Maskable- 64bit+<br />
<br />
A {{ic|-}} after {{ic|Enable}} means MSI is supported, but not used by the VM, while a {{ic|+}} says that the VM is using it.<br />
<br />
The procedure to enable it is quite complex, instructions and an overview of the setting can be found [https://forums.guru3d.com/showthread.php?t=378044 here].<br />
<br />
Other hints can be found on the [https://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support lime-technology's wiki], or on this article on [https://vfio.blogspot.it/2014/09/vfio-interrupts-and-how-to-coax-windows.html VFIO tips and tricks].<br />
<br />
Some tools named {{ic|MSI_util}} or similar are available on the Internet, but they do not work on Windows 10 64bit.<br />
<br />
In order to fix the issues enabling MSI on the 0 function of a nVidia card ({{ic|01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])}}) was not enough; it will also be required to enable it on the other function ({{ic|01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)}}) to fix the issue.<br />
<br />
=== No HDMI audio output on host when intel_iommu is enabled ===<br />
<br />
If after enabling {{ic|intel_iommu}} the HDMI output device of Intel GPU becomes unusable on the host then setting the option {{ic|igfx_off}} (i.e. {{ic|1=intel_iommu=on,igfx_off}}) might bring the audio back, please read {{ic|Graphics Problems?}} in [https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt Intel-IOMMU.txt] for details about setting {{ic|igfx_off}}.<br />
<br />
=== X doesn't start after enabling vfio_pci ===<br />
<br />
This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci or the Xorg log. [https://www.redhat.com/archives/vfio-users/2016-August/msg00025.html Source →]<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-intel.conf|<nowiki><br />
Section "Device"<br />
Identifier "Intel GPU"<br />
Driver "modesetting"<br />
BusID "PCI:0:2:0"<br />
EndSection<br />
</nowiki>}}<br />
<br />
=== Chromium ignores integrated graphics for acceleration ===<br />
<br />
Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics). It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system - the result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the {{ic|vfio-pci}} driver, and unusable on the host system, regardless of whenever a guest VM is running or not. This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general un-smoothness).<br />
<br />
This can be fixed by [[Chromium/Tips and tricks#Forcing specific GPU|explicitly telling Chromium which GPU you want to use]].<br />
<br />
=== VM only uses one core ===<br />
<br />
For some users, even if IOMMU is enabled and the core count is set to more than 1, the VM still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in {{ic|virt-manager}} and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU, not the total count.<br />
<br />
=== Passthrough seems to work but no output is displayed ===<br />
<br />
Make sure if you are using virt-manager that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the VM.<br />
<br />
=== virt-manager has permission issues ===<br />
<br />
If you are getting a permission error with virt-manager add the following to your {{ic|/etc/libvirt/qemu.conf}}:<br />
<br />
group="kvm"<br />
user="''user''"<br />
<br />
If that does not work make sure your user account is added to the kvm and libvirt [[groups]].<br />
<br />
== See also ==<br />
<br />
* [https://bbs.archlinux.org/viewtopic.php?id=162768 Discussion on Arch Linux forums] | [https://archive.is/kZYMt Archived link]<br />
* [https://docs.google.com/spreadsheet/ccc?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE User contributed hardware compatibility list]<br />
* [https://pastebin.com/rcnUZCv7 Example script from https://www.youtube.com/watch?v=37D2bRsthfI]<br />
* [https://vfio.blogspot.com/ Complete tutorial for PCI passthrough]<br />
* [https://www.redhat.com/archives/vfio-users/ VFIO users mailing list]<br />
* [https://webchat.freenode.net/?channels=vfio-users #vfio-users on freenode]<br />
* [https://www.youtube.com/watch?v=aLeWg11ZBn0 YouTube: Level1Linux - GPU Passthrough for Virtualization with Ryzen]</div>NeoTheFox