PRIME

From ArchWiki

PRIME is a technology used to manage hybrid graphics found on recent desktops and laptops (Optimus for NVIDIA, AMD Dynamic Switchable Graphics for Radeon). PRIME GPU offloading and Reverse PRIME are an attempt to support muxless hybrid graphics in the Linux kernel.

Installation

Open-source drivers

Remove any closed-source graphic drivers and replace them with the open source equivalent:

Reboot and check the list of attached graphic drivers:

$ xrandr --listproviders
Providers: number : 2
Provider 0: id: 0x7d cap: 0xb, Source Output, Sink Output, Sink Offload crtcs: 3 outputs: 4 associated providers: 1 name:Intel
Provider 1: id: 0x56 cap: 0xf, Source Output, Sink Output, Source Offload, Sink Offload crtcs: 6 outputs: 1 associated providers: 1 name:radeon

We can see that there are two graphic cards: Intel, the integrated card (id 0x7d), and Radeon, the discrete card (id 0x56), which should be used for GPU-intensive applications.

By default the Intel card is always used:

$ glxinfo | grep "OpenGL renderer"
OpenGL renderer string: Mesa DRI Intel(R) Ivybridge Mobile
Note: Sometimes, the displayed provider is "HAINAN @ pci:0000:03:00.0", not radeon. In this case you should use "HAINAN @ pci:0000:03:00.0" as the provider in the next command.

Closed-source drivers

To get PRIME functioning on the proprietary drivers, it is pretty much the same process. Follow the following articles to install the drivers:

  • AMDGPU PRO to install drivers for AMD GPUs.
  • NVIDIA to install drivers for NVIDIA GPUs.

After you have the driver installed, do not reboot or relaunch Xorg. Depending on your system configuration, this may render your Xorg system unusable until reconfigured.

Follow the instructions for the section on your designated use-case. You do not need to uninstall the open-source drivers for it to function, but you probably should, for the sake of preventing clutter and potential future issues.

PRIME GPU offloading

We want to render applications on the more powerful card and send the result to the card which has display connected.

The command xrandr --setprovideroffloadsink provider sink can be used to make a render offload provider send its output to the sink provider (the provider which has a display connected). The provider and sink identifiers can be numeric (0x7d, 0x56) or a case-sensitive name (Intel, radeon).

Note:
  • This setting is no longer necessary when using most default Xorg DDX (the xf86-video-* or built-in modesetting) drivers from the official repositories, as they have DRI3 enabled by default and will therefore automatically make these assignments. Explicitly setting them again does no harm, though.
  • GPU offloading is not supported by the closed-source drivers (for the NVIDIA driver this is no longer the case, for more info see #PRIME render offload below).

Example:

$ xrandr --setprovideroffloadsink radeon Intel

You may also use provider index instead of provider name:

$ xrandr --setprovideroffloadsink 1 0

This article or section needs language, wiki syntax or style improvements. See Help:Style for reference.

Reason: The following subsections are a messy, we name a subsection PRIME render offload without stating it's NVIDIA-centric, we shove generic PCI-E power management for the open source drivers inside, the merge flag from External GPU#Xorg rendered on iGPU, PRIME render offload to eGPU should probably be addressed on the other page, once this has cleaned up and we can link to here for the environment variables to use. (Discuss in Talk:PRIME)

For open source drivers - PRIME

To use your discrete card for the applications who need it the most (for example games, 3D modellers...), prepend the DRI_PRIME=1 environment variable:

$ DRI_PRIME=1 glxinfo | grep "OpenGL renderer"
OpenGL renderer string: Gallium 0.4 on AMD TURKS
Note: Instead of numerical value, you can also specify a PCI device name. Format used is similar to /sys/bus/pci/devices/, but prefix with pci- and replace semicolons and dots by underscores, e.g. DRI_PRIME=pci-0000_01_00_0.

Other applications will still use the less power-hungry integrated card. These settings are lost once the X server restarts, you may want to make a script and auto-run it at the startup of your desktop environment (alternatively, put it in /etc/X11/xinit/xinitrc.d/). This may reduce your battery life and increase heat though.

See Gentoo:AMDGPU#Test, if a discrete graphics card is in use for more information.

For DRI_PRIME to work on Vulkan applications vulkan-mesa-layers needs to be installed, as well as lib32-vulkan-mesa-layers for 32 bit applications.

PRIME render offload

NVIDIA driver since version 435.17 supports this method. The modesetting, xf86-video-amdgpu (450.57), and xf86-video-intel (455.38) are officially supported as iGPU drivers.

To run a program on the NVIDIA card you can use the prime-run script provided by nvidia-prime:

$ prime-run glxinfo | grep "OpenGL renderer"
$ prime-run vulkaninfo

PCI-Express Runtime D3 (RTD3) Power Management

Open-source drivers

Kernel PCI power management turns off the GPU when not used with PRIME offloading or reverse PRIME. This feature is supported by modesetting, xf86-video-amdgpu, xf86-video-intel, xf86-video-nouveau drivers.

The following command can be used to check current power state of each GPU:

$ cat /sys/class/drm/card*/device/power_state
NVIDIA
Note:
  • No configuration is generally needed for Ampere as this is enabled by default. For some Ampere users, udev rules may be necessary.
  • Some users with hybrid graphics are reporting that their discrete NVIDIA Ampere GPUs are failing to remain in the D3Cold power state after upgrading to newer NVIDIA drivers (seemingly >525) [1].

For Turing generation cards with Intel Coffee Lake or above CPUs as well as some Ryzen CPUs like the 5800H, it is possible to fully power down the GPU when not in use.

Note: If you plan to use suspend or hibernate, see NVIDIA/Tips and tricks#Preserve video memory after suspend.

The following udev rules are needed:

/etc/udev/rules.d/80-nvidia-pm.rules
# Enable runtime PM for NVIDIA VGA/3D controller devices on driver bind
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="auto"
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="auto"

# Disable runtime PM for NVIDIA VGA/3D controller devices on driver unbind
ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="on"
ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="on"

together with the following module parameters:

/etc/modprobe.d/nvidia-pm.conf
options nvidia "NVreg_DynamicPowerManagement=0x02"

Alternatively, you can install nvidia-prime-rtd3pmAUR which provides these two configuration files.

After you setup the udev rules and the module parameter either manually or using the AUR package, you will need to restart your Laptop.

To check if the NVIDIA GPU is turned off you can use this command:

$ cat /sys/bus/pci/devices/0000:01:00.0/power/runtime_status

You will see either suspended or running, if suspended is displayed, the GPU is turned off. Now the power draw will 0 Watts making the battery last longer.

This article or section needs language, wiki syntax or style improvements. See Help:Style for reference.

Reason: What are "some cases"? At least disclose the GPU. (Discuss in Talk:PRIME)

In some cases, none of the options above might be listed and instead the result will be active. This alone does not mean that the GPU is in the running state. In this case you can check the state using this command:

$ cat /sys/bus/pci/devices/0000:01:00.0/power/runtime_suspended_time

While the GPU is in suspended state, the counter will be incrementing every time you run the command. When the GPU's state becomes running it will stop incrementing.

We also need to enable nvidia-persistenced.service to avoid the kernel tearing down the device state whenever the NVIDIA device resources are no longer in use. [2]

Configure applications to render using GPU

This article or section is a candidate for merging with External GPU#Xorg rendered on iGPU, PRIME render offload to eGPU.

Notes: Target section mentions these variables. Perhaps this can be merged with the target section to avoid duplication? (Discuss in Talk:PRIME)

Even without enabling Dynamic Power Management, offload rendering of applications is required [3].

To run an application offloaded to the NVIDIA GPU with Dynamic Power Management enabled, add the following environment variables: [4]

__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia command

When using on a Steam game, the launcher command line can be set to:

__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia %command%
Note: The value of __NV_PRIME_RENDER_OFFLOAD may need to be set to 0 depending on the system. It is recommended to check which GPU is 0 and which is 1 since this variable specifies which GPU will be used.

Gnome integration

For GNOME integration, install switcheroo-control and enable switcheroo-control.service.

GNOME will respect the PrefersNonDefaultGPU property in the desktop entry. Alternatively, you can launch applications with GPU by right clicking on the icon and choosing Launch using Discrete Graphics Card.

Troubleshooting

If you have bumblebee installed, you should remove it because it blacklists the nvidia_drm driver which is required to load the NVIDIA driver by X server for offloading.

PRIME synchronization

When using PRIME, the primary GPU renders the screen content / applications, and passes it to the secondary GPU for display. Quoting an NVIDIA thread, "Traditional vsync can synchronize the rendering of the application with the copy into system memory, but there needs to be an additional mechanism to synchronize the copy into system memory with the iGPU’s display engine. Such a mechanism would have to involve communication between the dGPU’s and the iGPU’s drivers, unlike traditional vsync."

This synchronization is achieved using PRIME sync. To check if PRIME synchronization is enabled for your display, check the output of xrandr --prop.

To enable it run:

$ xrandr --output <output-name> --set "PRIME Synchronization" 1
Note:

Wayland-specific configuration

This article or section needs language, wiki syntax or style improvements. See Help:Style for reference.

Reason: It looks like this section duplicates all the previous ones, without explaining in details what is Wayland-specific about the environment variable used. (Discuss in Talk:PRIME)

Wayland needs less configuration than Xorg. There seems to be also preliminary support for GPU hotplugging in KDE's KWin and GNOME's Mutter (Issue 17 and Merge request 1562).

To use your discrete card prepend the DRI_PRIME= environment variable. The following examples assume a system with an Intel integrated card, an NVIDIA internal GPU and an AMD external GPU.

To use the integrated Intel chip, no modifications are required, since it is already the default:

glxinfo | grep 'OpenGL renderer'
OpenGL renderer string: Mesa Intel(R) Xe Graphics (TGL GT2)

For the AMD card with opensource drivers:

DRI_PRIME=pci-0000_06_00_0 glxinfo | grep 'OpenGL renderer'
OpenGL renderer string: AMD Radeon RX 5700 XT (navi10, LLVM 14.0.6, DRM 3.46, 5.18.17-hardened1-1-hardened)

For the NVIDIA card with proprietary drivers:

DRI_PRIME=pci-0000_01_00_0 __VK_LAYER_NV_optimus=NVIDIA_only __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo | grep 'OpenGL renderer string'
OpenGL renderer string: NVIDIA GeForce RTX 3050 Ti Laptop GPU/PCIe/SSE2

Multiple GPUs can be used at the same time on the same machine with Wayland.

Note: Hotplugging an external GPU is enabled with a limitation: re-logging is necessary to load the correct kernel modules.

See the mesa3d documentation for PRIME: https://docs.mesa3d.org/envvars.html

All Xwayland applications should work out of the box just like they do on Xorg. However, as of driver version 525.85, for native Wayland applications, only OpenGL works, provided you have modesetting enabled.

Vulkan Wayland applications will render an invisible window or might even crash since the NVIDIA driver uses hardware specific pixel layout which the Intel or AMD GPUs do not understand. More info on NVIDIA's GitHub issue 72.

Reverse PRIME

This article or section needs expansion.

Reason: Missing info about configuring Intel + AMD reverse prime for both open and closed amdgpu drivers (Discuss in Talk:PRIME)
Note:
  • Reverse PRIME is not supported for AMDGPU + NVIDIA on NVIDIA driver prior to 470 beta. See [5] for more details. An alternative for older versions is to use NVIDIA as the primary card as described in #Discrete card as primary GPU.
  • Currently when only external display is enabled, you will only get 1 FPS. See [6] for more details.

If the second GPU has outputs that are not accessible by the primary GPU, you can use Reverse PRIME to make use of them. This will involve using the primary GPU to render the images, and then pass them off to the second GPU.

It may work out of the box, however if not, please go through the following steps.

Configuration

This article or section needs language, wiki syntax or style improvements. See Help:Style for reference.

Reason: First half of the section has Intel + NVIDIA, the second half has Intel + Radeon. Identifying the GPU bus IDs should be delegated to a general section. (Discuss in Talk:PRIME)

First, identify integrated GPU BusID

lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (Mobile)
01:00.0 VGA compatible controller: NVIDIA Corporation TU117M [GeForce GTX 1650 Mobile / Max-Q] (rev a1)

In the above example Intel card has 00:02.0 which translates to PCI:0:2:0.

Set up your xorg.conf as follows and adjust BusID.

/etc/X11/xorg.conf
Section "ServerLayout"
        Identifier "layout"
        Screen 0 "intel"
        Inactive "nvidia"
        Option "AllowNVIDIAGPUScreens"
EndSection

Section "Device"
        Identifier "nvidia"
        Driver "nvidia"
EndSection

Section "Screen"
        Identifier "nvidia"
        Device "nvidia"
EndSection

Section "Device"
        Identifier "intel"
        Driver "modesetting"
        BusID "PCI:0:2:0"
EndSection

Section "Screen"
        Identifier "intel"
        Device "intel"
EndSection

The command xrandr --setprovideroutputsource provider source sets the provider as output for the source. For example:

$ xrandr --setprovideroutputsource radeon Intel

When this is done, the discrete card's outputs should be available in xrandr, and you could do something like:

$ xrandr --output HDMI-1 --auto --above LVDS1

to configure both internal as well as external displays.

Problems

If after reboot you only have one provider, it might be because when Xorg starts, the nvidia module is not loaded yet. You need to enable early module loading. See NVIDIA#Early loading for details.

User scenarios

Discrete card as primary GPU

Imagine following scenario: The LVDS1 (internal laptop screen) and VGA outputs are both only accessible through the integrated Intel GPU. The HDMI and Display Port outputs are attached to the discrete NVIDIA card. It is possible to use all four outputs by making use of the #Reverse PRIME technology as described above. However the performance might be slow, because all the rendering for all outputs is done by the integrated Intel card. To improve this situation it is possible to do the rendering by the discrete NVIDIA card, which then copies the framebuffers for the LVDS1 and VGA outputs to the Intel card.

Create the following Xorg configuration:

/etc/X11/xorg.conf.d/10-gpu.conf
Section "ServerLayout"
    Identifier "layout"
    Screen 0 "nouveau"
    Inactive "intel"
EndSection

Section "Device"
    Identifier  "nouveau"
    Driver      "nouveau"
    BusID       "PCI:x:x:x" # Sample: "PCI:1:0:0"
EndSection

Section "Screen"
    Identifier "nouveau"
    Device "nouveau"
EndSection

Section "Device"
    Identifier  "intel"
    Driver      "intel"
    BusID       "PCI:x:x:x"  # Sample: "PCI:0:2:0"
EndSection

Section "Screen"
    Identifier "intel"
    Device "intel"
EndSection
Note: For the proprietary NVIDIA driver see [7] instead.

Restart Xorg. The discrete NVIDIA card should be used now. The HDMI and Display Port outputs are the main outputs. The LVDS1 and VGA outputs are off. To enable them run:

$ xrandr --setprovideroutputsource Intel nouveau

The internal card's outputs should be available now in xrandr.

Note: With NVIDIA being used to render the screen, you may face sluggish scrolling or screen tearing. Check NVIDIA/Troubleshooting#Avoid screen tearing on how to mitigate it.

Troubleshooting

The factual accuracy of this article or section is disputed.

Reason: No sources to motivate the workarounds in this section (Discuss in Talk:PRIME)

XRandR specifies only 1 output provider

Delete/move /etc/X11/xorg.conf file and any other files relating to GPUs in /etc/X11/xorg.conf.d/. Restart the X server after this change.

If the video driver is blacklisted in /etc/modprobe.d/ or /usr/lib/modprobe.d/, load the module and restart X. This may be the case if you use the bbswitch module for NVIDIA GPUs.

Another possible problem is that Xorg might try to automatically assign monitors to your second GPU. Check the logs:

$ grep "No modes" ~/.local/share/xorg/Xorg.0.log
AMDGPU(0): No modes.

To solve this add the ServerLayout section with inactive device to your xorg.conf:

/etc/X11/xorg.conf
Section "ServerLayout"
  Identifier     "X.org Configured"
  Screen      0  "Screen0" 0 0 # Screen for your primary GPU
  Inactive       "Card1"       # Device for your second GPU
EndSection

When an application is rendered with the discrete card, it only renders a black screen

In some cases PRIME needs a composition manager to properly work. If your window manager doesn’t do compositing, you can use xcompmgr on top of it.

If you use Xfce, you can go to Menu > Settings > Window Manager Tweaks > Compositor and enable compositing, then try again your application.

Black screen with GL-based compositors

Currently there are issues with GL-based compositors and PRIME offloading. While Xrender-based compositors (xcompmgr, xfwm, compton's default backend, cairo-compmgr, and a few others) will work without issue, GL-based compositors (Mutter/muffin, Compiz, compton with GLX backend, Kwin's OpenGL backend, etc) will initially show a black screen, as if there was no compositor running. While you can force an image to appear by resizing the offloaded window, this is not a practical solution as it will not work for things such as full screen Wine applications. This means that desktop environments such as GNOME3 and Cinnamon have issues with using PRIME offloading.

Additionally if you are using an Intel IGP you might be able to fix the GL Compositing issue by running the IGP as UXA instead of SNA, however this may cause issues with the offloading process (ie, xrandr --listproviders may not list the discrete GPU).

For details see FDO Bug #69101.

One other way to approach this issue is by enabling DRI3 in the Intel driver. See the below issue for a sample configuration.

GNOME

You may find that disabling fullscreen undirect allows PRIME offloading to work correctly for full-screen applications.

Kernel crash/oops when using PRIME and switching windows/workspaces

Note: This has been tested on a system with Intel+AMD

Using DRI3 WITH a configuration file for the integrated card seems to fix this issue.

To enable DRI3, you need to create a configuration for the integrated card adding the DRI3 option:

Section "Device"
    Identifier "Intel Graphics"
    Driver "intel"
    Option "DRI" "3"
EndSection

After this you can use DRI_PRIME=1 WITHOUT having to run xrandr --setprovideroffloadsink radeon Intel as DRI3 will take care of the offloading.

Glitches/Ghosting synchronization problem on second monitor when using reverse PRIME

This problem can affect users when not using a composite manager, such as with i3. [8]

If you experience this problem under Gnome, then a possible fix is to set some environment variables in /etc/environment [9]

CLUTTER_PAINT=disable-clipped-redraws:disable-culling
CLUTTER_VBLANK=True

Error "radeon: Failed to allocate virtual address for buffer:" when launching GL application

This error is given when the power management in the kernel driver is running. You can overcome this error by appending radeon.runpm=0 to the kernel parameters in the bootloader.

Constant hangs/freezes with Vulkan applications/games using VSync with closed-source drivers and reverse PRIME

Some Vulkan applications (particularly ones using VK_PRESENT_MODE_FIFO_KHR and/or VK_PRESENT_MODE_FIFO_RELAXED_KHR, including Windows games ran with DXVK) will cause the GPU to lockup constantly (~5-10 seconds freezed, ~1 second working fine)[10] when ran on a system using reverse PRIME.

A GPU lockup will render any input unusable (this includes switching TTYs and using SysRq functions).

There is no known fix for this NVIDIA bug, but a few workarounds exist:

  • Turning Vsync off (not possible for some applications)
  • Turning PRIME Synchronization[11] off (will introduce screen tearing):
xrandr --output HDMI-0 --set "PRIME Synchronization" 0 #replace HDMI-0 with your xrandr output ID

You can verify if your configuration is affected by the issue simply by running vkcube from the vulkan-tools package.

Some programs have a delay when opening under Wayland

If you have RTD3 working (from #NVIDIA), when using Wayland you will experience some delay when a program opens, this is because it tries to power on the GPU first (which it takes ~1 second or more) and then it tries to open the program, wasting time and battery life. This is an NVIDIA driver problem. More details here

To solve this, add these 2 lines in your /etc/environment file:

/etc/environment
__EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json
__GLX_VENDOR_LIBRARY_NAME=mesa

Error when running Wine games with DXVK

When using PRIME offload, encountering the Major opcode of failed request: 156 (NV-GLX) is a know problem. The only known workaround is to start X session entirely on NVIDIA GPU. A user friendly way to switching between NVIDIA only and PRIME offload method is the optimus-manager utility or write some automation scripts yourself.

See also