Solid state drive/NVMe

From ArchWiki

NVM Express (NVMe) is a specification for accessing SSDs attached through the PCI Express bus. As a logical device interface, NVM Express has been designed from the ground up, capitalizing on the low latency and parallelism of PCI Express SSDs, and mirroring the parallelism of contemporary CPUs, platforms and applications.

Installation

NVMe devices should show up as /dev/nvme*. See Device file#NVMe for an explanation on their naming.

Extra userspace NVMe tools can be found in nvme-cli or nvme-cli-gitAUR.

See Solid State Drives for supported filesystems, maximizing performance, minimizing disk reads/writes, etc.

Management

Note: This section was adapted from [1].

List all the NVMe SSDs attached with name, serial number, size, LBA format and serial:

# nvme list

List information about a drive and features it supports in a human-friendly way:

# nvme id-ctrl -H /dev/nvme0
Tip: In order to make sense of the abbreviations used, see reference section "Identify Controller data structure" in the relevant NVMe specification, (e.g. bottom of page 172 for the 1.4a specification).

List information about a namespace and features it supports:

Note: Namespaces are the construct in NVMe technology that hold user data. An NVMe controller can have multiple namespaces attached to it. Most NVMe SSDs today just use a single namespace, but multi-tenant applications, virtualization and security have use cases for multiple namespaces.
# nvme id-ns /dev/nvme0n1
Tip: In order to make sense of the abbreviations used, see reference section "Identify Namespace data structure" in the relevant NVMe specification (e.g. page 163 for the 1.4a specification).

Output the NVMe error log page:

# nvme error-log /dev/nvme0
Tip: Look for output where error count does not equal 0 to find out if there are any errors in the error log.

Delete a namespace:

Warning: This command will delete all data on the specified namespace. Use with caution!
# nvme delete-ns /dev/nvme0n1

Create a new namespace, e.g creating a smaller size namespace to overprovision an SSD for improved endurance, performance, and latency:

# nvme create-ns /dev/nvme0

See nvme help and nvme(1) for a list of all commands along with a terse description.

SMART

Output the NVMe SMART log page for health status, temp, endurance, and more:

# nvme smart-log /dev/nvme0
Tip: Use the -H option to output even more information, e.g. nvme smart-log -H /dev/nvme0.

NVMe support was added to smartmontools in version 6.5.

Note: smartmontools official wiki reports this support as experimental.

Currently implemented features (as taken from the wiki):

  • Basic information about controller name, firmware, capacity (smartctl -i)
  • Controller and namespace capabilities (smartctl -c)
  • SMART overall-health self-assessment test result and warnings (smartctl -H)
  • NVMe SMART attributes (smartctl -A)
  • NVMe error log (smartctl -l error[,NUM])
  • Ability to fetch any nvme log (smartctl -l nvmelog,N,SIZE)
  • The smartd daemon tracks health (-H), error count (-l error) and temperature (-W DIFF,INFO,CRIT)

See S.M.A.R.T. and the official wiki entry for more information, and see this article for contextual information about the output.

Secure erase

See Solid state drive/Memory cell clearing#NVMe drive.

Firmware update

Generic

Firmware can be managed using nvme-cli. To display available slots and check whether Slot 1 is read only:

# nvme fw-log /dev/nvme0
Firmware Log for device:nvme0
afi  : 0x11
frs1 : 0x32303132345a3553 (S5Z42102)
frs2 : 0x32303132345a3553 (S5Z42102)
# nvme id-ctrl /dev/nvme0 -H | grep Firmware
  [9:9] : 0x1	Firmware Activation Notices Supported
  [4:4] : 0x1	Firmware Activate Without Reset Supported
  [3:1] : 0x2	Number of Firmware Slots
  [0:0] : 0	Firmware Slot 1 Read/Write

Download and commit firmware to specified slot. In the example below, firmware is first committed without activation (-a 0). Next, an existing image is activated (-a 2). Refer to the NVMe specification for details on firmware commit action values.

# nvme fw-download -f S5Z42_fw_S5Z42105.bin /dev/nvme0
Firmware download success
# nvme fw-commit -s 2 -a 0 /dev/nvme0
Success committing firmware action:0 slot:2
# nvme fw-log /dev/nvme0
Firmware Log for device:nvme0
afi  : 0x21
frs1 : 0x32303132345a3553 (S5Z42102)
frs2 : 0x35303132345a3553 (S5Z42105)
# nvme fw-commit -s 2 -a 2 /dev/nvme0
Success committing firmware action:2 slot:2

Finally reset the controller to load the new firmware:

Note: If Firmware Activate Without Reset is marked as supported as in the example above, this step may not be necessary.
# nvme reset /dev/nvme0

This can also be done manually if needed:

# echo 1 > /sys/class/nvme/nvme0/reset_controller

Intel/Solidigm

After Intel SSD business was acquired by SK Hynix[2][3], its "Memory and Storage Tool" (Intel MAS) lost support for SSDs and can now only be used to manage Optane devices.[4]

Solidigm, the US subsidiary formed from Intel's SSD business acquisition, provides a new utility to manage former Intel SSDs: "The Solidigm Storage Tool, also called SST, assists with managing Solidigm SSDs. It provides access to drive information and health, SMART Attributes, Firmware Updates, diagnostic scans, and secure erase."[5]

Install solidigm-sst-storage-tool-cliAUR, then check whether your drive has an update available:

# sst show -ssd
- ABCD012345NS512A -

Capacity : 512.11 GB (512,110,190,592 bytes)
DevicePath : /dev/nvme0n1
DeviceStatus : Healthy
Firmware : 004C
FirmwareUpdateAvailable : 005C
Index : 0
MaximumLBA : 1000215215
ModelNumber : INTEL SSDPEKNW512G8
ProductFamily : Intel SSD 660p Series
SMARTEnabled : True
SectorDataSize : 512
SerialNumber : ABCD012345NS512A

If so execute the load command as follows, using the index value given in the previous command:

# sst load -ssd index
WARNING! You have selected to update the drives firmware! 
Proceed with the update? (Y|N): Y
Updating firmware...
Firmware update successful

For more information, refer to the user guide provided on the tool's aforementioned official page.

Kingston

Kingston does not provide separate firmware downloads on their website, instead referring users to a Windows only utility. Firmware files appear to use a predictable naming scheme based on the firmware revision:

https://media.kingston.com/support/downloads/S5Z42105.zip

Then proceed with the generic flashing instructions.

Samsung

Next to "Samsung Magician Software" for Windows users Samsung also provides SSD firmware as bootable ISO images:

https://semiconductor.samsung.com/consumer-storage/support/tools/

They can be written onto a bootable CD or USB drive, or you can unpack the image and do everything live:

$ curl -OL https://samsung.com/.../xxx.iso
$ bsdtar -xf xxx.iso initrd
$ bsdtar -xf initrd root
# ./root/fumagician/fumagician

Instead of using the manufacturer's program you might prefer to use nvme-cli and upload the firmware manually as explained in the previous section:

$ ls -1 root/fumagician/*.enc
root/fumagician/1B2QJXD7.enc
root/fumagician/DSRD.enc

The first file is the firmware.

Western Digital

Western Digital only supports updating via their Windows based Dashboard software. However, the firmware can be downloaded directly if you know where to look.[6]

First, navigate to the list of all drives and find your drive (model=model_number).

Under your particular drive model there will be one or more <url> entries. If there are multiple URLs then you may need to try each one using the directions below and check the <dependency> tag for your current firmware version.

Now, download the drive-specific XML file:

$ curl https://wddashboarddownloads.wdc.com/url_entry

Inside this drive-specific XML file should be a <fwfile> tag with a xxxx.fluf filename. This is the name of the file you want; you can download it by replacing device_properties.xml from the previous URL with this filename.

A full URL example for a SN820X drive:

$ curl --remote-name https://wddashboarddownloads.wdc.com/wdDashboard/firmware/WD_BLACK_SN850X_2000GB/620331WD/620331WD.fluf

Once you have the .fluf file, updating can be performed using the generic flashing instructions. Be aware that this is not officially supported by Western Digital, may not work correctly, and could possibly damage your device. Be extra careful that you are updating with the correct drive and version of firmware.

Performance

Sector size

See Advanced Format#NVMe solid state drives.

Airflow

NVMe SSDs are known to be affected by high operating temperatures and will throttle performance over certain thresholds.[7]

Testing

Raw device performance tests can be run with hdparm:

# hdparm -Tt --direct /dev/nvme0n1

Power Saving (APST)

To check NVMe power states, install nvme-cli or nvme-cli-gitAUR, and run nvme get-feature /dev/nvme[0-9] -f 0x0c -H:

# nvme get-feature /dev/nvme0 -f 0x0c -H
get-feature:0xc (Autonomous Power State Transition), Current value:0x000001
        Autonomous Power State Transition Enable (APSTE): Enabled
        Auto PST Entries        .................

...

When APST is enabled the output should contain "Autonomous Power State Transition Enable (APSTE): Enabled" and there should be non-zero entries in the table below indicating the idle time before transitioning into each of the available states.

If APST is enabled but no non-zero states appear in the table, the latencies might be too high for any states to be enabled by default. The output of nvme id-ctrl /dev/nvme[0-9] (as the root user) should show the available non-operational power states of the NVME controller. If the total latency of any state (enlat + xlat) is greater than 25000 (25ms) you must pass a value at least that high as parameter default_ps_max_latency_us for the nvme_core kernel module. This should enable APST and make the table in nvme get-feature (as the root user) show the entries.

Troubleshooting

Controller failure due to broken APST support

Some NVMe devices may exhibit issues related to power saving (APST). This is a known issue for Kingston A2000 [8] as of firmware S5Z42105 and has previously been reported on Samsung NVMe drives (Linux v4.10) [9][10]

A failure renders the device unusable until system reset, with kernel logs similar to:

 nvme nvme0: I/O 566 QID 7 timeout, aborting
 nvme nvme0: I/O 989 QID 1 timeout, aborting
 nvme nvme0: I/O 990 QID 1 timeout, aborting
 nvme nvme0: I/O 840 QID 6 timeout, reset controller
 nvme nvme0: I/O 24 QID 0 timeout, reset controller
 nvme nvme0: Device not ready; aborting reset, CSTS=0x1
 ...
 nvme nvme0: Device not ready; aborting reset, CSTS=0x1
 nvme nvme0: Device not ready; aborting reset, CSTS=0x1
 nvme nvme0: failed to set APST feature (-19)

As a workaround, add the kernel parameter nvme_core.default_ps_max_latency_us=0 to completely disable APST, or set a custom threshold to disable specific states.

This article or section is out of date.

Reason: This kernel parameter may no longer be necessary with recent versions of Linux Kernel. (e.g., v4.14.221, v4.19.175, v5.4.97, v5.10.15, v5.11-rc7, and later). [11] (Discuss in Talk:Solid state drive/NVMe)

Since March 2021 a firmware update 9 from Kingston is available. As Kingston only supports Windows, downloads for Linux can be found via heise.de or github. It is expected that, as long as the kernel workaround is in place, the firmware update will not do much as the deepest powersaving states are not reached anyway.

# smartctl -c /dev/nvme0
Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.00W       -        -    0  0  0  0        0       0
 1 +     4.60W       -        -    1  1  1  1        0       0
 2 +     3.80W       -        -    2  2  2  2        0       0
 3 -   0.0450W       -        -    3  3  3  3     2000    2000
 4 -   0.0040W       -        -    4  4  4  4    15000   15000

The value passed is the maximum exit latency (Ex_Lat). For example, to disable PS4 set nvme_core.default_ps_max_latency_us=2000.

Controller failure due to broken suspend support

Some users (for example, see Laptop/HP) have reported suspend failures with certain NVMe drives. As above, the failure renders the device inoperable until system reset, with kernel messages

nvme nvme0: Device not ready; aborting reset, CSTS=0x3
nvme nvme0: Removing after probe failure status: -19

As a workaround, add the kernel parameter iommu=soft to use a software replacement for the hardware IOMMU. (For further details, see this documentation.) This has the potential to cause some slight processing overhead.

The factual accuracy of this article or section is disputed.

Reason: The following solution does not explain why it works, nor provide sources. (Discuss in Talk:Solid state drive/NVMe)

Also you can try kernel parameter amd_iommu=off or better amd_iommu=fullflush on HP laptops with AMD CPU and KIOXIA KBG40ZN* nvme's, after you get I/O error with messages like this:

Failed to rotate /var/log/journal/*/system.journal: Read-only file system
nvme nvme0: Device not ready; aborting reset, CSTS=0x3
BTRFS error (device nvme0n1): bdev /dev/nvme0n1p* errs: wr 2, rd 0, flush 0, corrupt 0, gen 0

See also