Difference between revisions of "ZFS"

From ArchWiki
Jump to: navigation, search
(renaming a zpool)
m (reworded and wrote in 3rd person)
(14 intermediate revisions by 4 users not shown)
Line 3: Line 3:
 
{{Article summary text|This page provides basic guidelines for installing the native ZFS Linux kernel module.}}
 
{{Article summary text|This page provides basic guidelines for installing the native ZFS Linux kernel module.}}
 
{{Article summary heading|Related}}
 
{{Article summary heading|Related}}
{{Article summary wiki|Installing Arch Linux on ZFS}}
+
{{Article summary wiki|Playing with ZFS}} - Learn ZFS by playing with a zpool made from virtual disks.
{{Article summary wiki|ZFS on FUSE}}
+
{{Article summary wiki|Installing Arch Linux on ZFS}} - Describes install Arch onto a ZFS root.
{{Article summary wiki|Playing with ZFS}}
+
{{Article summary wiki|ZFS on FUSE}} - ZFS via FUSE.
 +
{{Article summary heading|External Resources}}
 +
{{Article summary text|Aaron Toponce has authored a 17-part blog on ZFS which is an excellent read.}}
 +
{{Article summary link|1. VDEVs|https://pthree.org/2012/12/04/zfs-administration-part-i-vdevs/}}
 +
{{Article summary link|2. RAIDZ Levels|https://pthree.org/2012/12/05/zfs-administration-part-ii-raidz/}}
 +
{{Article summary link|3. The ZFS Intent Log|https://pthree.org/2012/12/06/zfs-administration-part-iii-the-zfs-intent-log/}}
 +
{{Article summary link|4. The ARC|https://pthree.org/2012/12/07/zfs-administration-part-iv-the-adjustable-replacement-cache/}}
 +
{{Article summary link|5. Import/export zpools|https://pthree.org/2012/12/10/zfs-administration-part-v-exporting-and-importing-zpools/}}
 +
{{Article summary link|6. Scrub and Resilver|https://pthree.org/2012/12/11/zfs-administration-part-vi-scrub-and-resilver/}}
 +
{{Article summary link|7. Zpool Properties|https://pthree.org/2012/12/12/zfs-administration-part-vii-zpool-properties/}}
 +
{{Article summary link|8. Zpool Best Practices|https://pthree.org/2012/12/13/zfs-administration-part-viii-zpool-best-practices-and-caveats/}}
 +
{{Article summary link|9. Copy on Write|https://pthree.org/2012/12/14/zfs-administration-part-ix-copy-on-write/}}
 +
{{Article summary link|10. Creating Filesystems|https://pthree.org/2012/12/17/zfs-administration-part-x-creating-filesystems/}}
 +
{{Article summary link|11. Compression and Deduplication|https://pthree.org/2012/12/18/zfs-administration-part-xi-compression-and-deduplication/}}
 +
{{Article summary link|12. Snapshots and Clones|https://pthree.org/2012/12/19/zfs-administration-part-xii-snapshots-and-clones/}}
 +
{{Article summary link|13. Send/receive Filesystems|https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/}}
 +
{{Article summary link|14. VZOLs|https://pthree.org/2012/12/21/zfs-administration-part-xiv-zvols/}}
 +
{{Article summary link|15. iSCSI, NFS, and Samba|https://pthree.org/2012/12/31/zfs-administration-part-xv-iscsi-nfs-and-samba/}}
 +
{{Article summary link|16. Get/Set Properties|https://pthree.org/2013/01/02/zfs-administration-part-xvi-getting-and-setting-properties/}}
 +
{{Article summary link|17. ZFS Best Practices|https://pthree.org/2013/01/03/zfs-administration-part-xvii-best-practices-and-caveats/}}
 +
{{Article summary link|Pingdom details how it backs up 5TB of MySQL data every day with ZFS|http://royal.pingdom.com/2013/06/04/zfs-backup/}}
 +
 
 +
 
 
{{Article summary end}}
 
{{Article summary end}}
 +
  
 
[[Wikipedia:ZFS|ZFS]] is an advanced filesystem created by [[Wikipedia:Sun Microsystems|Sun Microsystems]] (now owned by Oracle) and released for OpenSolaris in November 2005.  Features of ZFS include: pooled storage (integrated volume management -- zpool), [[Wikipedia:Copy-on-write|Copy-on-write]], [[Wikipedia:Snapshot (computer storage)|snapshots]], data integrity verification and automatic repair (scrubbing), [[Wikipedia:RAID-Z|RAID-Z]], and a maximum [[Wikipedia:Exabyte|16 Exabyte]] volume size.  ZFS is licensed under the [[Wikipedia:CDDL|Common Development and Distribution License]] (CDDL).
 
[[Wikipedia:ZFS|ZFS]] is an advanced filesystem created by [[Wikipedia:Sun Microsystems|Sun Microsystems]] (now owned by Oracle) and released for OpenSolaris in November 2005.  Features of ZFS include: pooled storage (integrated volume management -- zpool), [[Wikipedia:Copy-on-write|Copy-on-write]], [[Wikipedia:Snapshot (computer storage)|snapshots]], data integrity verification and automatic repair (scrubbing), [[Wikipedia:RAID-Z|RAID-Z]], and a maximum [[Wikipedia:Exabyte|16 Exabyte]] volume size.  ZFS is licensed under the [[Wikipedia:CDDL|Common Development and Distribution License]] (CDDL).
Line 15: Line 38:
  
 
==Playing with ZFS==
 
==Playing with ZFS==
The rest of this article cover basic setup and usage of ZFS on physical block devices (HDD and SSD for example).  Users wishing to experiment with ZFS on ''virtual block devices'' (known in ZFS terms as VDEVs) which can be simple files like ~/zfs0.img ~/zfs1.img ~/zfs2.img etc. with no possibility of real data loss are encouraged to see the [[Playing_with_ZFS]] article.  Common tasks like building a RAIDZ array, purposefully corrupting data and recovering it, etc. are covered.
+
The rest of this article cover basic setup and usage of ZFS on physical block devices (HDD and SSD for example).  Users wishing to experiment with ZFS on ''virtual block devices'' (known in ZFS terms as VDEVs) which can be simple files like ~/zfs0.img ~/zfs1.img ~/zfs2.img etc. with no possibility of real data loss are encouraged to see the [[Playing_with_ZFS]] article.  Common tasks like building a RAIDZ array, purposefully corrupting data and recovering it, snapshotting datasets, etc. are covered.
  
 
==Installation==
 
==Installation==
 
+
The requisite packages are available in the AUR and in an unofficial repo. Details are provided on the [[ZFS Installation]] article.
===Building from AUR===
+
 
+
The ZFS kernel module are related utils are available in the [[AUR]]; all are required:
+
*{{aur|spl-utils}}
+
*{{aur|spl}}
+
*{{aur|zfs-utils}}
+
*{{aur|zfs}}
+
 
+
{{note|The ZFS and SPL (Solaris Porting Layer is a Linux kernel module which provides many of the Solaris kernel APIs) kernel modules are tied to a specific kernel version. It would not be possible to apply any kernel updates until updated packages are uploaded to AUR or the archzfs repository.}}
+
 
+
Should you wish to update the core/linux package before the AUR/zfs and AUR/spl packages' dependency lists are updated, a possible work-around is to remove (uninstall) spl and zfs packages (the respective modules and file system may stay in-use), update the core/linux package, build + install zfs and spl packages - just do not forget to edit PKGBUILD and correct the core/linux version number in "depends" section to match the updated version). Finally, the system may be rebooted. [ This is only for the situation, when ZFS is not used for root filesystem. ]
+
 
+
===Unofficial repository===
+
 
+
For fast and effortless installation and updates, the [http://demizerone.com/archzfs "archzfs"] signed repository is available to add to your {{ic|pacman.conf}}:
+
 
+
{{hc|/etc/pacman.conf|<nowiki>
+
[demz-repo-core]
+
Server = http://demizerone.com/$repo/$arch</nowiki>
+
}}
+
 
+
The repository and packages are signed with the maintainer's PGP key which is verifiable here: http://demizerone.com. This key is not trusted by any of the Arch Linux master keys, so it will need to be locally signed before use. See [[pacman-key]].
+
 
+
Add the maintainer's key,
+
 
+
# pacman-key -r 0EE7A126
+
 
+
and locally sign to add it to the system's trust database,
+
 
+
# pacman-key --lsign-key 0EE7A126
+
 
+
Once the key has been signed, it is now possible to update the package database,
+
 
+
# pacman -Syyu
+
 
+
and install ZFS packages:
+
 
+
# pacman -S archzfs
+
 
+
===Archiso tracking repository===
+
 
+
ZFS can easily be used from within the archiso live environment by using the special archiso tracking repository for ZFS. This repository makes it easy to install Arch Linux on a root ZFS filesystem, or to mount ZFS pools from within an archiso live environment using an up-to-date live medium. To use this repository from the live environment, add the following server line to pacman.conf:
+
 
+
{{hc|/etc/pacman.conf|<nowiki>
+
[demz-repo-archiso]
+
Server = http://demizerone.com/$repo/$arch</nowiki>
+
}}
+
 
+
This repository and packages are also signed, so the key must be locally signed following the steps listed in the previous section before use. For a guide on how to install Arch Linux on to a root ZFS filesystem, see [[Installing Arch Linux on ZFS]].
+
  
 
==Configuration==
 
==Configuration==
Line 75: Line 49:
 
===mkinitramfs hook===
 
===mkinitramfs hook===
  
If you are using ZFS on your root filesystem, then you will need to add the ZFS hook to [[Mkinitcpio|mkinitcpio.conf]]; if you are not using ZFS for your root filesystem, then you do not need to add the ZFS hook.
+
Users wishing to use the root filesystem on ZFS need to add the ZFS hook to [[Mkinitcpio|mkinitcpio.conf]]. Others may omit this step.
  
You will need to change your [[kernel parameters]] to include the dataset you want to boot. You can use <code>zfs=bootfs</code> to use the ZFS bootfs (set via <code>zpool set bootfs=rpool/ROOT/arch rpool</code>) or you can set the [[kernel parameters]] to <code>zfs=<pool>/<dataset></code> to boot directly from a ZFS dataset.
+
Modify the [[kernel parameters]] to include the dataset holding the root filesystem by one of these methods:
 +
#Use <code>zfs=bootfs</code> to use the ZFS bootfs (set via <code>zpool set bootfs=rpool/ROOT/arch rpool</code>)
 +
#Set the [[kernel parameters]] to <code>zfs=<pool>/<dataset></code> to boot directly from a ZFS dataset.
  
 
To see all available options for the ZFS hook:
 
To see all available options for the ZFS hook:
Line 83: Line 59:
 
   $ mkinitcpio -H zfs
 
   $ mkinitcpio -H zfs
  
To use the mkinitcpio hook, you will need to add <code>zfs</code> to your <code>HOOKS</code> in <code>/etc/mkinitcpio.conf</code>:
+
To use the mkinitcpio hook, user need to add <code>zfs</code> to the <code>HOOKS</code> array in <code>/etc/mkinitcpio.conf</code>:
  
 
{{hc|/etc/mkinitcpio.conf|
 
{{hc|/etc/mkinitcpio.conf|
Line 93: Line 69:
 
{{note|It is not necessary to use the "fsck" hook with ZFS. ZFS automatically fixes any errors that occur within the filesystem. However, if the hook is required for another filesystem used on the system, such as ext4, the current ZFS packaging implementation does not yet properly handle fsck requests from mkinitcpio and an error is produced when generating a new ramdisk.}}
 
{{note|It is not necessary to use the "fsck" hook with ZFS. ZFS automatically fixes any errors that occur within the filesystem. However, if the hook is required for another filesystem used on the system, such as ext4, the current ZFS packaging implementation does not yet properly handle fsck requests from mkinitcpio and an error is produced when generating a new ramdisk.}}
  
It is important to place this after any hooks which are needed to prepare the drive before it is mounted. For example, if your ZFS volume is encrypted, then you will need to place encrypt before the zfs hook to unlock it first.
+
It is important to place this after any hooks which are needed to prepare the drive before it is mounted. For example, if the ZFS volume is encrypted, then place encrypt before the zfs hook to unlock it first.
  
 
Recreate the ramdisk:
 
Recreate the ramdisk:
Line 101: Line 77:
 
===Automatic Start===
 
===Automatic Start===
  
For ZFS to live by its "zero administration" namesake, the zfs daemon must be loaded at startup. A benefit to this is that it is not necessary to mount your zpool in {{ic|/etc/fstab}}; the zfs daemon imports and mounts one zfs pool automatically. The daemon mounts a zfs pool reading the file {{ic|/etc/zfs/zpool.cache}}, so the zfs pool that you want to automatically mounted must write the file there.
+
For ZFS to live by its "zero administration" namesake, the zfs daemon must be loaded at startup. A benefit to this is that it is not necessary to mount the zpool in {{ic|/etc/fstab}}; the zfs daemon imports and mounts one zfs pool automatically. The daemon mounts a zfs pool reading the file {{ic|/etc/zfs/zpool.cache}}.
  
 
Set a pool as to be automatically mounted by the zfs daemon:
 
Set a pool as to be automatically mounted by the zfs daemon:
Line 118: Line 94:
 
==Create a storage pool==
 
==Create a storage pool==
  
Use {{ic| # parted --list}} to see a list of all available drives. It is not necessary to partition your drives before creating the zfs filesystem, this will be done automatically. However, if you feel the need to completely wipe your drive before creating the filesystem, this can be easily done with the dd command.
+
Use {{ic| # parted --list}} to see a list of all available drives. It is not necessary nor recommended to partition the drives before creating the zfs filesystem.
  
  # dd if=/dev/zero of=/dev/<device>
+
Having identified the list of drives, it is now time to get the id's of the drives to add to the zpool. The [http://zfsonlinux.org/faq.html#WhatDevNamesShouldIUseWhenCreatingMyPool zfs on Linux developers recommend] using device ids when creating ZFS storage pools of less than 10 devices. To find the id's, simply:
 
+
It should not have to be stated, but be careful with this command!
+
 
+
Once you have the list of drives, it is now time to get the id's of the drives you will be using. The [http://zfsonlinux.org/faq.html#WhatDevNamesShouldIUseWhenCreatingMyPool zfs on Linux developers recommend] using device ids when creating ZFS storage pools of less than 10 devices. To find the id's for your device, simply:
+
  
 
   $ ls -lah /dev/disk/by-id/
 
   $ ls -lah /dev/disk/by-id/
Line 143: Line 115:
 
* '''-f''': Force creating the pool. This is to overcome the "EFI label error". See [[#does not contain an EFI label]].
 
* '''-f''': Force creating the pool. This is to overcome the "EFI label error". See [[#does not contain an EFI label]].
  
* '''-m''': The mount point of the pool. If this is not specified, than your pool will be mounted to {{ic|/<pool>}}.
+
* '''-m''': The mount point of the pool. If this is not specified, than the pool will be mounted to {{ic|/<pool>}}.
  
* '''pool''': This is the name of the pool. Change it to whatever you like.
+
* '''pool''': This is the name of the pool.
  
 
* '''raidz''': This is the type of virtual device that will be created from the pool of devices. Raidz is a special implementation of raid5. See [https://blogs.oracle.com/bonwick/entry/raid_z Jeff Bonwick's Blog -- RAID-Z] for more information about raidz.
 
* '''raidz''': This is the type of virtual device that will be created from the pool of devices. Raidz is a special implementation of raid5. See [https://blogs.oracle.com/bonwick/entry/raid_z Jeff Bonwick's Blog -- RAID-Z] for more information about raidz.
  
* '''ids''': The names of the drives or partitions that you want to include into your pool. Get it from {{ic|/dev/disk/by-id}}.
+
* '''ids''': The names of the drives or partitions that to include into the pool. Get it from {{ic|/dev/disk/by-id}}.
  
 
Here is an example for the full command:
 
Here is an example for the full command:
Line 159: Line 131:
 
   # zpool create -f -o ashift=12 -m /mnt/data bigdata raidz ata-ST3000DM001-9YN166_S1F0KDGY ata-ST3000DM001-9YN166_S1F0JKRR ata-ST3000DM001-9YN166_S1F0KBP8 ata-ST3000DM001-9YN166_S1F0JTM1
 
   # zpool create -f -o ashift=12 -m /mnt/data bigdata raidz ata-ST3000DM001-9YN166_S1F0KDGY ata-ST3000DM001-9YN166_S1F0JKRR ata-ST3000DM001-9YN166_S1F0KBP8 ata-ST3000DM001-9YN166_S1F0JTM1
  
If the command is successful, there will be no output. Using the {{ic|$ mount}} command will show that you pool is mounted. Using {{ic|# zpool status}} will show that your pool has been created.
+
If the command is successful, there will be no output. Using the {{ic|$ mount}} command will show that the pool is mounted. Using {{ic|# zpool status}} will show that the pool has been created.
  
 
{{hc|# zpool status|
 
{{hc|# zpool status|
Line 178: Line 150:
 
}}
 
}}
  
At this point it would be good to reboot your computer to make sure your ZFS pool is mounted at boot. It is best to deal with all errors before transferring your data.
+
At this point it would be good to reboot the machine to ensure that the ZFS pool is mounted at boot. It is best to deal with all errors before transferring data.
  
 
== Tuning ==
 
== Tuning ==
Line 190: Line 162:
 
Compression is just that, transparent compression of data.  Consult the man page for various options.  A recent advancement is the lz4 algorithm which offers excellent compression and performance.  Enable it (or any other) using the zfs command:
 
Compression is just that, transparent compression of data.  Consult the man page for various options.  A recent advancement is the lz4 algorithm which offers excellent compression and performance.  Enable it (or any other) using the zfs command:
 
  # zfs set compression=lz4 <pool>
 
  # zfs set compression=lz4 <pool>
 
{{Note|It is unclear if community/lz4 is needed to be installed for zfs to use lz4 compression.}}
 
  
 
Other options for zfs can be displayed again, using the zfs command:
 
Other options for zfs can be displayed again, using the zfs command:
Line 197: Line 167:
  
 
== Usage ==
 
== Usage ==
 +
 +
Users can optionally create a dataset under the zpool as opposed to manually creating directories under the zpool.  Datasets allow for an increased level of control (quotas for example) in addition to snapshots.  To be able to create and mount a dataset, a directory of the same name must not pre-exist in the zpool. To create a dataset, use:
 +
 +
  # zfs create <nameofzpool>/<nameofdataset>
 +
 +
It is then possible to apply ZFS specific attributes to the dataset. For example, one could assign a quota limit to a specific directory within a dataset:
 +
 +
  # zfs set quota=20G <nameofzpool>/<nameofdataset>/<directory>
  
 
To see all the commands available in ZFS, use :
 
To see all the commands available in ZFS, use :
Line 208: Line 186:
 
=== Scrub ===
 
=== Scrub ===
  
ZFS pools should be scrubbed at least once a week. To scrub your pool:
+
ZFS pools should be scrubbed at least once a week. To scrub the pool:
  
 
   # zpool scrub <pool>
 
   # zpool scrub <pool>
  
To do automatic scrubbing once a week, set the following line in your root crontab:
+
To do automatic scrubbing once a week, set the following line in the root crontab:
  
 
{{hc|# crontab -e|
 
{{hc|# crontab -e|
Line 220: Line 198:
 
}}
 
}}
  
Replace {{ic|<pool>}} with the name of your ZFS storage pool.
+
Replace {{ic|<pool>}} with the name of the ZFS pool.
  
 
=== Check zfs pool status ===
 
=== Check zfs pool status ===
  
To print a nice table with statistics about your ZFS pool, including and read/write errors, use
+
To print a nice table with statistics about the ZFS pool, including and read/write errors, use
  
 
   # zpool status -v
 
   # zpool status -v
Line 238: Line 216:
 
{{hc|# zpool status|no pools available}}
 
{{hc|# zpool status|no pools available}}
  
To find the name of your pool, see [[#Check zfs pool status]].
+
To find the name of the pool, see [[#Check zfs pool status]].
  
 
=== Export a storage pool ===
 
=== Export a storage pool ===
Line 255: Line 233:
 
  # zpool export oldname
 
  # zpool export oldname
 
  # zpool import oldname newname
 
  # zpool import oldname newname
 +
 +
=== Setting a Different Mount Point ===
 +
The mount point for a given zpool can be moved at will with one command:
 +
# zfs set mountpoint=/foo/bar poolname
  
 
=== Swap volume ===
 
=== Swap volume ===
  
ZFS does not allow to use swapfiles, but you can use a ZFS volume (ZVOL) as swap.  It is importart to set the ZVOL block size to match the system page size, which can be obtained by the <tt>getconf PAGESIZE</tt> command (default on x86_64 is 4KiB).  Other options useful for keeping the system running well in low-memory situations are keeping it always synced, turning off compression, and not caching the zvol data.
+
ZFS does not allow to use swapfiles, but users can use a ZFS volume (ZVOL) as swap.  It is importart to set the ZVOL block size to match the system page size, which can be obtained by the <tt>getconf PAGESIZE</tt> command (default on x86_64 is 4KiB).  Other options useful for keeping the system running well in low-memory situations are keeping it always synced, turning off compression, and not caching the zvol data.
  
 
Create a 8GiB zfs volume:
 
Create a 8GiB zfs volume:
Line 274: Line 256:
 
   # mkswap -f /dev/zvol/<pool>/swap
 
   # mkswap -f /dev/zvol/<pool>/swap
  
To make it permament you need to edit your {{ic|/etc/fstab}}.  ZVOLs support discard, which can potentionally help ZFS's block allocator and reduce fragmentation for all other datasets when your swap is not full.
+
To make it permanent, edit {{ic|/etc/fstab}}.  ZVOLs support discard, which can potentially help ZFS's block allocator and reduce fragmentation for all other datasets when/if swap is not full.
  
 
Add a line to {{ic|/etc/fstab}}:
 
Add a line to {{ic|/etc/fstab}}:
Line 282: Line 264:
 
=== Automatic snapshots ===
 
=== Automatic snapshots ===
  
The {{aur|zfs-auto-snapshot-git}} package from [[Arch User Repository|AUR]] provides a shell script to automate the management of snapshots, with each named by date and label (hourly, daily, etc), giving quick and convenient snapshotting of all your ZFS datasets.  The package also installs cron tasks for quarter-hourly, hourly, daily, weekly, and monthly snapshots.  You may want to adjust the --keep parameter from the defaults depending on how far back you'd like to go (the monthly script by default keeps data for up to a year).
+
The {{aur|zfs-auto-snapshot-git}} package from [[Arch User Repository|AUR]] provides a shell script to automate the management of snapshots, with each named by date and label (hourly, daily, etc), giving quick and convenient snapshotting of all ZFS datasets.  The package also installs cron tasks for quarter-hourly, hourly, daily, weekly, and monthly snapshots.  Optionally adjust the --keep parameter from the defaults depending on how far back the snapshots are to go (the monthly script by default keeps data for up to a year).
  
To prevent a dataset from being snapshotted at all, set <tt>com.sun:auto-snapshot=false</tt> on it.  Likewise, you can set more fine-grained control as well by label, so if you don't want any monthlies on a snapshot, for example, set <tt>com.sun:auto-snapshot:monthly=false</tt>.
+
To prevent a dataset from being snapshotted at all, set <tt>com.sun:auto-snapshot=false</tt> on it.  Likewise, set more fine-grained control as well by label, if, for example, no monthlies are to be kept on a snapshot, for example, set <tt>com.sun:auto-snapshot:monthly=false</tt>.
  
 
==Troubleshooting==
 
==Troubleshooting==
Line 302: Line 284:
 
   ZFS: No hostid found on kernel command line or /etc/hostid.
 
   ZFS: No hostid found on kernel command line or /etc/hostid.
  
This warning occurs because the ZFS module does not have access to the spl hosted. There are two solutions, for this. You can either place your spl hostid in the [[kernel parameters]] in your boot loader. For example, adding <code>spl.spl_hostid=0x00bab10c</code>.
+
This warning occurs because the ZFS module does not have access to the spl hosted. There are two solutions, for this. Either place the spl hostid in the [[kernel parameters]] in the boot loader. For example, adding <code>spl.spl_hostid=0x00bab10c</code>.
  
 
The other solution is to make sure that there is a hostid in <code>/etc/hostid</code>, and then regenerate the initramfs image. Which will copy the hostid into the initramfs image.
 
The other solution is to make sure that there is a hostid in <code>/etc/hostid</code>, and then regenerate the initramfs image. Which will copy the hostid into the initramfs image.
Line 312: Line 294:
 
==== Unexported pool ====
 
==== Unexported pool ====
  
If the new installation does not boot because the zpool cannot be imported, you will need to chroot into the installation and properly export the zpool. See [[ZFS#Emergency chroot repair with archzfs]].
+
If the new installation does not boot because the zpool cannot be imported, chroot into the installation and properly export the zpool. See [[ZFS#Emergency chroot repair with archzfs]].
  
 
Once inside the chroot environment, load the ZFS module and force import the zpool,
 
Once inside the chroot environment, load the ZFS module and force import the zpool,
Line 322: Line 304:
 
{{bc|# zpool export <pool>}}
 
{{bc|# zpool export <pool>}}
  
To see your available pools, use,
+
To see the available pools, use,
  
 
{{bc|# zpool status}}
 
{{bc|# zpool status}}
  
It is necessary to export a pool because of the way ZFS uses the hostid to track the system the zpool was created on. The hostid is generated partly based on your network setup. During the installation in the archiso your network configuration could be different generating a different hostid than the one contained in your new installation. Once the zfs filesystem is exported and then re-imported in the new installation, the hostid is reset. See [http://osdir.com/ml/zfs-discuss/2011-06/msg00227.html Re: Howto zpool import/export automatically? - msg#00227].
+
It is necessary to export a pool because of the way ZFS uses the hostid to track the system the zpool was created on. The hostid is generated partly based on the network setup. During the installation in the archiso the network configuration could be different generating a different hostid than the one contained in the new installation. Once the zfs filesystem is exported and then re-imported in the new installation, the hostid is reset. See [http://osdir.com/ml/zfs-discuss/2011-06/msg00227.html Re: Howto zpool import/export automatically? - msg#00227].
  
If ZFS complains about "pool may be in use" after every reboot, you should properly export pool as described above, and then rebuild ramdisk in normally booted system:
+
If ZFS complains about "pool may be in use" after every reboot, properly export pool as described above, and then rebuild ramdisk in normally booted system:
  
 
{{bc|# mkinitcpio -p linux}}
 
{{bc|# mkinitcpio -p linux}}
Line 334: Line 316:
 
==== Incorrect hostid ====
 
==== Incorrect hostid ====
  
First of all double check you actually exported the pool correctly. Exporting the zpool clears the hostid marking the ownership. So during the first boot the zpool should mount correctly. If it does not there is some other problem.
+
Double check that the pool is properly exported. Exporting the zpool clears the hostid marking the ownership. So during the first boot the zpool should mount correctly. If it does not there is some other problem.
  
Reboot again, if the zfs pool refuses to mount it means your hostid is not yet correctly set in the early boot phase and it confuses zfs. So you have to manually tell zfs the correct number, once the hostid is coherent across the reboots the zpool will mount correctly.
+
Reboot again, if the zfs pool refuses to mount it means the hostid is not yet correctly set in the early boot phase and it confuses zfs. Manually tell zfs the correct number, once the hostid is coherent across the reboots the zpool will mount correctly.
  
Boot using zfs_force and write down your hostid. This one is just an example.
+
Boot using zfs_force and write down the hostid. This one is just an example.
 
  % hostid
 
  % hostid
 
  0a0af0f8
 
  0a0af0f8
  
You can always ignore the check adding {{ic|zfs_force&#61;1}} in your [[kernel parameters]], but it is not advisable as a permanent solution.
+
Users can always ignore the check adding {{ic|zfs_force&#61;1}} in the [[kernel parameters]], but it is not advisable as a permanent solution.
  
 
== Tips and tricks ==
 
== Tips and tricks ==
Line 348: Line 330:
 
===Embed the archzfs packages into an archiso===
 
===Embed the archzfs packages into an archiso===
  
It is a good idea make an installation media with the needed software included. Otherwise, you will need the latest archiso installation media burned to a CD or a USB key.  
+
It is a good idea make an installation media with the needed software included. Otherwise, the latest archiso installation media burned to a CD or a USB key is required.  
  
 
To embed {{ic|zfs}} in the archiso, from an existing install, download the {{ic|archiso}} package.
 
To embed {{ic|zfs}} in the archiso, from an existing install, download the {{ic|archiso}} package.
Line 375: Line 357:
 
More informations about the process can be read in [http://kroweer.wordpress.com/2011/09/07/creating-a-custom-arch-linux-live-usb/ this guide] or in the [[Archiso]] article.
 
More informations about the process can be read in [http://kroweer.wordpress.com/2011/09/07/creating-a-custom-arch-linux-live-usb/ this guide] or in the [[Archiso]] article.
  
If you are installing onto a UEFI system, see [[Unified Extensible Firmware Interface#Create UEFI bootable USB from ISO]] for creating UEFI compatible installation media.
+
If installing onto a UEFI system, see [[Unified Extensible Firmware Interface#Create UEFI bootable USB from ISO]] for creating UEFI compatible installation media.
  
 
=== Emergency chroot repair with archzfs ===
 
=== Emergency chroot repair with archzfs ===
  
Here is how to use the archiso to get into your ZFS filesystem for maintenance.
+
Here is how to use the archiso to get into the ZFS filesystem for maintenance.
  
Boot the latest archiso and bring up your network:
+
Boot the latest archiso and bring up the network:
  
 
     # wifi-menu
 
     # wifi-menu
Line 394: Line 376:
 
     # pacman -Syy
 
     # pacman -Syy
  
(optional) Install your favorite text editor:
+
(optional) Install a text editor:
  
 
     # pacman -S vim
 
     # pacman -S vim
Line 402: Line 384:
 
{{hc|/etc/pacman.conf|<nowiki>
 
{{hc|/etc/pacman.conf|<nowiki>
 
[demz-repo-archiso]
 
[demz-repo-archiso]
 +
SigLevel = Required
 
Server = http://demizerone.com/$repo/$arch</nowiki>}}
 
Server = http://demizerone.com/$repo/$arch</nowiki>}}
  
Line 416: Line 399:
 
     # modprobe zfs
 
     # modprobe zfs
  
Import your pool:
+
Import the pool:
  
 
     # zpool import -a -R /mnt
 
     # zpool import -a -R /mnt
  
Mount your boot partitions (if you have them):
+
Mount the boot partitions (if any):
  
 
     # mount /dev/sda2 /mnt/boot
 
     # mount /dev/sda2 /mnt/boot
 
     # mount /dev/sda1 /mnt/boot/efi
 
     # mount /dev/sda1 /mnt/boot/efi
  
Chroot into your ZFS filesystem:
+
Chroot into the ZFS filesystem:
  
 
     # arch-chroot /mnt /bin/bash
 
     # arch-chroot /mnt /bin/bash
  
Check your kernel version:
+
Check the kernel version:
  
 
     # pacman -Qi linux
 
     # pacman -Qi linux
 
     # uname -r
 
     # uname -r
  
uname will show the kernel version of the archiso. If they are different, you will need to run depmod (in the chroot) with the correct kernel version of your chroot installation:
+
uname will show the kernel version of the archiso. If they are different, run depmod (in the chroot) with the correct kernel version of the chroot installation:
  
 
     # depmod -a 3.6.9-1-ARCH (version gathered from pacman -Qi linux)
 
     # depmod -a 3.6.9-1-ARCH (version gathered from pacman -Qi linux)
  
This will load the correct kernel modules for the kernel version installed in your chroot installation.
+
This will load the correct kernel modules for the kernel version installed in the chroot installation.
  
Regenerate your ramdisk:
+
Regenerate the ramdisk:
  
 
     # mkinitcpio -p linux
 
     # mkinitcpio -p linux

Revision as of 19:01, 7 November 2013

Summary help replacing me
This page provides basic guidelines for installing the native ZFS Linux kernel module.
Related
Playing with ZFS - Learn ZFS by playing with a zpool made from virtual disks.
Installing Arch Linux on ZFS - Describes install Arch onto a ZFS root.
ZFS on FUSE - ZFS via FUSE.
External Resources
Aaron Toponce has authored a 17-part blog on ZFS which is an excellent read.
1. VDEVs
2. RAIDZ Levels
3. The ZFS Intent Log
4. The ARC
5. Import/export zpools
6. Scrub and Resilver
7. Zpool Properties
8. Zpool Best Practices
9. Copy on Write
10. Creating Filesystems
11. Compression and Deduplication
12. Snapshots and Clones
13. Send/receive Filesystems
14. VZOLs
15. iSCSI, NFS, and Samba
16. Get/Set Properties
17. ZFS Best Practices
Pingdom details how it backs up 5TB of MySQL data every day with ZFS



ZFS is an advanced filesystem created by Sun Microsystems (now owned by Oracle) and released for OpenSolaris in November 2005. Features of ZFS include: pooled storage (integrated volume management -- zpool), Copy-on-write, snapshots, data integrity verification and automatic repair (scrubbing), RAID-Z, and a maximum 16 Exabyte volume size. ZFS is licensed under the Common Development and Distribution License (CDDL).

Described as "The last word in filesystems" ZFS is stable, fast, secure, and future-proof. Being licensed under the GPL incompatible CDDL, it is not possible for ZFS to be distributed along with the Linux Kernel. This requirement, however, does not prevent a native Linux kernel module from being developed and distributed by a third party, as is the case with zfsonlinux.org (ZOL).

ZOL is a project funded by the Lawrence Livermore National Laboratory to develop a native Linux kernel module for its massive storage requirements and super computers.

Playing with ZFS

The rest of this article cover basic setup and usage of ZFS on physical block devices (HDD and SSD for example). Users wishing to experiment with ZFS on virtual block devices (known in ZFS terms as VDEVs) which can be simple files like ~/zfs0.img ~/zfs1.img ~/zfs2.img etc. with no possibility of real data loss are encouraged to see the Playing_with_ZFS article. Common tasks like building a RAIDZ array, purposefully corrupting data and recovering it, snapshotting datasets, etc. are covered.

Installation

The requisite packages are available in the AUR and in an unofficial repo. Details are provided on the ZFS Installation article.

Configuration

ZFS is considered a "zero administration" filesystem by its creators; therefore, configuring ZFS is very straight forward. Configuration is done primarily with two commands: zfs and zpool.

mkinitramfs hook

Users wishing to use the root filesystem on ZFS need to add the ZFS hook to mkinitcpio.conf. Others may omit this step.

Modify the kernel parameters to include the dataset holding the root filesystem by one of these methods:

  1. Use zfs=bootfs to use the ZFS bootfs (set via zpool set bootfs=rpool/ROOT/arch rpool)
  2. Set the kernel parameters to zfs=<pool>/<dataset> to boot directly from a ZFS dataset.

To see all available options for the ZFS hook:

 $ mkinitcpio -H zfs

To use the mkinitcpio hook, user need to add zfs to the HOOKS array in /etc/mkinitcpio.conf:

/etc/mkinitcpio.conf
...
HOOKS="base udev autodetect modconf encrypt zfs filesystems usbinput"
...
Note: It is not necessary to use the "fsck" hook with ZFS. ZFS automatically fixes any errors that occur within the filesystem. However, if the hook is required for another filesystem used on the system, such as ext4, the current ZFS packaging implementation does not yet properly handle fsck requests from mkinitcpio and an error is produced when generating a new ramdisk.

It is important to place this after any hooks which are needed to prepare the drive before it is mounted. For example, if the ZFS volume is encrypted, then place encrypt before the zfs hook to unlock it first.

Recreate the ramdisk:

 # mkinitcpio -p linux

Automatic Start

For ZFS to live by its "zero administration" namesake, the zfs daemon must be loaded at startup. A benefit to this is that it is not necessary to mount the zpool in /etc/fstab; the zfs daemon imports and mounts one zfs pool automatically. The daemon mounts a zfs pool reading the file /etc/zfs/zpool.cache.

Set a pool as to be automatically mounted by the zfs daemon:

# zpool set cachefile=/etc/zfs/zpool.cache <pool>

Systemd

Enable the service so it is automatically started at boot time:

 # systemctl enable zfs.service

To manually start the daemon:

 # systemctl start zfs.service

Create a storage pool

Use # parted --list to see a list of all available drives. It is not necessary nor recommended to partition the drives before creating the zfs filesystem.

Having identified the list of drives, it is now time to get the id's of the drives to add to the zpool. The zfs on Linux developers recommend using device ids when creating ZFS storage pools of less than 10 devices. To find the id's, simply:

 $ ls -lah /dev/disk/by-id/

The ids should look similar to the following:

 lrwxrwxrwx 1 root root  9 Aug 12 16:26 ata-ST3000DM001-9YN166_S1F0JKRR -> ../../sdc
 lrwxrwxrwx 1 root root  9 Aug 12 16:26 ata-ST3000DM001-9YN166_S1F0JTM1 -> ../../sde
 lrwxrwxrwx 1 root root  9 Aug 12 16:26 ata-ST3000DM001-9YN166_S1F0KBP8 -> ../../sdd
 lrwxrwxrwx 1 root root  9 Aug 12 16:26 ata-ST3000DM001-9YN166_S1F0KDGY -> ../../sdb

Now, finally, create the ZFS pool:

 # zpool create -f -m <mount> <pool> raidz <ids>
  • create: subcommand to create the pool.
  • -m: The mount point of the pool. If this is not specified, than the pool will be mounted to /<pool>.
  • pool: This is the name of the pool.
  • raidz: This is the type of virtual device that will be created from the pool of devices. Raidz is a special implementation of raid5. See Jeff Bonwick's Blog -- RAID-Z for more information about raidz.
  • ids: The names of the drives or partitions that to include into the pool. Get it from /dev/disk/by-id.

Here is an example for the full command:

 # zpool create -f -m /mnt/data bigdata raidz ata-ST3000DM001-9YN166_S1F0KDGY ata-ST3000DM001-9YN166_S1F0JKRR ata-ST3000DM001-9YN166_S1F0KBP8 ata-ST3000DM001-9YN166_S1F0JTM1

In case Advanced Format disks are used which have a native sector size of 4096 bytes instead of 512 bytes, the automated sector size detection algorithm of ZFS might detect 512 bytes because the backwards compatibility with legacy systems. This would result in degraded performance. To make sure a correct sector size is used, the ashift=12 option should be used (See the ZFS on Linux FAQ). The full command would in this case be:

 # zpool create -f -o ashift=12 -m /mnt/data bigdata raidz ata-ST3000DM001-9YN166_S1F0KDGY ata-ST3000DM001-9YN166_S1F0JKRR ata-ST3000DM001-9YN166_S1F0KBP8 ata-ST3000DM001-9YN166_S1F0JTM1

If the command is successful, there will be no output. Using the $ mount command will show that the pool is mounted. Using # zpool status will show that the pool has been created.

# zpool status
  pool: bigdata
 state: ONLINE
 scan: none requested
config:

        NAME                                       STATE     READ WRITE CKSUM
        bigdata                                    ONLINE       0     0     0
          -0                                       ONLINE       0     0     0
            ata-ST3000DM001-9YN166_S1F0KDGY-part1  ONLINE       0     0     0
            ata-ST3000DM001-9YN166_S1F0JKRR-part1  ONLINE       0     0     0
            ata-ST3000DM001-9YN166_S1F0KBP8-part1  ONLINE       0     0     0
            ata-ST3000DM001-9YN166_S1F0JTM1-part1  ONLINE       0     0     0

errors: No known data errors

At this point it would be good to reboot the machine to ensure that the ZFS pool is mounted at boot. It is best to deal with all errors before transferring data.

Tuning

Although many knobs are available on a zfs pool, there are two major ones user can consider:

  • atime
  • compression

Atime is enabled by default but for most users, it represents superfluous writes to the zpool and it can be disabled using the zfs command:

# zfs set atime=off <pool>

Compression is just that, transparent compression of data. Consult the man page for various options. A recent advancement is the lz4 algorithm which offers excellent compression and performance. Enable it (or any other) using the zfs command:

# zfs set compression=lz4 <pool>

Other options for zfs can be displayed again, using the zfs command:

# sudo zfs get all <pool>

Usage

Users can optionally create a dataset under the zpool as opposed to manually creating directories under the zpool. Datasets allow for an increased level of control (quotas for example) in addition to snapshots. To be able to create and mount a dataset, a directory of the same name must not pre-exist in the zpool. To create a dataset, use:

 # zfs create <nameofzpool>/<nameofdataset>

It is then possible to apply ZFS specific attributes to the dataset. For example, one could assign a quota limit to a specific directory within a dataset:

 # zfs set quota=20G <nameofzpool>/<nameofdataset>/<directory>

To see all the commands available in ZFS, use :

 $ man zfs

or:

 $ man zpool

Scrub

ZFS pools should be scrubbed at least once a week. To scrub the pool:

 # zpool scrub <pool>

To do automatic scrubbing once a week, set the following line in the root crontab:

# crontab -e
...
30 19 * * 5 zpool scrub <pool>
...

Replace <pool> with the name of the ZFS pool.

Check zfs pool status

To print a nice table with statistics about the ZFS pool, including and read/write errors, use

 # zpool status -v

Destroy a storage pool

ZFS makes it easy to destroy a mounted storage pool, removing all metadata about the ZFS device. This command destroys any data contained in the pool:

 # zpool destroy <pool>

And now when checking the status:

# zpool status
no pools available

To find the name of the pool, see #Check zfs pool status.

Export a storage pool

If a storage pool is to be used on another system, it will first need to be exported. It is also necessary to export a pool if it has been imported from the archiso as the hostid is different in the archiso as it is in the booted system. The zpool command will refuse to import any storage pools that have not been exported. It is possible to force the import with the -f argument, but this is considered bad form.

Any attempts made to import an un-exported storage pool will result in an error stating the storage pool is in use by another system. This error can be produced at boot time abruptly abandoning the system in the busybox console and requiring an archiso to do an emergency repair by either exporting the pool, or adding the zfs_force=1 to the kernel boot parameters (which is not ideal). See #On boot the zfs pool does not mount stating: "pool may be in use from other system"

To export a pool,

 # zpool export bigdata

Rename a Zpool

Renaming a zpool that is already created is accomplished in 2 steps:

# zpool export oldname
# zpool import oldname newname

Setting a Different Mount Point

The mount point for a given zpool can be moved at will with one command:

# zfs set mountpoint=/foo/bar poolname

Swap volume

ZFS does not allow to use swapfiles, but users can use a ZFS volume (ZVOL) as swap. It is importart to set the ZVOL block size to match the system page size, which can be obtained by the getconf PAGESIZE command (default on x86_64 is 4KiB). Other options useful for keeping the system running well in low-memory situations are keeping it always synced, turning off compression, and not caching the zvol data.

Create a 8GiB zfs volume:


 # zfs create -V 8G -b $(getconf PAGESIZE) \
              -o compression=off \
              -o primarycache=metadata \
              -o secondarycache=none \
              -o sync=always \
              -o com.sun:auto-snapshot=false <pool>/swap

Prepare it as swap partition:

 # mkswap -f /dev/zvol/<pool>/swap

To make it permanent, edit /etc/fstab. ZVOLs support discard, which can potentially help ZFS's block allocator and reduce fragmentation for all other datasets when/if swap is not full.

Add a line to /etc/fstab:

 /dev/zvol/<pool>/swap none swap discard 0 0

Automatic snapshots

The zfs-auto-snapshot-gitAUR package from AUR provides a shell script to automate the management of snapshots, with each named by date and label (hourly, daily, etc), giving quick and convenient snapshotting of all ZFS datasets. The package also installs cron tasks for quarter-hourly, hourly, daily, weekly, and monthly snapshots. Optionally adjust the --keep parameter from the defaults depending on how far back the snapshots are to go (the monthly script by default keeps data for up to a year).

To prevent a dataset from being snapshotted at all, set com.sun:auto-snapshot=false on it. Likewise, set more fine-grained control as well by label, if, for example, no monthlies are to be kept on a snapshot, for example, set com.sun:auto-snapshot:monthly=false.

Troubleshooting

does not contain an EFI label

The following error will occur when attempting to create a zfs filesystem,

 /dev/disk/by-id/<id> does not contain an EFI label but it may contain partition

The way to overcome this is to use -f with the zfs create command.

No hostid found

An error that occurs at boot with the following lines appearing before initscript output:

 ZFS: No hostid found on kernel command line or /etc/hostid.

This warning occurs because the ZFS module does not have access to the spl hosted. There are two solutions, for this. Either place the spl hostid in the kernel parameters in the boot loader. For example, adding spl.spl_hostid=0x00bab10c.

The other solution is to make sure that there is a hostid in /etc/hostid, and then regenerate the initramfs image. Which will copy the hostid into the initramfs image.

 # mkinitcpio -p linux

On boot the zfs pool does not mount stating: "pool may be in use from other system"

Unexported pool

If the new installation does not boot because the zpool cannot be imported, chroot into the installation and properly export the zpool. See ZFS#Emergency chroot repair with archzfs.

Once inside the chroot environment, load the ZFS module and force import the zpool,

# zpool import -a -f

now export the pool:

# zpool export <pool>

To see the available pools, use,

# zpool status

It is necessary to export a pool because of the way ZFS uses the hostid to track the system the zpool was created on. The hostid is generated partly based on the network setup. During the installation in the archiso the network configuration could be different generating a different hostid than the one contained in the new installation. Once the zfs filesystem is exported and then re-imported in the new installation, the hostid is reset. See Re: Howto zpool import/export automatically? - msg#00227.

If ZFS complains about "pool may be in use" after every reboot, properly export pool as described above, and then rebuild ramdisk in normally booted system:

# mkinitcpio -p linux

Incorrect hostid

Double check that the pool is properly exported. Exporting the zpool clears the hostid marking the ownership. So during the first boot the zpool should mount correctly. If it does not there is some other problem.

Reboot again, if the zfs pool refuses to mount it means the hostid is not yet correctly set in the early boot phase and it confuses zfs. Manually tell zfs the correct number, once the hostid is coherent across the reboots the zpool will mount correctly.

Boot using zfs_force and write down the hostid. This one is just an example.

% hostid
0a0af0f8

Users can always ignore the check adding zfs_force=1 in the kernel parameters, but it is not advisable as a permanent solution.

Tips and tricks

Embed the archzfs packages into an archiso

It is a good idea make an installation media with the needed software included. Otherwise, the latest archiso installation media burned to a CD or a USB key is required.

To embed zfs in the archiso, from an existing install, download the archiso package.

# pacman -S archiso

Start the process:

# cp -r /usr/share/archiso/configs/releng /root/media

Edit the packages.x86_64 file adding those lines:

spl-utils
spl
zfs-utils
zfs

Edit the pacman.conf file adding those lines (TODO, correctly embed keys in the installation media?):

[demz-repo-archiso]
SigLevel = Never
Server = http://demizerone.com/$repo/$arch

Add other packages in packages.both, packages.i686, or packages.x86_64 if needed and create the image.

# ./build.sh -v

The image will be in the /root/media/out directory.

More informations about the process can be read in this guide or in the Archiso article.

If installing onto a UEFI system, see Unified Extensible Firmware Interface#Create UEFI bootable USB from ISO for creating UEFI compatible installation media.

Emergency chroot repair with archzfs

Here is how to use the archiso to get into the ZFS filesystem for maintenance.

Boot the latest archiso and bring up the network:

   # wifi-menu
   # ip link set eth0 up

Test the network connection:

   # ping google.com

Sync the pacman package database:

   # pacman -Syy

(optional) Install a text editor:

   # pacman -S vim

Add archzfs archiso repository to pacman.conf:

/etc/pacman.conf
[demz-repo-archiso]
SigLevel = Required
Server = http://demizerone.com/$repo/$arch

Sync the pacman package database:

   # pacman -Syy

Install the ZFS package group:

   # pacman -S archzfs

Load the ZFS kernel modules:

   # modprobe zfs

Import the pool:

   # zpool import -a -R /mnt

Mount the boot partitions (if any):

   # mount /dev/sda2 /mnt/boot
   # mount /dev/sda1 /mnt/boot/efi

Chroot into the ZFS filesystem:

   # arch-chroot /mnt /bin/bash

Check the kernel version:

   # pacman -Qi linux
   # uname -r

uname will show the kernel version of the archiso. If they are different, run depmod (in the chroot) with the correct kernel version of the chroot installation:

   # depmod -a 3.6.9-1-ARCH (version gathered from pacman -Qi linux)

This will load the correct kernel modules for the kernel version installed in the chroot installation.

Regenerate the ramdisk:

   # mkinitcpio -p linux

There should be no errors.

See also