https://wiki.archlinux.org/api.php?action=feedcontributions&user=Alaskanarcher&feedformat=atomArchWiki - User contributions [en]2024-03-28T09:53:15ZUser contributionsMediaWiki 1.41.0https://wiki.archlinux.org/index.php?title=Install_Arch_Linux_on_ZFS&diff=574318Install Arch Linux on ZFS2019-05-30T20:07:17Z<p>Alaskanarcher: Native encryption is now part of the stable releases from 0.8.0 and on. Remove warning about it not being stable, and update version requirements.</p>
<hr />
<div>[[Category:Installation process]]<br />
[[ja:ZFS に Arch Linux をインストール]]<br />
{{Related articles start}}<br />
{{Related|ZFS}}<br />
{{Related|Experimenting with ZFS}}<br />
{{Related articles end}}<br />
This article details the steps required to install Arch Linux onto a ZFS root filesystem.<br />
<br />
== Installation ==<br />
<br />
See [[ZFS#Installation]] for installing the ZFS packages. If installing Arch Linux onto ZFS from the archiso, it would be easier to use the [[Unofficial user repositories#archzfs|archzfs]] repository.<br />
<br />
There is also a interactive installer, [https://github.com/danboid/ALEZ ALEZ], which provides an easy way to get a ZFS system up and running if you do not require much customization.<br />
<br />
=== Embedding archzfs into archiso ===<br />
<br />
See [[ZFS#Embed_the_archzfs_packages_into_an_archiso|ZFS]] article.<br />
<br />
== Partition the destination drive ==<br />
<br />
Review [[Partitioning]] for information on determining the partition table type to use for ZFS. ZFS supports GPT and MBR partition tables.<br />
<br />
ZFS manages its own partitions, so only a basic partition table scheme is required. The partition that will contain the ZFS filesystem should be of the type {{ic|bf00}}, or "Solaris Root".<br />
<br />
Drives larger than 2TB require a GPT partition table. GRUB on BIOS/GPT configurations require a small (1~2MiB) BIOS boot partition to embed its image of boot code.<br />
<br />
Depending upon your machine's firmware or your choice of boot mode, booting may or may not require an EFI partition. On a BIOS machine (or a UEFI machine booting in legacy mode) EFI partition is not required. Consult [[Arch boot process#Boot loader]] for more information.<br />
<br />
=== Partition scheme ===<br />
<br />
Here is an example of a basic partition scheme that could be employed for your ZFS root install on a BIOS/MBR installation using GRUB:<br />
<br />
{{bc|<nowiki><br />
Part Size Type<br />
---- ---- -------------------------<br />
1 XXXG Solaris Root (bf00)</nowiki><br />
}}<br />
<br />
Using GRUB on a BIOS (or UEFI machine in legacy boot mode) machine but using a GPT partition table:<br />
<br />
{{bc|<nowiki><br />
Part Size Type<br />
---- ---- -------------------------<br />
1 2M BIOS boot partition (ef02)<br />
2 XXXG Solaris Root (bf00)</nowiki><br />
}}<br />
<br />
Another example, this time using a UEFI-specific bootloader (such as [[rEFInd]]) and GPT:<br />
<br />
{{bc|<nowiki><br />
Part Size Type<br />
---- ---- -------------------------<br />
1 100M EFI boot partition (ef00)<br />
2 XXXG Solaris Root (bf00)</nowiki><br />
}}<br />
<br />
ZFS does not support swap files. If you require a swap partition, see [[ZFS#Swap volume]] for creating a swap ZVOL.<br />
<br />
{{Tip|Bootloaders with support for ZFS are described in [[#Install and configure the bootloader]].}}<br />
{{Warning|Several GRUB bugs ([https://savannah.gnu.org/bugs/?42861 bug #42861], [https://github.com/zfsonlinux/grub/issues/5 zfsonlinux/grub/issues/5]) complicate installing it on ZFS partitions, see [[#Install and configure the bootloader]] for a workaround}}<br />
<br />
=== Example parted commands ===<br />
Here are some example commands to partition a drive for the second scenario above ie using BIOS/legacy boot mode with a GPT partition table and a (slighty more than) 1MB BIOS boot partition for GRUB:<br />
<br />
# parted /dev/sdx<br />
(parted)mklabel gpt<br />
(parted)mkpart non-fs 0% 2<br />
(parted)mkpart primary 2 100%<br />
(parted)set 1 bios_grub on<br />
(parted)set 2 boot on<br />
(parted)quit<br />
<br />
You can achieve the above in a single command like so:<br />
<br />
parted --script /dev/sdx mklabel gpt mkpart non-fs 0% 2 mkpart primary 2 100% set 1 bios_grub on set 2 boot on<br />
<br />
If you are creating an EFI partition then that should have the boot flag set instead of the root partition.<br />
<br />
== Format the destination disk ==<br />
<br />
If you have opted for a boot partition as well as any other non-ZFS system partitions then format them. Do not do anything to the Solaris partition nor to the BIOS boot partition. ZFS will manage the first, and your bootloader the second.<br />
<br />
== Setup the ZFS filesystem ==<br />
<br />
First, make sure the ZFS modules are loaded,<br />
<br />
# modprobe zfs<br />
<br />
=== Create the root zpool ===<br />
<br />
# zpool create -f zroot /dev/disk/by-id/''id-to-partition-partx''<br />
<br />
{{Warning|<br />
* Always use id names when working with ZFS, otherwise import errors will occur.<br />
* The zpool command will normally activate all features. See [[ZFS#GRUB-compatible pool creation]] when using [[GRUB]].}}<br />
<br />
=== Create your datasets ===<br />
<br />
Instead of using conventional disk partitions, ZFS has the concept of datasets to manage your storage. Unlike disk partitions, datasets have no fixed size and allow for different attributes, such as compression, to be applied per dataset. Normal ZFS datasets are mounted automatically by ZFS whilst legacy datasets are required to be mounted using fstab or with the traditional mount command.<br />
<br />
One of the most useful features of ZFS is boot environments. Boot environments allow you to create a bootable snapshot of your system that you can revert to at any time instantly by simply rebooting and booting from that boot environment. This can make doing system updates much safer and is also incredibly useful for developing and testing software. In order to be able to use a boot environment manager such as [https://github.com/b333z/beadm beadm], or {{AUR|zedenv}} to manage boot environments, your datasets must be configured properly. Key to this are that you split your data directories (such as {{ic|/home}}) into datasets that are distinct from your system datasets and that you do not place data in the root of the pool as this cannot be moved afterwards. <br />
<br />
You should always create a dataset for at least your root filesystem and in nearly all cases you will also want {{ic|/home}} to be in a separate dataset. You may decide you want your logs to persist over boot environments. If you are a running any software that stores data outside of {{ic|/home}} (such as is the case for database servers) you should structure your datasets so that the data directories of the software you want to run are separated out from the root dataset.<br />
<br />
With these example commands, we will create a basic boot environment compatible configuration comprising of just root and {{ic|/home}} datasets with lz4 compression to save space and improve IO performance:<br />
<br />
# zfs create -o mountpoint=none zroot/data<br />
# zfs create -o mountpoint=none zroot/ROOT<br />
# zfs create -o compression=lz4 -o mountpoint=/ zroot/ROOT/default<br />
# zfs create -o compression=lz4 -o mountpoint=/home zroot/data/home<br />
<br />
{{Note|You will need to enable ACL support on the pool that will house {{ic|/var/log/journal}}, i.e. {{ic|1=zfs set acltype=posixacl ...}}. See [[Systemd#systemd-tmpfiles-setup.service fails to start at boot]] for more information.}}<br />
<br />
=== Configure the root filesystem ===<br />
<br />
If you have just created your zpool, it will be mounted in a dir at the root of your tree named after the pool (ie /zroot). If the following set commands fail, you may need to unmount any ZFS filesystems first:<br />
<br />
# zfs unmount -a<br />
<br />
Now set the mount points of the datasets:<br />
<br />
# zfs set mountpoint=/ zroot/ROOT/default<br />
# zfs set mountpoint=legacy zroot/data/home<br />
<br />
{{Note|{{ic|/etc/fstab}} mounts occur before zfs mounts, so don't use zfs mountpoints on directories with subfolders configured to be mounted by {{ic|/etc/fstab}}.}}<br />
<br />
and put them in {{ic|/etc/fstab}}<br />
{{hc|/etc/fstab|<br />
# <file system> <dir> <type> <options> <dump> <pass><br />
zroot/ROOT/default / zfs defaults,noatime 0 0<br />
zroot/data/home /home zfs defaults,noatime 0 0}}<br />
<br />
All legacy datasets must be listed in {{ic|/etc/fstab}} or they will not be mounted at boot.<br />
<br />
Set the bootfs property on the descendant root filesystem so the boot loader knows where to find the operating system.<br />
<br />
# zpool set bootfs=zroot/ROOT/default zroot<br />
<br />
Export the pool,<br />
<br />
# zpool export zroot<br />
<br />
{{Warning|Do not skip this, otherwise you will be required to use {{ic|-f}} when importing your pools. This unloads the imported pool.}}<br />
{{Note|This might fail if you added a swap partition. You need to turn it off with the ''swapoff'' command.}}<br />
<br />
Finally, re-import the pool,<br />
<br />
# zpool import -d /dev/disk/by-id -R /mnt zroot<br />
<br />
{{Note|{{ic|-d}} is not the actual device id, but the {{ic|/dev/by-id}} directory containing the symbolic links.<br />
If this command fails and you are asked to import your pool via its numeric ID, run {{ic|zpool import}} to <br />
find out the ID of your pool then use a command such as:<br />
{{ic|zpool import 9876543212345678910 -R /mnt zroot}}<br />
}}<br />
<br />
If there is an error in this step, you can export the pool to redo the command. The ZFS filesystem is now ready to use.<br />
<br />
Be sure to bring the {{ic|zpool.cache}} file into your new system. This is required later for the ZFS daemon to start.<br />
<br />
# cp /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache<br />
<br />
if you do not have {{ic|/etc/zfs/zpool.cache}}, create it:<br />
<br />
# zpool set cachefile=/etc/zfs/zpool.cache zroot<br />
<br />
== Install and configure Arch Linux ==<br />
<br />
Follow the following steps using the [[Installation guide]]. It will be noted where special consideration must be taken for ZFSonLinux.<br />
<br />
* First mount any legacy or non-ZFS boot or system partitions using the mount command.<br />
<br />
* Install the base system.<br />
<br />
* The procedure described in [[Installation guide#Fstab]] is usually overkill for ZFS. ZFS usually auto mounts its own partitions, so we do not need ZFS partitions in {{ic|fstab}} file, unless the user made legacy datasets of system directories. To generate the {{ic|fstab}} for filesystems, use:<br />
<br />
# genfstab -U -p /mnt >> /mnt/etc/fstab<br />
<br />
* Edit the {{ic|/etc/fstab}}:<br />
<br />
{{Note|<br />
* If you chose to create legacy datasets for system directories, keep them in this {{ic|fstab}}!<br />
* Comment out all non-legacy datasets apart from the root dataset, the swap file and the boot/EFI partition. It is a convention to replace the swap's uuid with {{ic|/dev/zvol/zroot/swap}}.<br />
}}<br />
<br />
* You need to add the [[Unofficial_user_repositories#archzfs|Arch ZFS]] repository to {{ic|/etc/pacman.conf}}, sign its key and [[install]] '''zfs-linux''' (or '''zfs-linux-lts''' if you are running the LTS kernel) within the arch-chroot before you can update the ramdisk with ZFS support.<br />
<br />
* When creating the initial ramdisk, first edit {{ic|/etc/mkinitcpio.conf}} and add {{ic|zfs}} before filesystems. Also, move {{ic|keyboard}} hook before {{ic|zfs}} so you can type in console if something goes wrong. You may also remove fsck (if you are not using Ext3 or Ext4). Your {{ic|HOOKS}} line should look something like this:<br />
HOOKS="base udev autodetect modconf block keyboard zfs filesystems"<br />
<br />
When using systemd in the initrd, you need to install {{AUR|mkinitcpio-sd-zfs}} and add the {{ic|sd-zfs}} hook after the {{ic|systemd}} hook instead of the {{ic|zfs}} hook. Keep in mind that this hook uses different kernel parameters than the default {{ic|zfs}} hook, more information can be found at the [https://github.com/dasJ/sd-zfs project page].<br />
<br />
{{Note|<br />
* If you are using a separate dataset for {{ic|/usr}} and have followed the instructions below, you must make sure you have the {{ic|usr}} hook enabled after {{ic|zfs}}, or your system will not boot.<br />
}}<br />
<br />
* [[Regenerate the initramfs]].<br />
<br />
== Install and configure the bootloader ==<br />
<br />
=== Using GRUB with BIOS and EFI motherboards ===<br />
<br />
Install GRUB onto your disk as instructed here: [[GRUB#BIOS systems]] or [[GRUB#UEFI systems]]. The GRUB [https://www.gnu.org/software/grub/manual/grub.html#Configuration manual] provides detailed information on manually configuring the software which you can supplement with [[GRUB]] and [[GRUB/Tips and tricks]].<br />
<br />
==== error: failed to get canonical path of ====<br />
<br />
{{ic|grub-mkconfig}} fails to properly generate entries for systems hosted on ZFS.<br />
<br />
# grub-mkconfig -o /boot/grub/grub.cfg<br />
/usr/bin/grub-probe: error: failed to get canonical path of `/dev/bus-Your_Disk_ID-part#'<br />
<br />
grub-install: error: failed to get canonical path of `/dev/bus-Your_Disk_ID-part#'<br />
<br />
To work around this you must set this environment variable: {{ic|1=ZPOOL_VDEV_NAME_PATH=1}}. For example:<br />
<br />
# ZPOOL_VDEV_NAME_PATH=1 grub-mkconfig -o /boot/grub/grub.cfg<br />
<br />
<br />
==== Booting your kernel and initrd from ZFS ====<br />
<br />
You may skip this section if you have your kernel and initrd on a separate {{ic|/boot}} partition using something like ext4 or vfat.<br />
<br />
Otherwise grub needs to load your kernel and initrd are from a ZFS dataset the kernel and initrd paths have to be in the following format:<br />
<br />
/dataset/@/actual/path <br />
<br />
Example with Arch installed on the root dataset:<br />
<br />
{{hc|/boot/grub/grub.cfg|<nowiki><br />
set timeout=5<br />
set default=0<br />
<br />
menuentry "Arch Linux" {<br />
search -u UUID<br />
linux /@/boot/vmlinuz-linux zfs=zroot rw<br />
initrd /@/boot/initramfs-linux.img<br />
}<br />
</nowiki>}}<br />
<br />
Example with Arch installed on a nested dataset:<br />
<br />
{{hc|/boot/grub/grub.cfg|<nowiki><br />
set timeout=5<br />
set default=0<br />
<br />
menuentry "Arch Linux" {<br />
search -u UUID<br />
linux /ROOT/default/@/boot/vmlinuz-linux zfs=zroot/ROOT/default rw <br />
initrd /ROOT/default/@/boot/initramfs-linux.img<br />
}<br />
</nowiki>}}<br />
<br />
Example with a separate non-ZFS /boot partition and Arch installed on a nested dataset: <br />
<br />
{{hc|/boot/grub/grub.cfg|<nowiki><br />
set timeout=5<br />
set default=0<br />
<br />
menuentry "Arch Linux" {<br />
search -u UUID<br />
linux /vmlinuz-linux zfs=zroot/ROOT/default rw<br />
initrd /initramfs-linux.img<br />
}<br />
</nowiki>}}<br />
<br />
=== Using rEFInd with UEFI motherboards ===<br />
<br />
Use {{ic|EFISTUB}} and {{ic|rEFInd}} for the UEFI boot loader. The kernel parameters in {{ic|refind_linux.conf}} for ZFS should include {{ic|1=zfs=bootfs}} or {{ic|1=zfs=zroot}} so the system can boot from ZFS. The {{ic|root}} and {{ic|rootfstype}} parameters are not needed.<br />
<br />
== Unmount and restart ==<br />
<br />
We are almost done!<br />
# exit<br />
# umount /mnt/boot (if you have a legacy boot partition)<br />
# zfs umount -a<br />
# zpool export zroot<br />
Now reboot.<br />
<br />
{{Warning|If you do not properly export the zpool, the pool will refuse to import in the ramdisk environment and you will be stuck at the busybox terminal.}}<br />
<br />
== After the first boot ==<br />
<br />
If everything went fine up to this point, your system will boot. Once.<br />
For your system to be able to reboot without issues, you need to enable the {{ic|zfs.target}} to auto mount the pools and set the hostid.<br />
<br />
For each pool you want automatically mounted execute:<br />
# zpool set cachefile=/etc/zfs/zpool.cache <pool><br />
Enable the target with [[systemd]]:<br />
# systemctl enable zfs.target<br />
<br />
{{Note|Beginning with ZOL version 0.6.5.8 the ZFS service unit files have been changed so that you need to explicitly enable any ZFS services you want to run.<br />
<br />
See [https://github.com/archzfs/archzfs/issues/72 https://github.com/archzfs/archzfs/issues/72] for more information.<br />
<br />
}}<br />
<br />
In order to mount zfs pools automatically on boot you need to enable the following services and targets:<br />
<br />
# systemctl enable zfs-import-cache<br />
# systemctl enable zfs-mount<br />
# systemctl enable zfs-import.target<br />
<br />
When running ZFS on root, the machine's hostid will not be available at the time of mounting the root filesystem. There are two solutions to this. You can either place your spl hostid in the [[kernel parameters]] in your boot loader. For example, adding {{ic|<nowiki>spl.spl_hostid=0x00bab10c</nowiki>}}, to get your number use the {{ic|hostid}} command.<br />
<br />
The other, and suggested, solution is to make sure that there is a hostid in {{ic|/etc/hostid}}, and then regenerate the initramfs image which will copy the hostid into the initramfs image. To write the hostid file safely you need to use the {{ic|zgenhostid}} command.<br />
<br />
To use the libc-generated hostid (recommended):<br />
# zgenhostid $(hostid)<br />
To use a custom hostid (must be hexadecimal and 8 characters long):<br />
# zgenhostid deadbeef<br />
To let the tool generate a hostid:<br />
# zgenhostid<br />
<br />
Don't forget to regenerate your image using [[mkinitcpio]]. Your system should work and reboot properly now.<br />
<br />
On some machines the export of the pool on shutdown sometimes fails. If your pool is only used from ''exactly one'' machine (no danger of accidently simultaneously importing it on two different machines), it is safe to add<br />
zfs_force=1<br />
to your kernel options, which forces the import of the pool.<br />
<br />
== Native encryption ==<br />
To use native ZFS encryption, you will need a recent enough zfs package like {{AUR|zfs-linux-git}} 0.7.0.r26 or newer, or {{AUR|zfs-linux}} 0.8.0 or newer, and embed it into the archiso.<br />
Then just follow the normal procedure shown before with the exception that you add the following parameters when creating the dataset:<br />
# zfs create -o encryption=on -o keyformat=passphrase -o mountpoint=none zroot/ROOT<br />
# zfs create -o encryption=on -o keyformat=passphrase -o mountpoint=none zroot/data<br />
If you want a single passphrase for both your root and home partition, encrypt only one dataset instead:<br />
# zfs create -o encryption=on -o keyformat=passphrase -o mountpoint=none zroot/encr<br />
# zfs create -o mountpoint=none zroot/encr/ROOT<br />
# zfs create -o mountpoint=none zroot/encr/data<br />
<br />
When importing the pool use {{ic|-l}}, to decrypt all datasets<br />
# zpool import -d /dev/disk/by-id -R /mnt -l zroot<br />
<br />
On reboot, you will be asked for your passphrase.<br />
<br />
== See also ==<br />
<br />
* [https://github.com/dajhorn/pkg-zfs/wiki/HOWTO-install-Ubuntu-to-a-Native-ZFS-Root-Filesystem HOWTO install Ubuntu to a Native ZFS Root]<br />
* [http://lildude.co.uk/zfs-cheatsheet ZFS cheatsheet]<br />
* [http://www.funtoo.org/wiki/ZFS_Install_Guide Funtoo ZFS install guide]</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=ZFS&diff=574317ZFS2019-05-30T20:02:56Z<p>Alaskanarcher: Update note about native encryption only being available in -git packages. Native encryption is not included in the stable release 0.8.0.</p>
<hr />
<div>[[Category:File systems]]<br />
[[Category:Oracle]]<br />
[[ja:ZFS]]<br />
[[ru:ZFS]]<br />
[[zh-hans:ZFS]]<br />
{{Related articles start}}<br />
{{Related|File systems}}<br />
{{Related|ZFS/Virtual disks}}<br />
{{Related|Installing Arch Linux on ZFS}}<br />
{{Related articles end}}<br />
<br />
[[Wikipedia:ZFS|ZFS]] is an advanced filesystem created by [[Wikipedia:Sun Microsystems|Sun Microsystems]] (now owned by Oracle) and released for OpenSolaris in November 2005. <br />
<br />
Features of ZFS include: pooled storage (integrated volume management – zpool), [[Wikipedia:Copy-on-write|Copy-on-write]], [[Wikipedia:Snapshot (computer storage)|snapshots]], data integrity verification and automatic repair (scrubbing), [[Wikipedia:RAID-Z|RAID-Z]], a maximum [[Wikipedia:Exabyte|16 Exabyte]] file size, and a maximum 256 Quadrillion [[Wikipedia:Zettabyte|Zettabytes]] storage with no limit on number of filesystems (datasets) or files[http://docs.oracle.com/cd/E19253-01/819-5461/zfsover-2/index.html]. ZFS is licensed under the [[Wikipedia:CDDL|Common Development and Distribution License]] (CDDL).<br />
<br />
Described as [http://web.archive.org/web/20060428092023/http://www.sun.com/2004-0914/feature/ "The last word in filesystems"] ZFS is stable, fast, secure, and future-proof. Being licensed under the CDDL, and thus incompatible with GPL, it is not possible for ZFS to be distributed along with the Linux Kernel. This requirement, however, does not prevent a native Linux kernel module from being developed and distributed by a third party, as is the case with [http://zfsonlinux.org/ zfsonlinux.org] (ZOL).<br />
<br />
ZOL is a project funded by the [https://www.llnl.gov/ Lawrence Livermore National Laboratory] to develop a native Linux kernel module for its massive storage requirements and super computers.<br />
<br />
{{Note|<br />
Due to potential legal incompatibilities between CDDL license of ZFS code and GPL of the Linux kernel ([https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/ ],[[wikipedia:Common_Development_and_Distribution_License#GPL_compatibility|CDDL-GPL]],[[wikipedia:ZFS#Linux|ZFS in Linux]]) - ZFS development is not supported by the kernel.<br />
<br />
As a result:<br />
<br />
* ZFSonLinux project must keep up with Linux kernel versions. After making stable ZFSonLinux release - Arch ZFS maintainers release them.<br />
* This situation sometimes locks down the normal rolling update process by unsatisfied dependencies because the new kernel version, proposed by update, is unsupported by ZFSonLinux.<br />
}}<br />
<br />
== Installation ==<br />
<br />
=== General ===<br />
<br />
{{Warning|Unless you use the [[dkms]] versions of these packages, the ZFS and SPL kernel modules are tied to a specific kernel version. It would not be possible to apply any kernel updates until updated packages are uploaded to AUR or the [[Unofficial user repositories#archzfs|archzfs]] repository.}}<br />
<br />
{{Tip| You can [[downgrade]] your linux version to the one from [[Unofficial user repositories#archzfs|archzfs]] repo if your current kernel is newer.}}<br />
<br />
Install from the [[Arch User Repository]] or the [[Unofficial user repositories#archzfs|archzfs]] repository:<br />
<br />
* {{AUR|zfs-linux}} for [http://zfsonlinux.org/ stable] releases.<br />
* {{AUR|zfs-linux-git}} for [https://github.com/zfsonlinux/zfs/releases development] releases (with support of newer kernel versions).<br />
* {{AUR|zfs-linux-lts}} for stable releases for LTS kernels.<br />
* {{AUR|zfs-linux-lts-git}} for [https://github.com/zfsonlinux/zfs/releases development] releases for LTS kernels.<br />
* {{AUR|zfs-linux-hardened}} for stable releases for hardened kernels.<br />
* {{AUR|zfs-linux-hardened-git}} for [https://github.com/zfsonlinux/zfs/releases development] releases for hardened kernels.<br />
* {{AUR|zfs-linux-zen}} for stable releases for zen kernels.<br />
* {{AUR|zfs-linux-zen-git}} for [https://github.com/zfsonlinux/zfs/releases development] releases for zen kernels.<br />
* {{AUR|zfs-dkms}} for versions with dynamic kernel module support.<br />
* {{AUR|zfs-dkms-git}} for [https://github.com/zfsonlinux/zfs/releases development] releases for versions with dynamic kernel module support.<br />
<br />
These branches have (according to them) dependencies on the {{ic|zfs-utils}}, {{ic|spl}}, {{ic|spl-utils}} packages. SPL (Solaris Porting Layer) is a Linux Kernel module implementing Solaris APIs for ZFS compatibility.<br />
<br />
Test the installation by issuing {{ic|zpool status}} on the command line. If an "insmod" error is produced, try {{ic|depmod -a}}.<br />
<br />
=== Root on ZFS ===<br />
<br />
When performing an Arch install on ZFS, {{AUR|zfs-linux}} or {{AUR|zfs-dkms}} and its dependencies can be installed in the archiso environment as outlined in the previous section.<br />
<br />
It may be useful to prepare a [[#Embed the archzfs packages into an archiso|customized archiso]] with ZFS support builtin. For a much more detailed guide on installing Arch with ZFS as its root file system, see [[Installing Arch Linux on ZFS]].<br />
<br />
=== DKMS ===<br />
<br />
Users can make use of DKMS [[Dynamic Kernel Module Support]] to rebuild the ZFS modules automatically with every kernel upgrade. <br />
<br />
Install {{AUR|zfs-dkms}} or {{AUR|zfs-dkms-git}} and apply the post-install instructions given by these packages.<br />
<br />
{{Tip|Add an {{ic|IgnorePkg}} entry to [[pacman.conf]] to prevent these packages from upgrading when doing a regular update.}}<br />
<br />
== Experimenting with ZFS ==<br />
<br />
Users wishing to experiment with ZFS on ''virtual block devices'' (known in ZFS terms as VDEVs) which can be simple files like {{ic|~/zfs0.img}} {{ic|~/zfs1.img}} {{ic|~/zfs2.img}} etc. with no possibility of real data loss are encouraged to see the [[Experimenting with ZFS]] article. Common tasks like building a RAIDZ array, purposefully corrupting data and recovering it, snapshotting datasets, etc. are covered.<br />
<br />
== Configuration ==<br />
<br />
ZFS is considered a "zero administration" filesystem by its creators; therefore, configuring ZFS is very straight forward. Configuration is done primarily with two commands: {{ic|zfs}} and {{ic|zpool}}.<br />
<br />
=== Automatic Start ===<br />
<br />
For ZFS to live by its "zero administration" namesake, the zfs daemon must be loaded at startup. A benefit to this is that it is not necessary to mount the zpool in {{ic|/etc/fstab}}; the zfs daemon can import and mount zfs pools automatically. The daemon mounts the zfs pools reading the file {{ic|/etc/zfs/zpool.cache}}.<br />
<br />
For each [[#Importing a pool created by id|imported pool]] you want automatically mounted by the zfs daemon execute:<br />
<br />
# zpool set cachefile=/etc/zfs/zpool.cache <pool><br />
<br />
Enable the service so it is automatically started at boot time:<br />
<br />
# systemctl enable zfs.target<br />
<br />
To manually start the daemon:<br />
<br />
# systemctl start zfs.target<br />
<br />
{{Note|Beginning with ZOL version 0.6.5.8 the ZFS service unit files have been changed so that you need to explicitly enable any ZFS services you want to run. See [https://github.com/archzfs/archzfs/issues/72 https://github.com/archzfs/archzfs/issues/72] for more information.}}<br />
<br />
In order to mount zfs pools automatically on boot you need to enable the following services and targets:<br />
<br />
# systemctl enable zfs-import-cache<br />
# systemctl enable zfs-mount<br />
# systemctl enable zfs-import.target<br />
<br />
or, as explained on [https://github.com/archzfs/archzfs/issues/72 the GitHub issue], use the [https://www.freedesktop.org/software/systemd/man/systemd.preset.html systemd preset file]:<br />
<br />
# systemctl preset $(tail -n +2 /usr/lib/systemd/system-preset/50-zfs.preset | cut -d ' ' -f 2)<br />
<br />
== Creating a storage pool ==<br />
<br />
Make sure the wanted disks contain an empty [[GPT]]/[[MBR]] [[partition table]]. It is not necessary nor recommended to partition the drives before creating the ZFS filesystem.<br />
<br />
{{Note|If some or all device have been used in a software RAID set it is paramount to [[mdadm#Prepare the devices|erase any old RAID configuration information]].}}<br />
<br />
{{Warning|For [[Advanced Format]] Disks with 4KB sector size, an ashift of 12 is recommended for best performance. Advanced Format disks emulate a sector size of 512 bytes for compatibility with legacy systems, this causes ZFS to sometimes use an ashift option number that is not ideal. Once the pool has been created, the only way to change the ashift option is to recreate the pool. Using an ashift of 12 would also decrease available capacity. See [https://github.com/zfsonlinux/zfs/wiki/faq#performance-considerations 1.10 What’s going on with performance?], [https://github.com/zfsonlinux/zfs/wiki/faq#advanced-format-disks 1.15 How does ZFS on Linux handle Advanced Format disks?], and [http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks ZFS and Advanced Format disks].}}<br />
<br />
=== Identify disks ===<br />
<br />
[https://github.com/zfsonlinux/zfs/wiki/faq#selecting-dev-names-when-creating-a-pool ZFS on Linux] recommends using device IDs when creating ZFS storage pools of less than 10 devices. Use [[Persistent block device naming#by-id and by-path]] to identify the list of drives to be used for ZFS pool. <br />
<br />
The disk IDs should look similar to the following:<br />
<br />
{{hc|# ls -lh /dev/disk/by-id/|<br />
lrwxrwxrwx 1 root root 9 Aug 12 16:26 ata-ST3000DM001-9YN166_S1F0JKRR -> ../../sdc<br />
lrwxrwxrwx 1 root root 9 Aug 12 16:26 ata-ST3000DM001-9YN166_S1F0JTM1 -> ../../sde<br />
lrwxrwxrwx 1 root root 9 Aug 12 16:26 ata-ST3000DM001-9YN166_S1F0KBP8 -> ../../sdd<br />
lrwxrwxrwx 1 root root 9 Aug 12 16:26 ata-ST3000DM001-9YN166_S1F0KDGY -> ../../sdb}}<br />
<br />
{{Warning|If you create zpools using device names (e.g. {{ic|/dev/sda}},{{ic|/dev/sdb}},...) ZFS might not be able to detect zpools intermittently on boot.}}<br />
<br />
==== Using GPT labels ====<br />
<br />
{{Style|Missing references to [[Persistent block device naming]], it is useless to explain the differences (or even what they are) here.}}<br />
<br />
Disk labels and UUID can also be used for ZFS mounts by using [[GUID Partition Table|GPT]] partitions. ZFS drives have labels but Linux is unable to read them at boot. Unlike [[Master Boot Record|MBR]] partitions, GPT partitions directly support both UUID and labels independent of the format inside the partition. Partitioning rather than using the whole disk for ZFS offers two additional advantages. The OS does not generate bogus partition numbers from whatever unpredictable data ZFS has written to the partition sector, and if desired, you can easily over provision SSD drives, and slightly over provision spindle drives to ensure that different models with slightly different sector counts can zpool replace into your mirrors. This is a lot of organization and control over ZFS using readily available tools and techniques at almost zero cost.<br />
<br />
Use [[GUID Partition Table|gdisk]] to partition the all or part of the drive as a single partition. gdisk does not automatically name partitions so if partition labels are desired use gdisk command "c" to label the partitions. Some reasons you might prefer labels over UUID are: labels are easy to control, labels can be titled to make the purpose of each disk in your arrangement readily apparent, and labels are shorter and easier to type. These are all advantages when the server is down and the heat is on. GPT partition labels have plenty of space and can store most international characters [[wikipedia:GUID_Partition_Table#Partition_entries]] allowing large data pools to be labeled in an organized fashion.<br />
<br />
Drives partitioned with GPT have labels and UUID that look like this. <br />
<br />
# ls -l /dev/disk/by-partlabel<br />
# lrwxrwxrwx 1 root root 10 Apr 30 01:44 zfsdata1 -> ../../sdd1<br />
# lrwxrwxrwx 1 root root 10 Apr 30 01:44 zfsdata2 -> ../../sdc1<br />
# lrwxrwxrwx 1 root root 10 Apr 30 01:59 zfsl2arc -> ../../sda1<br />
<br />
# ls -l /dev/disk/by-partuuid<br />
# lrwxrwxrwx 1 root root 10 Apr 30 01:44 148c462c-7819-431a-9aba-5bf42bb5a34e -> ../../sdd1<br />
# lrwxrwxrwx 1 root root 10 Apr 30 01:59 4f95da30-b2fb-412b-9090-fc349993df56 -> ../../sda1<br />
# lrwxrwxrwx 1 root root 10 Apr 30 01:44 e5ccef58-5adf-4094-81a7-3bac846a885f -> ../../sdc1<br />
<br />
{{Tip|To minimize typing and copy/paste errors, set a local variable with the target PARTUUID: {{ic|<nowiki>$ UUID=$( lsblk --noheadings --output PARTUUID /dev/sd</nowiki>''XY''<nowiki> )</nowiki>}} }}<br />
<br />
=== Creating ZFS pools ===<br />
<br />
To create a ZFS pool:<br />
<br />
# zpool create -f -m <mount> <pool> [raidz(2|3)|mirror] <ids><br />
<br />
{{Tip|One may want to read [[#Advanced Format disks]] first as it may be recommend to set {{ic|ashift}} on pool creation.}}<br />
<br />
* '''create''': subcommand to create the pool.<br />
<br />
* '''-f''': Force creating the pool. This is to overcome the "EFI label error". See [[#Does not contain an EFI label]].<br />
<br />
* '''-m''': The mount point of the pool. If this is not specified, then the pool will be mounted to {{ic|/<pool>}}.<br />
<br />
* '''pool''': This is the name of the pool.<br />
<br />
* '''raidz(2|3)|mirror''': This is the type of virtual device that will be created from the pool of devices, raidz is a single disk of parity, raidz2 for 2 disks of parity and raidz3 for 3 disks of parity, similar to raid5 and raid6. Also available is '''mirror''', which is similar to raid1 or raid10, but is not constrained to just 2 device. If not specified, each device will be added as a vdev which is similar to raid0. After creation, a device can be added to each single drive vdev to turn it into a mirror, which can be useful for migrating data.<br />
<br />
* '''ids''': The [[Persistent_block_device_naming#by-id_and_by-path|ID's]] of the drives or partitions that to include into the pool.<br />
<br />
Create pool with single raidz vdev:<br />
<br />
# zpool create -f -m /mnt/data bigdata \<br />
raidz \<br />
ata-ST3000DM001-9YN166_S1F0KDGY \<br />
ata-ST3000DM001-9YN166_S1F0JKRR \<br />
ata-ST3000DM001-9YN166_S1F0KBP8 \<br />
ata-ST3000DM001-9YN166_S1F0JTM1<br />
<br />
Create pool with two mirror vdevs:<br />
<br />
# zpool create -f -m /mnt/data bigdata \<br />
mirror \<br />
ata-ST3000DM001-9YN166_S1F0KDGY \<br />
ata-ST3000DM001-9YN166_S1F0JKRR \<br />
mirror \<br />
ata-ST3000DM001-9YN166_S1F0KBP8 \<br />
ata-ST3000DM001-9YN166_S1F0JTM1<br />
<br />
==== Advanced Format disks ====<br />
<br />
At pool creation, '''ashift=12''' should always be used, except with SSDs that have 8k sectors where '''ashift=13''' is correct. A vdev of 512 byte disks using 4k sectors will not experience performance issues, but a 4k disk using 512 byte sectors will. Since '''ashift''' cannot be changed after pool creation, even a pool with only 512 byte disks should use 4k because those disks may need to be replaced with 4k disks or the pool may be expanded by adding a vdev composed of 4k disks. Because correct detection of 4k disks is not reliable, {{ic|<nowiki>-o ashift=12</nowiki>}} should always be specified during pool creation. See the [https://github.com/zfsonlinux/zfs/wiki/faq#advanced-format-disks ZFS on Linux FAQ] for more details.<br />
<br />
{{Tip|Use {{man|8|blockdev}} (part of {{Pkg|util-linux}}) to print the sector size reported by the device's ioctls: {{ic|# blockdev --getpbsz /dev/sd''XY''}}.}}<br />
<br />
Create pool with ashift=12 and single raidz vdev:<br />
<br />
# zpool create -f -o ashift=12 -m /mnt/data bigdata \<br />
raidz \<br />
ata-ST3000DM001-9YN166_S1F0KDGY \<br />
ata-ST3000DM001-9YN166_S1F0JKRR \<br />
ata-ST3000DM001-9YN166_S1F0KBP8 \<br />
ata-ST3000DM001-9YN166_S1F0JTM1<br />
<br />
==== GRUB-compatible pool creation ====<br />
<br />
{{note|This section frequently goes out of date with updates to GRUB and ZFS. Consult the manual pages for the most up-to-date information.}}<br />
<br />
By default, ''zpool create'' enables all features on a pool. If {{ic|/boot}} resides on ZFS when using [[GRUB]] you must only enable features supported by GRUB otherwise GRUB will not be able to read the pool. GRUB 2.02 supports the read-write features {{ic|lz4_compress}}, {{ic|hole_birth}}, {{ic|embedded_data}}, {{ic|extensible_dataset}}, and {{ic|large_blocks}}; this is not suitable for all the features of ZFSonLinux 0.7.1, and must have unsupported features disabled.<br />
<br />
You can create a pool with the incompatible features disabled:<br />
<br />
# zpool create -o feature@multi_vdev_crash_dump=disabled \<br />
-o feature@large_dnode=disabled \<br />
-o feature@sha512=disabled \<br />
-o feature@skein=disabled \<br />
-o feature@edonr=disabled \<br />
$POOL_NAME $VDEVS<br />
<br />
When running the git version of ZFS on Linux, make sure to also add {{ic|1=-o feature@encryption=disabled}}.<br />
<br />
=== Verifying pool status ===<br />
<br />
If the command is successful, there will be no output. Using the [[mount]] command will show that the pool is mounted. Using {{ic|zpool status}} will show that the pool has been created:<br />
<br />
{{hc|# zpool status -v|<br />
pool: bigdata<br />
state: ONLINE<br />
scan: none requested<br />
config:<br />
<br />
NAME STATE READ WRITE CKSUM<br />
bigdata ONLINE 0 0 0<br />
-0 ONLINE 0 0 0<br />
ata-ST3000DM001-9YN166_S1F0KDGY-part1 ONLINE 0 0 0<br />
ata-ST3000DM001-9YN166_S1F0JKRR-part1 ONLINE 0 0 0<br />
ata-ST3000DM001-9YN166_S1F0KBP8-part1 ONLINE 0 0 0<br />
ata-ST3000DM001-9YN166_S1F0JTM1-part1 ONLINE 0 0 0<br />
<br />
errors: No known data errors<br />
}}<br />
<br />
At this point it would be good to reboot the machine to ensure that the ZFS pool is mounted at boot. It is best to deal with all errors before transferring data.<br />
<br />
=== Importing a pool created by id ===<br />
<br />
Eventually a pool may fail to auto mount and you need to import to bring your pool back. Take care to avoid the most obvious solution.<br />
<br />
{{Warning|Do not run {{ic|zpool import ''pool''}}! This will import your pools using {{ic|/dev/sd?}} which will lead to problems the next time you rearrange your drives. This may be as simple as rebooting with a USB drive left in the machine, which harkens back to a time when PCs would not boot when a floppy disk was left in a machine.}}<br />
<br />
Adapt one of the following commands to import your pool so that pool imports retain the persistence they were created with:<br />
<br />
# zpool import -d /dev/disk/by-id bigdata<br />
# zpool import -d /dev/disk/by-partlabel bigdata<br />
# zpool import -d /dev/disk/by-partuuid bigdata<br />
<br />
{{Note|Use the {{ic|-l}} flag when importing a pool that contains [[#Native encryption|encrypted datasets keys]]:<br />
# zpool import -l -d /dev/disk/by-id bigdata<br />
}}<br />
<br />
Finally check the state of the pool:<br />
<br />
# zpool status -v bigdata<br />
<br />
== Tuning ==<br />
<br />
=== General ===<br />
<br />
ZFS pools can be further adjusted using parameters, most commonly {{ic|atime}} and {{ic|compression}}.<br />
<br />
To retrieve the current pool parameter status:<br />
<br />
# zfs get all <pool><br />
<br />
To disable access time (atime), which is enabled by default:<br />
<br />
# zfs set atime=off <pool><br />
<br />
As an alternative to turning off atime completely, {{ic|relatime}} is available. This brings the default ext4/XFS atime semantics to ZFS, where access time is only updated if the modified time or changed time changes, or if the existing access time has not been updated within the past 24 hours. It is a compromise between {{ic|1=atime=off}} and {{ic|1=atime=on}}. This property ''only'' takes effect if {{ic|atime}} is {{ic|on}}:<br />
<br />
# zfs set atime=on <pool><br />
# zfs set relatime=on <pool><br />
<br />
Compression is just that, transparent compression of data. ZFS supports a few different algorithms, presently lz4 is the default, ''gzip'' is also available for seldom-written yet highly-compressible data; consult the [http://open-zfs.org/wiki/Performance_tuning#Compression OpenZFS Wiki] for more details. <br />
<br />
To enable compression:<br />
<br />
# zfs set compression=on <pool><br />
<br />
{{Note| To reset a property of a pool and/or dataset to it's default state, use {{ic|zfs inherit}}:<br />
# zfs inherit -rS atime <pool><br />
# zfs inherit -rS atime <pool>/<dataset><br />
}}<br />
<br />
=== Scrubbing ===<br />
<br />
Whenever data is read and ZFS encounters an error, it is silently repaired when possible, rewritten back to disk and logged so you can obtain an overview of errors on your pools. There is no fsck or equivalent tool for ZFS. Instead, ZFS supports a feature known as scrubbing. This traverses through all the data in a pool and verifies that all blocks can be read.<br />
<br />
To scrub a pool:<br />
<br />
# zpool scrub <pool><br />
<br />
To cancel a running scrub:<br />
<br />
# zpool scrub -s <pool><br />
<br />
==== How often should I do this? ====<br />
<br />
From the Oracle blog post [https://blogs.oracle.com/wonders-of-zfs-storage/disk-scrub-why-and-when-v2 Disk Scrub - Why and When?]:<br />
<br />
:This question is challenging for Support to answer, because as always the true answer is "It Depends". So before I offer a general guideline, here are a few tips to help you create an answer more tailored to your use pattern.<br />
<br />
:* What is the expiration of your oldest backup? You should probably scrub your data at least as often as your oldest tapes expire so that you have a known-good restore point.<br />
:* How often are you experiencing disk failures? While the recruitment of a hot-spare disk invokes a "resilver" -- a targeted scrub of just the VDEV which lost a disk -- you should probably scrub at least as often as you experience disk failures on average in your specific environment.<br />
:* How often is the oldest piece of data on your disk read? You should scrub occasionally to prevent very old, very stale data from experiencing bit-rot and dying without you knowing it.<br />
<br />
:If any of your answers to the above are "I do not know", the general guideline is: you should probably be scrubbing your zpool at least once per month. It is a schedule that works well for most use cases, provides enough time for scrubs to complete before starting up again on all but the busiest & most heavily-loaded systems, and even on very large zpools (192+ disks) should complete fairly often between disk failures.<br />
<br />
In the [https://pthree.org/2012/12/11/zfs-administration-part-vi-scrub-and-resilver/ ZFS Administration Guide] by Aaron Toponce, he advises to scrub consumer disks once a week.<br />
<br />
==== Start with a service or timer ====<br />
Using a [[systemd]] timer/service it is possible to automatically scrub pools monthly:<br />
<br />
{{hc|/etc/systemd/system/zfs-scrub@.timer|2=<br />
[Unit]<br />
Description=Monthly zpool scrub on %i<br />
<br />
[Timer]<br />
OnCalendar=monthly<br />
AccuracySec=1h<br />
Persistent=true<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
}}<br />
<br />
{{hc|/etc/systemd/system/zfs-scrub@.service|2=<br />
[Unit]<br />
Description=zpool scrub on %i<br />
<br />
[Service]<br />
Nice=19<br />
IOSchedulingClass=idle<br />
KillSignal=SIGINT<br />
ExecStart=/usr/bin/zpool scrub %i<br />
}}<br />
<br />
[[Enable]]/[[start]] {{ic|zfs-scrub@''pool-to-scrub''.timer}} unit for monthly scrubbing the specified zpool.<br />
<br />
=== Destroy a storage pool ===<br />
<br />
ZFS makes it easy to destroy a mounted storage pool, removing all metadata about the ZFS device.<br />
<br />
{{Warning|This command destroys '''any data''' containing in the pool and/or dataset.}}<br />
<br />
To destroy the pool:<br />
# zpool destroy <pool><br />
<br />
To destroy a dataset:<br />
# zpool destroy <pool>/<dataset><br />
<br />
And now when checking the status:<br />
<br />
{{hc|# zpool status|<br />
no pools available<br />
}}<br />
<br />
=== Exporting a storage pool ===<br />
<br />
If a storage pool is to be used on another system, it will first need to be exported. It is also necessary to export a pool if it has been imported from the archiso as the hostid is different in the archiso as it is in the booted system. The zpool command will refuse to import any storage pools that have not been exported. It is possible to force the import with the {{ic|-f}} argument, but this is considered bad form.<br />
<br />
Any attempts made to import an un-exported storage pool will result in an error stating the storage pool is in use by another system. This error can be produced at boot time abruptly abandoning the system in the busybox console and requiring an archiso to do an emergency repair by either exporting the pool, or adding the {{ic|zfs_force&#61;1}} to the kernel boot parameters (which is not ideal). See [[#On boot the zfs pool does not mount stating: "pool may be in use from other system"]]<br />
<br />
To export a pool:<br />
<br />
# zpool export <pool><br />
<br />
=== Renaming a zpool ===<br />
<br />
Renaming a zpool that is already created is accomplished in 2 steps:<br />
<br />
# zpool export oldname<br />
# zpool import oldname newname<br />
<br />
=== Setting a different mount point ===<br />
<br />
The mount point for a given zpool can be moved at will with one command:<br />
# zfs set mountpoint=/foo/bar poolname<br />
<br />
=== SSD Caching ===<br />
<br />
You can add SSD devices as a write intent log (external ZIL or SLOG) and also as a layer 2 adaptive replacement cache (L2ARC). The process to add them is very similar to adding a new VDEV.<br />
<br />
All of the below references to device-id are the IDs from {{ic|/dev/disk/by-id/*}}.<br />
<br />
==== SLOG ====<br />
<br />
To add a mirrored SLOG:<br />
<br />
# zpool add <pool> log mirror <device-id-1> <device-id-2><br />
<br />
Or to add a single device SLOG (unsafe):<br />
<br />
# zpool add <pool> log <device-id><br />
<br />
Because the SLOG device stores data that has not been written to the pool, it is important to use devices that can finish writes when power is lost. It is also important to use redundancy, since a device failure can cause data loss. In addition, the SLOG is only used for sync writes, so may not provide any performance improvement.<br />
<br />
==== L2ARC ====<br />
<br />
To add L2ARC:<br />
<br />
# zpool add <pool> cache <device-id><br />
<br />
Because every block cached in L2ARC uses a small amount of memory, it is generally only useful in workloads where the amount of hot data is *bigger* than the maximum amount of memory that can fit in the computer, but small enough to fit into L2ARC. It is also cleared at reboot and is only a read cache, so redundancy is unnecessary. Un-intuitively, L2ARC can actually harm performance since it takes memory away from ARC.<br />
<br />
=== ZVOLs ===<br />
<br />
ZFS volumes (ZVOLs) can suffer from the same block size-related issues as RDBMSes, but it is worth noting that the default recordsize for ZVOLs is 8 KiB already. If possible, it is best to align any partitions contained in a ZVOL to your recordsize (current versions of fdisk and gdisk by default automatically align at 1MiB segments, which works), and file system block sizes to the same size. Other than this, you might tweak the '''recordsize''' to accommodate the data inside the ZVOL as necessary (though 8 KiB tends to be a good value for most file systems, even when using 4 KiB blocks on that level).<br />
<br />
==== RAIDZ and Advanced Format physical disks ====<br />
<br />
Each block of a ZVOL gets its own parity disks, and if you have physical media with logical block sizes of 4096B, 8192B, or so on, the parity needs to be stored in whole physical blocks, and this can drastically increase the space requirements of a ZVOL, requiring 2× or more physical storage capacity than the ZVOL's logical capacity. Setting the '''recordsize''' to 16k or 32k can help reduce this footprint drastically.<br />
<br />
See [https://github.com/zfsonlinux/zfs/issues/1807 ZFS on Linux issue #1807 for details]<br />
<br />
=== I/O Scheduler ===<br />
<br />
When the pool is imported, for whole disk vdevs, the block device I/O scheduler is set to {{ic|zfs_vdev_scheduler}} [https://github.com/zfsonlinux/zfs/wiki/ZFS-on-Linux-Module-Parameters#zfs_vdev_scheduler]. The most common schedulers are: ''noop'', ''cfq'', ''bfq'', and ''deadline''.<br />
<br />
In some cases, the scheduler is not changeable using this method. Known schedulers that cannot be changed are: ''scsi_mq'' and ''none''. In these cases, the scheduler is unchanged and an error message can be reported to logs. [[Improving_performance#Changing_I/O_scheduler|Manually setting]] one of the common schedulers used by {{ic|zfs_vdev_scheduler}} can result in more consistent performance.<br />
<br />
== Creating datasets ==<br />
<br />
Users can optionally create a dataset under the zpool as opposed to manually creating directories under the zpool. Datasets allow for an increased level of control (quotas for example) in addition to snapshots. To be able to create and mount a dataset, a directory of the same name must not pre-exist in the zpool. To create a dataset, use:<br />
<br />
# zfs create <nameofzpool>/<nameofdataset><br />
<br />
It is then possible to apply ZFS specific attributes to the dataset. For example, one could assign a quota limit to a specific directory within a dataset:<br />
<br />
# zfs set quota=20G <nameofzpool>/<nameofdataset>/<directory><br />
<br />
To see all the commands available in ZFS, see {{man|8|zfs|url=}} or {{man|8|zpool|url=}}.<br />
<br />
=== Native encryption ===<br />
<br />
ZFS offers the following supported encryption options: {{ic|aes-128-ccm}}, {{ic|aes-192-ccm}}, {{ic|aes-256-ccm}}, {{ic|aes-128-gcm}}, {{ic|aes-192-gcm}} and {{ic|aes-256-gcm}}. When encryption is set to {{ic|on}}, {{ic|aes-256-ccm}} will be used.<br />
<br />
The following keyformats are supported: {{ic|passphrase}}, {{ic|raw}}, {{ic|hex}}.<br />
<br />
One can also specify/increase the default iterations of PBKDF2 when using {{ic|passphrase}} with {{ic|-o pbkdf2iters <n>}}, although it may increase the decryption time.<br />
<br />
{{Note|<br />
* Native ZFS encryption has been made available in the stable 0.8.0 release or newer. Previously it was only available in development versions provided by packages like {{AUR|zfs-linux-git}}, {{AUR|zfs-dkms-git}} or other development builds. Users who were only using the development versions for the native encryption, may now switch to the stable releases if they wish.<br />
* To import a pool with keys, one needs to specify the {{ic|-l}} flag, without this flag encrypted datasets will be left unavailable until the keys are loaded. See [[#Importing a pool created by id]].}}<br />
<br />
To create a dataset including native encryption with a passphrase, use:<br />
<br />
# zfs create -o encryption=on -o keyformat=passphrase <nameofzpool>/<nameofdataset><br />
<br />
To use a key instead of using a passphrase:<br />
<br />
# dd if=/dev/random of=/path/to/key bs=1 count=32<br />
# zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///path/to/key <nameofzpool>/<nameofdataset><br />
<br />
To verify the key location:<br />
# zfs get keylocation <nameofzpool>/<nameofdataset><br />
<br />
To change the key location:<br />
<br />
# zfs set keylocation=file:///path/to/key <nameofzpool>/<nameofdataset><br />
<br />
You can also manually load the keys by using one of the following commands:<br />
<br />
# zfs load-key <nameofzpool>/<nameofdataset> # load key for a specific dataset<br />
# zfs load-key -a # load all keys<br />
# zfs load-key -r zpool/dataset # load all keys in a dataset<br />
<br />
To mount the created encrypted dataset:<br />
<br />
# zfs mount <nameofzpool>/<nameofdataset><br />
<br />
==== Unlock at boot time ====<br />
It is possible to automatically unlock a pool dataset on boot time by using a [[systemd]] unit. For example create the following service to unlock any dataset of the pool named ''tank'': <br />
<br />
{{hc|/etc/systemd/system/zfskey-tank@.service|2=<br />
[Unit]<br />
Description=Load pool encryption keys<br />
#Before=systemd-user-sessions.service<br />
Before=zfs-mount.service<br />
After=zfs-import.target<br />
<br />
[Service]<br />
Type=oneshot<br />
RemainAfterExit=yes<br />
ExecStart=/usr/bin/bash -c '/usr/bin/zfs load-key %j/%i<br />
<br />
[Install]<br />
WantedBy=zfs-mount.service<br />
}}<br />
<br />
[[Enable]]/[[start]] the service for each encrypted dataset, e.g. {{ic|# systemctl enable zfskey-tank@dataset}}.<br />
<br />
{{Note|The {{ic|1=Before=systemd-user-sessions.service}} ensures that systemd-ask-password is invoked before the local IO devices are handed over to the [[desktop environment]].}}<br />
<br />
An alternative is to load all possible keys:<br />
<br />
{{hc|etc/systemd/system/zfs-load-key.service|2=<br />
[Unit]<br />
Description=Load encryption keys<br />
Before=zfs-mount.service<br />
After=zfs-import.target<br />
<br />
[Service]<br />
Type=oneshot<br />
RemainAfterExit=yes<br />
ExecStart=/usr/bin/bash -c '/usr/bin/zfs load-key -a'<br />
<br />
[Install]<br />
WantedBy=zfs-mount.service<br />
}}<br />
<br />
[[Enable]]/[[start]] {{ic|zfs-load-key.service}}.<br />
<br />
=== Swap volume ===<br />
<br />
{{Warning|On systems with extremely high memory pressure, using a zvol for swap can result in lockup, regardless of how much swap is still available. This issue is currently being investigated in https://github.com/zfsonlinux/zfs/issues/7734}}<br />
<br />
ZFS does not allow to use swapfiles, but users can use a ZFS volume (ZVOL) as swap. It is important to set the ZVOL block size to match the system page size, which can be obtained by the {{ic|getconf PAGESIZE}} command (default on x86_64 is 4KiB). Another option useful for keeping the system running well in low-memory situations is not caching the ZVOL data.<br />
<br />
Create a 8 GiB zfs volume:<br />
<br />
# zfs create -V 8G -b $(getconf PAGESIZE) \<br />
-o logbias=throughput -o sync=always\<br />
-o primarycache=metadata \<br />
-o com.sun:auto-snapshot=false <pool>/swap<br />
<br />
Prepare it as swap partition:<br />
<br />
# mkswap -f /dev/zvol/<pool>/swap<br />
# swapon /dev/zvol/<pool>/swap<br />
<br />
To make it permanent, edit {{ic|/etc/fstab}}. ZVOLs support discard, which can potentially help ZFS's block allocator and reduce fragmentation for all other datasets when/if swap is not full.<br />
<br />
Add a line to {{ic|/etc/fstab}}:<br />
<br />
/dev/zvol/<pool>/swap none swap discard 0 0<br />
<br />
=== Access Control Lists ===<br />
<br />
To use [[ACL]] on a ZFS pool:<br />
<br />
# zfs set acltype=posixacl <nameofzpool>/<nameofdataset><br />
# zfs set xattr=sa <nameofzpool>/<nameofdataset><br />
<br />
Setting {{ic|xattr}} is recommended for performance reasons [https://github.com/zfsonlinux/zfs/issues/170#issuecomment-27348094].<br />
<br />
It may be preferable to enable ACL on the zpool instead. Setting {{ic|1=aclinherit=passthrough}} may be wanted as the default mode is {{ic|restricted}} [https://docs.oracle.com/cd/E19120-01/open.solaris/817-2271/gbaaz/index.html]:<br />
<br />
# zfs set aclinherit=passthrough <nameofzpool><br />
# zfs set acltype=posixacl <nameofzpool><br />
# zfs set xattr=sa <nameofzpool><br />
<br />
=== Databases ===<br />
<br />
ZFS, unlike most other file systems, has a variable record size, or what is commonly referred to as a block size. By default, the recordsize on ZFS is 128KiB, which means it will dynamically allocate blocks of any size from 512B to 128KiB depending on the size of file being written. This can often help fragmentation and file access, at the cost that ZFS would have to allocate new 128KiB blocks each time only a few bytes are written to.<br />
<br />
Most RDBMSes work in 8KiB-sized blocks by default. Although the block size is tunable for [[MySQL|MySQL/MariaDB]], [[PostgreSQL]], and [[Oracle]], all three of them use an 8KiB block size ''by default''. For both performance concerns and keeping snapshot differences to a minimum (for backup purposes, this is helpful), it is usually desirable to tune ZFS instead to accommodate the databases, using a command such as:<br />
<br />
# zfs set recordsize=8K <pool>/postgres<br />
<br />
These RDBMSes also tend to implement their own caching algorithm, often similar to ZFS's own ARC. In the interest of saving memory, it is best to simply disable ZFS's caching of the database's file data and let the database do its own job:<br />
<br />
# zfs set primarycache=metadata <pool>/postgres<br />
<br />
If your pool has no configured log devices, ZFS reserves space on the pool's data disks for its intent log (the ZIL). ZFS uses this for crash recovery, but databases are often syncing their data files to the file system on their own transaction commits anyway. The end result of this is that ZFS will be committing data '''twice''' to the data disks, and it can severely impact performance. You can tell ZFS to prefer to not use the ZIL, and in which case, data is only committed to the file system once. Setting this for non-database file systems, or for pools with configured log devices, can actually ''negatively'' impact the performance, so beware:<br />
<br />
# zfs set logbias=throughput <pool>/postgres<br />
<br />
These can also be done at file system creation time, for example:<br />
<br />
# zfs create -o recordsize=8K \<br />
-o primarycache=metadata \<br />
-o mountpoint=/var/lib/postgres \<br />
-o logbias=throughput \<br />
<pool>/postgres<br />
<br />
Please note: these kinds of tuning parameters are ideal for specialized applications like RDBMSes. You can easily ''hurt'' ZFS's performance by setting these on a general-purpose file system such as your /home directory.<br />
<br />
=== /tmp ===<br />
<br />
If you would like to use ZFS to store your /tmp directory, which may be useful for storing arbitrarily-large sets of files or simply keeping your RAM free of idle data, you can generally improve performance of certain applications writing to /tmp by disabling file system sync. This causes ZFS to ignore an application's sync requests (eg, with {{ic|fsync}} or {{ic|O_SYNC}}) and return immediately. While this has severe application-side data consistency consequences (never disable sync for a database!), files in /tmp are less likely to be important and affected. Please note this does ''not'' affect the integrity of ZFS itself, only the possibility that data an application expects on-disk may not have actually been written out following a crash.<br />
<br />
# zfs set sync=disabled <pool>/tmp<br />
<br />
Additionally, for security purposes, you may want to disable '''setuid''' and '''devices''' on the /tmp file system, which prevents some kinds of privilege-escalation attacks or the use of device nodes:<br />
<br />
# zfs set setuid=off <pool>/tmp<br />
# zfs set devices=off <pool>/tmp<br />
<br />
Combining all of these for a create command would be as follows:<br />
<br />
# zfs create -o setuid=off -o devices=off -o sync=disabled -o mountpoint=/tmp <pool>/tmp<br />
<br />
Please note, also, that if you want /tmp on ZFS, you will need to mask (disable) [[systemd]]'s automatic tmpfs-backed /tmp, else ZFS will be unable to mount your dataset at boot-time or import-time:<br />
<br />
# systemctl mask tmp.mount<br />
<br />
== Troubleshooting ==<br />
<br />
=== Creating a zpool fails ===<br />
<br />
If the following error occurs then it can be fixed.<br />
<br />
# the kernel failed to rescan the partition table: 16<br />
# cannot label 'sdc': try using parted(8) and then provide a specific slice: -1<br />
<br />
One reason this can occur is because [https://github.com/zfsonlinux/zfs/issues/2582 ZFS expects pool creation to take less than 1 second]. This is a reasonable assumption under ordinary conditions, but in many situations it may take longer. Each drive will need to be cleared again before another attempt can be made.<br />
<br />
# parted /dev/sda rm 1<br />
# parted /dev/sda rm 1<br />
# dd if=/dev/zero of=/dev/sdb bs=512 count=1<br />
# zpool labelclear /dev/sda<br />
<br />
A brute force creation can be attempted over and over again, and with some luck the ZPool creation will take less than 1 second.<br />
Once cause for creation slowdown can be slow burst read writes on a drive. By reading from the disk in parallell to ZPool creation, it may be possible to increase burst speeds.<br />
<br />
# dd if=/dev/sda of=/dev/null<br />
<br />
This can be done with multiple drives by saving the above command for each drive to a file on separate lines and running <br />
<br />
# cat $FILE | parallel<br />
<br />
Then run ZPool creation at the same time.<br />
<br />
=== ZFS is using too much RAM ===<br />
<br />
By default, ZFS caches file operations ([[wikipedia:Adaptive replacement cache|ARC]]) using up to two-thirds of available system memory on the host. To adjust the ARC size, add the following to the [[Kernel parameters]] list:<br />
<br />
zfs.zfs_arc_max=536870912 # (for 512MB)<br />
<br />
For a more detailed description, as well as other configuration options, see [http://wiki.gentoo.org/wiki/ZFS#ARC gentoo-wiki:zfs#arc].<br />
<br />
=== Does not contain an EFI label ===<br />
<br />
The following error will occur when attempting to create a zfs filesystem,<br />
<br />
/dev/disk/by-id/<id> does not contain an EFI label but it may contain partition<br />
<br />
The way to overcome this is to use {{ic|-f}} with the zfs create command.<br />
<br />
=== No hostid found ===<br />
<br />
An error that occurs at boot with the following lines appearing before initscript output:<br />
<br />
ZFS: No hostid found on kernel command line or /etc/hostid.<br />
<br />
This warning occurs because the ZFS module does not have access to the spl hosted. There are two solutions, for this. Either place the spl hostid in the [[kernel parameters]] in the boot loader. For example, adding {{ic|1=spl.spl_hostid=0x00bab10c}}.<br />
<br />
The other solution is to make sure that there is a hostid in {{ic|/etc/hostid}}, and then [[regenerate the initramfs]] image. Which will copy the hostid into the initramfs image.<br />
<br />
=== Pool cannot be found while booting from SAS/SCSI devices ===<br />
<br />
In case you are booting a SAS/SCSI based, you might occassionally get boot problems where the pool you are trying to boot from cannot be found. A likely reason for this is that your devices are initialized too late into the process. That means that zfs cannot find any devices at the time when it tries to assemble your pool.<br />
<br />
In this case you should force the scsi driver to wait for devices to come online before continuing. You can do this by putting this into {{ic|/etc/modprobe.d/zfs.conf}}:<br />
<br />
{{hc|1=/etc/modprobe.d/zfs.conf|2=<br />
options scsi_mod scan=sync<br />
}}<br />
<br />
Afterwards, [[regenerate the initramfs]].<br />
<br />
This works because the zfs hook will copy the file at {{ic|/etc/modprobe.d/zfs.conf}} into the initcpio which will then be used at build time.<br />
<br />
=== On boot the zfs pool does not mount stating: "pool may be in use from other system" ===<br />
<br />
==== Unexported pool ====<br />
<br />
If the new installation does not boot because the zpool cannot be imported, chroot into the installation and properly export the zpool. See [[#Emergency chroot repair with archzfs]].<br />
<br />
Once inside the chroot environment, load the ZFS module and force import the zpool,<br />
<br />
# zpool import -a -f<br />
<br />
now export the pool:<br />
<br />
# zpool export <pool><br />
<br />
To see the available pools, use,<br />
<br />
# zpool status<br />
<br />
It is necessary to export a pool because of the way ZFS uses the hostid to track the system the zpool was created on. The hostid is generated partly based on the network setup. During the installation in the archiso the network configuration could be different generating a different hostid than the one contained in the new installation. Once the zfs filesystem is exported and then re-imported in the new installation, the hostid is reset. See [http://osdir.com/ml/zfs-discuss/2011-06/msg00227.html Re: Howto zpool import/export automatically? - msg#00227].<br />
<br />
If ZFS complains about "pool may be in use" after every reboot, properly export pool as described above, and then [[regenerate the initramfs]] in normally booted system.<br />
<br />
==== Incorrect hostid ====<br />
<br />
Double check that the pool is properly exported. Exporting the zpool clears the hostid marking the ownership. So during the first boot the zpool should mount correctly. If it does not there is some other problem.<br />
<br />
Reboot again, if the zfs pool refuses to mount it means the hostid is not yet correctly set in the early boot phase and it confuses zfs. Manually tell zfs the correct number, once the hostid is coherent across the reboots the zpool will mount correctly.<br />
<br />
Boot using zfs_force and write down the hostid. This one is just an example.<br />
<br />
{{hc|$ hostid|<br />
0a0af0f8<br />
}}<br />
<br />
This number have to be added to the [[kernel parameters]] as {{ic|1=spl.spl_hostid=0x0a0af0f8}}. Another solution is writing the hostid inside the initram image, see the [[Installing Arch Linux on ZFS#After the first boot|installation guide]] explanation about this.<br />
<br />
Users can always ignore the check adding {{ic|1=zfs_force=1}} in the [[kernel parameters]], but it is not advisable as a permanent solution.<br />
<br />
=== Devices have different sector alignment ===<br />
<br />
Once a drive has become faulted it should be replaced A.S.A.P. with an identical drive.<br />
<br />
# zpool replace bigdata ata-ST3000DM001-9YN166_S1F0KDGY ata-ST3000DM001-1CH166_W1F478BD -f<br />
<br />
but in this instance, the following error is produced:<br />
<br />
cannot replace ata-ST3000DM001-9YN166_S1F0KDGY with ata-ST3000DM001-1CH166_W1F478BD: devices have different sector alignment<br />
<br />
ZFS uses the ashift option to adjust for physical block size. When replacing the faulted disk, ZFS is attempting to use {{ic|<nowiki>ashift=12</nowiki>}}, but the faulted disk is using a different ashift (probably {{ic|<nowiki>ashift=9</nowiki>}}) and this causes the resulting error. <br />
<br />
For Advanced Format Disks with 4KB blocksize, an ashift of 12 is recommended for best performance. See [https://github.com/zfsonlinux/zfs/wiki/faq#performance-considerations 1.10 What’s going on with performance?] and [http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks ZFS and Advanced Format disks].<br />
<br />
Use zdb to find the ashift of the zpool: {{ic|zdb | grep ashift}}, then use the {{ic|-o}} argument to set the ashift of the replacement drive:<br />
<br />
# zpool replace bigdata ata-ST3000DM001-9YN166_S1F0KDGY ata-ST3000DM001-1CH166_W1F478BD -o ashift=9 -f<br />
<br />
Check the zpool status for confirmation:<br />
<br />
{{hc|# zpool status -v|<br />
pool: bigdata<br />
state: DEGRADED<br />
status: One or more devices is currently being resilvered. The pool will<br />
continue to function, possibly in a degraded state.<br />
action: Wait for the resilver to complete.<br />
scan: resilver in progress since Mon Jun 16 11:16:28 2014<br />
10.3G scanned out of 5.90T at 81.7M/s, 20h59m to go<br />
2.57G resilvered, 0.17% done<br />
config:<br />
<br />
NAME STATE READ WRITE CKSUM<br />
bigdata DEGRADED 0 0 0<br />
raidz1-0 DEGRADED 0 0 0<br />
replacing-0 OFFLINE 0 0 0<br />
ata-ST3000DM001-9YN166_S1F0KDGY OFFLINE 0 0 0<br />
ata-ST3000DM001-1CH166_W1F478BD ONLINE 0 0 0 (resilvering)<br />
ata-ST3000DM001-9YN166_S1F0JKRR ONLINE 0 0 0<br />
ata-ST3000DM001-9YN166_S1F0KBP8 ONLINE 0 0 0<br />
ata-ST3000DM001-9YN166_S1F0JTM1 ONLINE 0 0 0<br />
<br />
errors: No known data errors<br />
}}<br />
<br />
=== Pool resilvering stuck/restarting/slow? ===<br />
<br />
According to the ZFSonLinux github it is a known issue since 2012 with ZFS-ZED which causes the resilvering process to constantly restart, sometimes get stuck and be generally slow for some hardware. The simplest mitigation is to stop zfs-zed.service until the resilver completes<br />
<br />
=== Fix slow boot caused by failed import of unavailable pools in the initramfs zpool.cache ===<br />
<br />
Your boot time can be significantly impacted if you update your intitramfs (eg when doing a kernel update) when you have additional but non-permanently attached pools imported because these pools will get added to your initramfs zpool.cache and ZFS will attempt to import these extra pools on every boot, regardless of whether you have exported it and removed it from your regular zpool.cache.<br />
<br />
If you notice ZFS trying to import unavailable pools at boot, first run:<br />
<br />
$ zdb -C<br />
<br />
To check you zpool.cache for pools you do not want imported at boot. If this command is showing (a) additional, currently unavailable pool(s), run:<br />
<br />
# zpool set cachefile=/etc/zfs/zpool.cache zroot<br />
<br />
To clear the zpool.cache of any pools other than the pool named zroot. Sometimes there is no need to refresh your zpool.cache, but instead all you need to do is [[regenerate the initramfs]].<br />
<br />
== Tips and tricks ==<br />
<br />
=== Embed the archzfs packages into an archiso ===<br />
<br />
Follow the [[Archiso]] steps for creating a fully functional Arch Linux live CD/DVD/USB image.<br />
<br />
Enable the [[Unofficial user repositories#archzfs|archzfs]] repository:<br />
<br />
{{hc|~/archlive/pacman.conf|<nowiki><br />
...<br />
[archzfs]<br />
Server = http://archzfs.com/$repo/x86_64<br />
</nowiki>}}<br />
<br />
Add the {{ic|archzfs-linux}} group to the list of packages to be installed (the {{ic|archzfs}} repository provides packages for the x86_64 architecture only).<br />
<br />
{{hc|~/archlive/packages.x86_64|<br />
...<br />
archzfs-linux<br />
}}<br />
<br />
Complete [[Archiso#Build_the_ISO|Build the ISO]] to finally build the iso.<br />
<br />
{{Note|If you later have problems running modprobe zfs, you should include the linux-headers in the packages.x86_64. }}<br />
<br />
=== Automatic snapshots ===<br />
<br />
==== ZFS Automatic Snapshot Service for Linux ====<br />
<br />
The {{AUR|zfs-auto-snapshot-git}} package from [[AUR]] provides a shell script to automate the management of snapshots, with each named by date and label (hourly, daily, etc), giving quick and convenient snapshotting of all ZFS datasets. The package also installs cron tasks for quarter-hourly, hourly, daily, weekly, and monthly snapshots. Optionally adjust the {{ic|--keep parameter}} from the defaults depending on how far back the snapshots are to go (the monthly script by default keeps data for up to a year).<br />
<br />
To prevent a dataset from being snapshotted at all, set {{ic|1=com.sun:auto-snapshot=false}} on it. Likewise, set more fine-grained control as well by label, if, for example, no monthlies are to be kept on a snapshot, for example, set {{ic|1=com.sun:auto-snapshot:monthly=false}}.<br />
<br />
{{Note|zfs-auto-snapshot-git will not create snapshots during scrubbing ([[#Scrub|scrub]]{{Broken section link}}). It is possible to override this by editing provided systemd unit ([[Systemd#Editing provided units]]) and removing `--skip-scrub` from `ExecStart` line. Consequences not known, someone please edit.}}<br />
<br />
==== ZFS Snapshot Manager ====<br />
<br />
The {{AUR|zfs-snap-manager}} package from [[AUR]] provides a python service that takes daily snapshots from a configurable set of ZFS datasets and cleans them out in a [[wikipedia:Backup rotation scheme#Grandfather-father-son|"Grandfather-father-son"]] scheme. It can be configured to e.g. keep 7 daily, 5 weekly, 3 monthly and 2 yearly snapshots. <br />
<br />
The package also supports configurable replication to other machines running ZFS by means of {{ic|zfs send}} and {{ic|zfs receive}}. If the destination machine runs this package as well, it could be configured to keep these replicated snapshots for a longer time. This allows a setup where a source machine has only a few daily snapshots locally stored, while on a remote storage server a much longer retention is available.<br />
<br />
=== Creating a share ===<br />
<br />
ZFS has support for creating shares by SMB or [[NFS]].<br />
<br />
==== NFS ====<br />
<br />
Make sure [[NFS]] has been installed/configured, note there is no need to edit the {{ic|/etc/exports}} file. For sharing over NFS the services {{ic|nfs-server.service}} and {{ic|zfs-share.service}} should be [[start|started]].<br />
<br />
To make a pool available on the network:<br />
<br />
# zfs set sharenfs=on <nameofzpool><br />
<br />
To make a dataset available on the network:<br />
<br />
# zfs set sharenfs=on <nameofzpool>/<nameofdataset><br />
<br />
To enable read/write access for a specific ip-range(s):<br />
<br />
# zfs set sharenfs="rw=@192.168.1.100/24,rw=@10.0.0.0/24" <nameofzpool>/<nameofdataset><br />
<br />
To check if the dataset is exported successful:<br />
<br />
{{hc|# showmount -e `hostname`|<br />
Export list for hostname:<br />
/path/of/dataset 192.168.1.100/24<br />
}}<br />
<br />
To view the current loaded exports state in more detail, use:<br />
<br />
{{hc|# exportfs -v|2=<br />
/path/of/dataset<br />
192.168.1.100/24(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,secure,no_root_squash,no_all_squash)<br />
}}<br />
<br />
==== SMB ====<br />
<br />
When sharing smb shares configuring usershares in your smb.conf will allow ZFS to setup and create the shares.<br />
<br />
{{bc|1=<br />
# [global]<br />
# usershare path = /var/lib/samba/usershares<br />
# usershare max shares = 100<br />
# usershare allow guests = yes<br />
# usershare owner only = no<br />
}}<br />
<br />
Create and set permissions on the user directory as root<br />
<br />
# mkdir /var/lib/samba/usershares<br />
# chmod +t /var/lib/samba/usershares<br />
<br />
=== Encryption in ZFS using dm-crypt ===<br />
<br />
The stable release version of ZFS on Linux does not support encryption directly, but zpools can be created in dm-crypt block devices. Since the zpool is created on the plain-text abstraction, it is possible to have the data encrypted while having all the advantages of ZFS like deduplication, compression, and data robustness.<br />
<br />
dm-crypt, possibly via LUKS, creates devices in {{ic|/dev/mapper}} and their name is fixed. So you just need to change {{ic|zpool create}} commands to point to that names. The idea is configuring the system to create the {{ic|/dev/mapper}} block devices and import the zpools from there. Since zpools can be created in multiple devices (raid, mirroring, striping, ...), it is important all the devices are encrypted otherwise the protection might be partially lost.<br />
<br />
For example, an encrypted zpool can be created using plain dm-crypt (without LUKS) with:<br />
<br />
# cryptsetup --hash=sha512 --cipher=twofish-xts-plain64 --offset=0 \<br />
--key-file=/dev/sdZ --key-size=512 open --type=plain /dev/sdX enc<br />
# zpool create zroot /dev/mapper/enc<br />
<br />
In the case of a root filesystem pool, the {{ic|mkinitcpio.conf}} HOOKS line will enable the keyboard for the password, create the devices, and load the pools. It will contain something like:<br />
<br />
HOOKS="... keyboard encrypt zfs ..."<br />
<br />
Since the {{ic|/dev/mapper/enc}} name is fixed no import errors will occur.<br />
<br />
Creating encrypted zpools works fine. But if you need encrypted directories, for example to protect your users' homes, ZFS loses some functionality.<br />
<br />
ZFS will see the encrypted data, not the plain-text abstraction, so compression and deduplication will not work. The reason is that encrypted data has always high entropy making compression ineffective and even from the same input you get different output (thanks to salting) making deduplication impossible. To reduce the unnecessary overhead it is possible to create a sub-filesystem for each encrypted directory and use [[eCryptfs]] on it.<br />
<br />
For example to have an encrypted home: (the two passwords, encryption and login, must be the same)<br />
<br />
# zfs create -o compression=off -o dedup=off -o mountpoint=/home/<username> <zpool>/<username><br />
# useradd -m <username><br />
# passwd <username><br />
# ecryptfs-migrate-home -u <username><br />
<log in user and complete the procedure with ecryptfs-unwrap-passphrase><br />
<br />
=== Emergency chroot repair with archzfs ===<br />
<br />
To get into the ZFS filesystem from live system for maintenance, there are two options:<br />
<br />
# Build custom archiso with ZFS as described in [[#Embed the archzfs packages into an archiso]].<br />
# Boot the latest official archiso and bring up the network. Then enable [[Unofficial_user_repositories#archzfs|archzfs]] repository inside the live system as usual, sync the pacman package database and install the ''archzfs-archiso-linux'' package.<br />
<br />
To start the recovery, load the ZFS kernel modules:<br />
<br />
# modprobe zfs<br />
<br />
Import the pool:<br />
<br />
# zpool import -a -R /mnt<br />
<br />
Mount the boot partitions (if any):<br />
<br />
# mount /dev/sda2 /mnt/boot<br />
# mount /dev/sda1 /mnt/boot/efi<br />
<br />
Chroot into the ZFS filesystem:<br />
<br />
# arch-chroot /mnt /bin/bash<br />
<br />
Check the kernel version:<br />
<br />
# pacman -Qi linux<br />
# uname -r<br />
<br />
uname will show the kernel version of the archiso. If they are different, run depmod (in the chroot) with the correct kernel version of the chroot installation:<br />
<br />
# depmod -a 3.6.9-1-ARCH (version gathered from pacman -Qi linux but using the matching kernel modules directory name under the chroot's /lib/modules)<br />
<br />
This will load the correct kernel modules for the kernel version installed in the chroot installation.<br />
<br />
[[Regenerate the initramfs]]. There should be no errors.<br />
<br />
=== Bind mount ===<br />
<br />
Here a bind mount from /mnt/zfspool to /srv/nfs4/music is created. The configuration ensures that the zfs pool is ready before the bind mount is created.<br />
<br />
==== fstab ====<br />
<br />
See [http://www.freedesktop.org/software/systemd/man/systemd.mount.html systemd.mount] for more information on how systemd converts fstab into mount unit files with [http://www.freedesktop.org/software/systemd/man/systemd-fstab-generator.html systemd-fstab-generator].<br />
<br />
{{hc|/etc/fstab|<nowiki><br />
/mnt/zfspool /srv/nfs4/music none bind,defaults,nofail,x-systemd.requires=zfs-mount.service 0 0<br />
</nowiki>}}<br />
<br />
=== Monitoring / Mailing on Events ===<br />
<br />
See [https://ramsdenj.com/2016/08/29/arch-linux-on-zfs-part-3-followup.html ZED: The ZFS Event Daemon] for more information.<br />
<br />
An email forwarder, such as [[S-nail]] (installed as part of {{Grp|base}}), is required to accomplish this. Test it to be sure it is working correctly.<br />
<br />
Uncomment the following in the configuration file:<br />
<br />
{{hc|/etc/zfs/zed.d/zed.rc|<nowiki><br />
ZED_EMAIL_ADDR="root"<br />
ZED_EMAIL_PROG="mailx"<br />
ZED_NOTIFY_VERBOSE=0<br />
ZED_EMAIL_OPTS="-s '@SUBJECT@' @ADDRESS@"<br />
</nowiki>}}<br />
<br />
Update 'root' in {{ic|1=ZED_EMAIL_ADDR="root"}} to the email address you want to receive notifications at.<br />
<br />
If you are keeping your mailrc in your home directory, you can tell mail to get it from there by setting {{ic|MAILRC}}:<br />
<br />
{{hc|/etc/zfs/zed.d/zed.rc|2=<br />
export MAILRC=/home/<user>/.mailrc<br />
}}<br />
<br />
This works because ZED sources this file, so {{ic|mailx}} sees this environment variable.<br />
<br />
If you want to receive an email no matter the state of your pool, you will want to set {{ic|1=ZED_NOTIFY_VERBOSE=1}}. You will need to do this temporary to test.<br />
<br />
[[Start]] and [[enable]] {{ic|zfs-zed.service}}.<br />
<br />
With {{ic|1=ZED_NOTIFY_VERBOSE=1}}, you can test by running a scrub as root: {{ic|1=zpool scrub <pool-name>}}.<br />
<br />
=== Wrap shell commands in pre & post snapshots ===<br />
<br />
Since it is so cheap to make a snapshot, we can use this as a measure of security for sensitive commands such as system and package upgrades. If we make a snapshot before, and one after, we can later diff these snapshots to find out what changed on the filesystem after the command executed. Furthermore we can also rollback in case the outcome was not desired.<br />
<br />
E.g.:<br />
<br />
# zfs snapshot -r zroot@pre<br />
# pacman -Syu<br />
# zfs snapshot -r zroot@post<br />
# zfs diff zroot@pre zroot@post <br />
# zfs rollback zroot@pre<br />
<br />
A utility that automates the creation of pre and post snapshots around a shell command is [https://gist.github.com/erikw/eeec35be33e847c211acd886ffb145d5 znp].<br />
<br />
E.g.:<br />
<br />
# znp pacman -Syu<br />
# znp find / -name "something*" -delete<br />
<br />
and you would get snapshots created before and after the supplied command, and also output of the commands logged to file for future reference so we know what command created the diff seen in a pair of pre/post snapshots.<br />
<br />
=== Remote unlocking of ZFS encrypted root ===<br />
<br />
As of [https://github.com/archzfs/archzfs/pull/261 PR #261], {{ic|archzfs}} supports SSH unlocking of natively-encrypted ZFS datasets. This section describes how to use this feature, and is largely based on [[dm-crypt/Specialties#Remote unlocking (hooks: netconf, dropbear, tinyssh, ppp)]].<br />
<br />
#Install {{Pkg|mkinitcpio-netconf}} to provide hooks for setting up early user space networking.<br />
#Choose an SSH server to use in early user space. The options are {{Pkg|mkinitcpio-tinyssh}} or {{Pkg|mkinitcpio-dropbear}}, and are mutually exclusive.<br />
##If using {{Pkg|mkinitcpio-tinyssh}}, it is also recommended to install {{Pkg|tinyssh-convert}} or {{AUR|tinyssh-convert-git}}. This tool converts an existing OpenSSH hostkey to the TinySSH key format, preserving the key fingerprint and avoiding connection warnings. The TinySSH and Dropbear mkinitcpio install scripts will automatically convert existing hostkeys when generating a new initcpio image.<br />
#Decide whether to use an existing OpenSSH key or generate a new one (recommended) for the host that will be connecting to and unlocking the encrypted ZFS machine. Copy the public key into {{ic|/etc/tinyssh/root_key}} or {{ic|/etc/dropbear/root_key}}. When generating the initcpio image, this file will be added to {{ic|authorized_keys}} for the root user and is only valid in the initrd environment.<br />
#Add the {{ic|1=ip=}} [[kernel parameter]] to your boot loader configuration. The {{ic|ip}} string is [https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt highly configurable]. A simple DHCP example is shown below.{{bc|1=ip=:::::eth0:dhcp}}<br />
#Edit {{ic|/etc/mkinitcpio.conf}} to include the {{ic|netconf}}, {{ic|dropbear}} or {{ic|tinyssh}}, and {{ic|zfsencryptssh}} hooks before the {{ic|zfs}} hook:{{bc|1=HOOKS=(... netconf <tinyssh>{{!}}<dropbear> zfsencryptssh zfs ...)}}<br />
#[[Regenerate the initramfs]].<br />
#Reboot and try it out!<br />
<br />
====Changing the SSH server port====<br />
<br />
By default, {{Pkg|mkinitcpio-tinyssh}} and {{Pkg|mkinitcpio-dropbear}} listen on port {{ic|22}}. You may wish to change this.<br />
<br />
For '''TinySSH''', copy {{ic|/usr/lib/initcpio/hooks/tinyssh}} to {{ic|/etc/initcpio/hooks/tinyssh}}, and find/modify the following line in the {{ic|run_hook()}} function:<br />
<br />
{{hc|/etc/initcpio/hooks/tinyssh|<br />
/usr/bin/tcpserver -HRDl0 0.0.0.0 <new_port> /usr/sbin/tinysshd -v /etc/tinyssh/sshkeydir &<br />
}}<br />
<br />
For '''Dropbear''', copy {{ic|/usr/lib/initcpio/hooks/dropbear}} to {{ic|/etc/initcpio/hooks/dropbear}}, and find/modify the following line in the {{ic|run_hook()}} function:<br />
<br />
{{hc|/etc/initcpio/hooks/tinyssh|<br />
/usr/sbin/dropbear -E -s -j -k -p <new_port><br />
}}<br />
<br />
[[Regenerate the initramfs]].<br />
<br />
==== Unlocking from a Windows machine using PuTTY/Plink ====<br />
<br />
First, we need to use {{ic|puttygen.exe}} to import and convert the OpenSSH key generated earlier into PuTTY's ''.ppk'' private key format. Let us call it {{ic|zfs_unlock.ppk}} for this example.<br />
<br />
The mkinitcpio-netconf process above does not setup a shell (nor do we need need one). However, because there is no shell, PuTTY will immediately close after a successful connection. This can be disabled in the PuTTY SSH configuration (''Connection -> SSH -> [X] Do not start a shell or command at all''), but it still does not allow us to see stdout or enter the encryption passphrase. Instead, we use {{ic|plink.exe}} with the following parameters:<br />
<br />
plink.exe -ssh -l root -i c:\path\to\zfs_unlock.ppk <hostname><br />
<br />
The plink command can be put into a batch script for ease of use.<br />
<br />
== See also ==<br />
<br />
* [https://pthree.org/2012/12/04/zfs-administration-part-i-vdevs/ Aaron Toponce's 17-part blog on ZFS]<br />
* [http://zfsonlinux.org/ ZFS on Linux]<br />
* [https://github.com/zfsonlinux/zfs/wiki/faq ZFS on Linux FAQ]<br />
* [https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/zfs.html FreeBSD Handbook -- The Z File System]<br />
* [https://docs.oracle.com/cd/E19253-01/819-5461/index.html Oracle Solaris ZFS Administration Guide]<br />
* [http://www.solarisinternals.com/wiki/index.php/ZFS_Troubleshooting_Guide Solaris Internals -- ZFS Troubleshooting Guide]{{Dead link|2017|05|30}}<br />
* [http://royal.pingdom.com/2013/06/04/zfs-backup/ How Pingdom uses ZFS to back up 5TB of MySQL data every day]<br />
* [https://www.linuxquestions.org/questions/linux-from-scratch-13/%5Bhow-to%5D-add-zfs-to-the-linux-kernel-4175514510/ Tutorial on adding the modules to a custom kernel]<br />
* [https://github.com/danboid/creating-ZFS-disks-under-Linux How to create cross platform ZFS disks under Linux]<br />
* [https://blog.heckel.xyz/2017/01/08/zfs-encryption-openzfs-zfs-on-linux/ How-To: Using ZFS Encryption at Rest in OpenZFS (ZFS on Linux, ZFS on FreeBSD, …)]</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=Talk:Go_package_guidelines&diff=572853Talk:Go package guidelines2019-05-09T19:41:09Z<p>Alaskanarcher: Add whitespace</p>
<hr />
<div>==GOPATH has been removed from /etc/profile.d/go.sh==<br />
/etc/profile.d/go.sh since go 1.2-1 doesn't contain GOPATH and breaks PKGBUILD templates.<br />
https://projects.archlinux.org/svntogit/community.git/commit/trunk?h=packages/go&id=fa3a43875223f38c577093e2b6dc14c604a78b08<br />
I really don't know if /usr/lib/go/src is the right place for sources or they should go to /usr/lib/go/site/src.<br />
In either case, $GOPATH is not reset after sourcing /etc/profile.d/go.sh, so src ends up in $srcdir.<br />
[[User:Vulpesvelox|Vulpesvelox]] ([[User talk:Vulpesvelox|talk]]) 10:44, 24 January 2014 (UTC)<br />
<br />
==Use of GOPATH==<br />
<br />
In the {{ic|check()}} function of the sample PKGBUILD using {{ic|go get}}, the GOPATH is set to {{ic|"$GOPATH:$srcdir"}}. This fails if GOPATH is empty.<br />
Beside, I don't see the point of setting this to a possibly existing $GOPATH when the code.<br />
I think PKGBUILDs should ''not'' rely on the the user's repository, that is, it should be sandboxed. If the users already have a repo, they probably don't need the PKGBUILD in the first place.<br />
All they need is run {{ic|go get -u blah/blah}} and same for testing.<br />
What do you think?<br />
-- [[User:Ambrevar|Ambrevar]] ([[User talk:Ambrevar|talk]]) 07:39, 1 May 2016 (UTC)<br />
<br />
== Generate version using pkgver() ==<br />
<br />
I tweaked one of the examples on this page to include a automatically generated version. Perhaps worth considering:<br />
<br />
<br />
# Maintainer: Harold Drost <baelish@bluecell.net><br />
<br />
pkgname=okta-kubectl-auth-git<br />
pkgver=0.1.1.r2.gd731ef2<br />
pkgrel=1<br />
pkgdesc="Okta auth plugin for kubectl"<br />
arch=("x86_64")<br />
makedepends=("go")<br />
url="https://github.com/jetstack/okta-kubectl-auth"<br />
license=("Apache")<br />
options=('!strip' '!emptydirs')<br />
_gourl="github.com/jetstack/okta-kubectl-auth"<br />
<br />
prepare() {<br />
GOPATH="$srcdir" go get -fix -v -x ${_gourl}<br />
}<br />
<br />
pkgver() {<br />
cd "$srcdir/src/$_gourl"<br />
git describe --long --tags | sed 's/\([^-]*-g\)/r\1/;s/-/./g'<br />
}<br />
<br />
check() {<br />
GOPATH="$GOPATH:$srcdir" go test -v -x ${_gourl}<br />
}<br />
<br />
package() {<br />
mkdir -p "$pkgdir/usr/bin"<br />
install -p -m755 "$srcdir/bin/"* "$pkgdir/usr/bin"<br />
<br />
# Package license (if available)<br />
for f in LICENSE COPYING LICENSE.* COPYING.*; do<br />
if [ -e "$srcdir/src/$_gourl/$f" ]; then<br />
install -Dm644 "$srcdir/src/$_gourl/$f" \<br />
"$pkgdir/usr/share/licenses/$pkgname/$f"<br />
fi<br />
done<br />
}<br />
<br />
# vim:set ts=2 sw=2 et:<br />
<br />
I used {{ic|prepare()}} for the steps which were in {{ic|build()}} as it's run before {{ic|pkgver()}}. I don't think it's unreasonable to call the "go get" step preparation so I don't think it's a concern. --[[User:Baelish|Baelish]] ([[User talk:Baelish|talk]]) 19:52, 27 July 2018 (UTC)<br />
<br />
== Go Modules and Caching ==<br />
<br />
I was recently building a package and noticed that it errored in CI but worked locally. It turned out I had a cached version of a dependency that had been removed upstream. This made me wonder if it wouldn't be good practice to recommend setting: GOCACHE="${srcdir}/cache" or similar before downloading packages so that doing a clean build also destroys the cache.<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:44, 22 January 2019 (UTC)<br />
<br />
I'm not sure if this is desirable for most users. When I'm not building in a chroot, I want makepkg to use my local cache to build with. This saves me time when building. <br />
<br />
The problem you experienced is a side-effect of the golang community relying on third party hosting for dependencies that aren't immutable, like github. Projects like [https://github.com/gomods/athens Athens] address this by providing a proxy that caches any dependency ever requested of it. If run locally using such a proxy also saves a lot of time downloading dependencies if running in a clean chroot. It can also be set up to run for your CI so that a changed repo doesn't stop your build anymore. This requires exporting GOPROXY in your build to point at the proxy endpoint.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:39, 9 May 2019 (UTC)<br />
<br />
== Go Modules, -git packages, and pkgver ==<br />
<br />
For anyone maintaining -git packages of Go tools, the following can be put in pkgver() to output the version that Go Modules uses if you don't have any semver compatible tags. This way, your packages can have a version that someone requiring your tool can drop in their go.mod file:<br />
<br />
git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:v0.0.0_%cd_%h --no-patch HEAD<br />
<br />
—[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:56, 22 January 2019 (UTC)<br />
<br />
EDIT: and if you want to automatically pick up the tag if they start using them, something like this will work:<br />
<br />
TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0" | sed s/-/_/g)<br />
SUFFIX=$(git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:%cd_%h --no-patch HEAD)<br />
printf '%s_%s' "$TAG" "$SUFFIX"<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 15:55, 22 January 2019 (UTC)<br />
<br />
== More detail about best practices for LDFLAGS and patching makefiles ==<br />
<br />
The page currently suggests patching the Makefile or "Export GOFLAGS - This is less desirable as we are dropping flags from LDFLAGS."<br />
<br />
Could someone explain how LDFLAGS should be used when running go build? That would help me understand why the use of GOFLAGS is less desirable, and how I might patch an existing Makefile to make use of LDFLAGS.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:20, 9 May 2019 (UTC)<br />
<br />
Additionally, can anyone comment on whether CGO_LDFLAGS should be set for projects that use CGO?<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:40, 9 May 2019 (UTC)<br />
<br />
== Add section suggesting the use of GOPROXY when building packages in a clean chroot ==<br />
<br />
When building in a clean chroot, `go build` must fetch all dependencies again. Using a go modules proxy locally like [https://github.com/gomods/athens Athens] dramatically speeds up subsequent builds by allowing all dependencies to be cached for future builds in the clean chroot. I find this extremely helpful when working on my golang PKGBUILDs.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 19:40, 9 May 2019 (UTC)</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=Talk:Go_package_guidelines&diff=572852Talk:Go package guidelines2019-05-09T19:40:52Z<p>Alaskanarcher: Add signature</p>
<hr />
<div>==GOPATH has been removed from /etc/profile.d/go.sh==<br />
/etc/profile.d/go.sh since go 1.2-1 doesn't contain GOPATH and breaks PKGBUILD templates.<br />
https://projects.archlinux.org/svntogit/community.git/commit/trunk?h=packages/go&id=fa3a43875223f38c577093e2b6dc14c604a78b08<br />
I really don't know if /usr/lib/go/src is the right place for sources or they should go to /usr/lib/go/site/src.<br />
In either case, $GOPATH is not reset after sourcing /etc/profile.d/go.sh, so src ends up in $srcdir.<br />
[[User:Vulpesvelox|Vulpesvelox]] ([[User talk:Vulpesvelox|talk]]) 10:44, 24 January 2014 (UTC)<br />
<br />
==Use of GOPATH==<br />
<br />
In the {{ic|check()}} function of the sample PKGBUILD using {{ic|go get}}, the GOPATH is set to {{ic|"$GOPATH:$srcdir"}}. This fails if GOPATH is empty.<br />
Beside, I don't see the point of setting this to a possibly existing $GOPATH when the code.<br />
I think PKGBUILDs should ''not'' rely on the the user's repository, that is, it should be sandboxed. If the users already have a repo, they probably don't need the PKGBUILD in the first place.<br />
All they need is run {{ic|go get -u blah/blah}} and same for testing.<br />
What do you think?<br />
-- [[User:Ambrevar|Ambrevar]] ([[User talk:Ambrevar|talk]]) 07:39, 1 May 2016 (UTC)<br />
<br />
== Generate version using pkgver() ==<br />
<br />
I tweaked one of the examples on this page to include a automatically generated version. Perhaps worth considering:<br />
<br />
<br />
# Maintainer: Harold Drost <baelish@bluecell.net><br />
<br />
pkgname=okta-kubectl-auth-git<br />
pkgver=0.1.1.r2.gd731ef2<br />
pkgrel=1<br />
pkgdesc="Okta auth plugin for kubectl"<br />
arch=("x86_64")<br />
makedepends=("go")<br />
url="https://github.com/jetstack/okta-kubectl-auth"<br />
license=("Apache")<br />
options=('!strip' '!emptydirs')<br />
_gourl="github.com/jetstack/okta-kubectl-auth"<br />
<br />
prepare() {<br />
GOPATH="$srcdir" go get -fix -v -x ${_gourl}<br />
}<br />
<br />
pkgver() {<br />
cd "$srcdir/src/$_gourl"<br />
git describe --long --tags | sed 's/\([^-]*-g\)/r\1/;s/-/./g'<br />
}<br />
<br />
check() {<br />
GOPATH="$GOPATH:$srcdir" go test -v -x ${_gourl}<br />
}<br />
<br />
package() {<br />
mkdir -p "$pkgdir/usr/bin"<br />
install -p -m755 "$srcdir/bin/"* "$pkgdir/usr/bin"<br />
<br />
# Package license (if available)<br />
for f in LICENSE COPYING LICENSE.* COPYING.*; do<br />
if [ -e "$srcdir/src/$_gourl/$f" ]; then<br />
install -Dm644 "$srcdir/src/$_gourl/$f" \<br />
"$pkgdir/usr/share/licenses/$pkgname/$f"<br />
fi<br />
done<br />
}<br />
<br />
# vim:set ts=2 sw=2 et:<br />
<br />
I used {{ic|prepare()}} for the steps which were in {{ic|build()}} as it's run before {{ic|pkgver()}}. I don't think it's unreasonable to call the "go get" step preparation so I don't think it's a concern. --[[User:Baelish|Baelish]] ([[User talk:Baelish|talk]]) 19:52, 27 July 2018 (UTC)<br />
<br />
== Go Modules and Caching ==<br />
<br />
I was recently building a package and noticed that it errored in CI but worked locally. It turned out I had a cached version of a dependency that had been removed upstream. This made me wonder if it wouldn't be good practice to recommend setting: GOCACHE="${srcdir}/cache" or similar before downloading packages so that doing a clean build also destroys the cache.<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:44, 22 January 2019 (UTC)<br />
<br />
I'm not sure if this is desirable for most users. When I'm not building in a chroot, I want makepkg to use my local cache to build with. This saves me time when building. <br />
<br />
The problem you experienced is a side-effect of the golang community relying on third party hosting for dependencies that aren't immutable, like github. Projects like [https://github.com/gomods/athens Athens] address this by providing a proxy that caches any dependency ever requested of it. If run locally using such a proxy also saves a lot of time downloading dependencies if running in a clean chroot. It can also be set up to run for your CI so that a changed repo doesn't stop your build anymore. This requires exporting GOPROXY in your build to point at the proxy endpoint.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:39, 9 May 2019 (UTC)<br />
<br />
== Go Modules, -git packages, and pkgver ==<br />
<br />
For anyone maintaining -git packages of Go tools, the following can be put in pkgver() to output the version that Go Modules uses if you don't have any semver compatible tags. This way, your packages can have a version that someone requiring your tool can drop in their go.mod file:<br />
<br />
git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:v0.0.0_%cd_%h --no-patch HEAD<br />
<br />
—[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:56, 22 January 2019 (UTC)<br />
<br />
EDIT: and if you want to automatically pick up the tag if they start using them, something like this will work:<br />
<br />
TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0" | sed s/-/_/g)<br />
SUFFIX=$(git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:%cd_%h --no-patch HEAD)<br />
printf '%s_%s' "$TAG" "$SUFFIX"<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 15:55, 22 January 2019 (UTC)<br />
<br />
== More detail about best practices for LDFLAGS and patching makefiles ==<br />
<br />
The page currently suggests patching the Makefile or "Export GOFLAGS - This is less desirable as we are dropping flags from LDFLAGS."<br />
<br />
Could someone explain how LDFLAGS should be used when running go build? That would help me understand why the use of GOFLAGS is less desirable, and how I might patch an existing Makefile to make use of LDFLAGS.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:20, 9 May 2019 (UTC)<br />
<br />
Additionally, can anyone comment on whether CGO_LDFLAGS should be set for projects that use CGO?<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:40, 9 May 2019 (UTC)<br />
<br />
== Add section suggesting the use of GOPROXY when building packages in a clean chroot ==<br />
<br />
When building in a clean chroot, `go build` must fetch all dependencies again. Using a go modules proxy locally like [https://github.com/gomods/athens Athens] dramatically speeds up subsequent builds by allowing all dependencies to be cached for future builds in the clean chroot. I find this extremely helpful when working on my golang PKGBUILDs.<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 19:40, 9 May 2019 (UTC)</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=Talk:Go_package_guidelines&diff=572851Talk:Go package guidelines2019-05-09T19:40:35Z<p>Alaskanarcher: /* Add section suggesting the use of GOPROXY when building packages in a clean chroot */ new section</p>
<hr />
<div>==GOPATH has been removed from /etc/profile.d/go.sh==<br />
/etc/profile.d/go.sh since go 1.2-1 doesn't contain GOPATH and breaks PKGBUILD templates.<br />
https://projects.archlinux.org/svntogit/community.git/commit/trunk?h=packages/go&id=fa3a43875223f38c577093e2b6dc14c604a78b08<br />
I really don't know if /usr/lib/go/src is the right place for sources or they should go to /usr/lib/go/site/src.<br />
In either case, $GOPATH is not reset after sourcing /etc/profile.d/go.sh, so src ends up in $srcdir.<br />
[[User:Vulpesvelox|Vulpesvelox]] ([[User talk:Vulpesvelox|talk]]) 10:44, 24 January 2014 (UTC)<br />
<br />
==Use of GOPATH==<br />
<br />
In the {{ic|check()}} function of the sample PKGBUILD using {{ic|go get}}, the GOPATH is set to {{ic|"$GOPATH:$srcdir"}}. This fails if GOPATH is empty.<br />
Beside, I don't see the point of setting this to a possibly existing $GOPATH when the code.<br />
I think PKGBUILDs should ''not'' rely on the the user's repository, that is, it should be sandboxed. If the users already have a repo, they probably don't need the PKGBUILD in the first place.<br />
All they need is run {{ic|go get -u blah/blah}} and same for testing.<br />
What do you think?<br />
-- [[User:Ambrevar|Ambrevar]] ([[User talk:Ambrevar|talk]]) 07:39, 1 May 2016 (UTC)<br />
<br />
== Generate version using pkgver() ==<br />
<br />
I tweaked one of the examples on this page to include a automatically generated version. Perhaps worth considering:<br />
<br />
<br />
# Maintainer: Harold Drost <baelish@bluecell.net><br />
<br />
pkgname=okta-kubectl-auth-git<br />
pkgver=0.1.1.r2.gd731ef2<br />
pkgrel=1<br />
pkgdesc="Okta auth plugin for kubectl"<br />
arch=("x86_64")<br />
makedepends=("go")<br />
url="https://github.com/jetstack/okta-kubectl-auth"<br />
license=("Apache")<br />
options=('!strip' '!emptydirs')<br />
_gourl="github.com/jetstack/okta-kubectl-auth"<br />
<br />
prepare() {<br />
GOPATH="$srcdir" go get -fix -v -x ${_gourl}<br />
}<br />
<br />
pkgver() {<br />
cd "$srcdir/src/$_gourl"<br />
git describe --long --tags | sed 's/\([^-]*-g\)/r\1/;s/-/./g'<br />
}<br />
<br />
check() {<br />
GOPATH="$GOPATH:$srcdir" go test -v -x ${_gourl}<br />
}<br />
<br />
package() {<br />
mkdir -p "$pkgdir/usr/bin"<br />
install -p -m755 "$srcdir/bin/"* "$pkgdir/usr/bin"<br />
<br />
# Package license (if available)<br />
for f in LICENSE COPYING LICENSE.* COPYING.*; do<br />
if [ -e "$srcdir/src/$_gourl/$f" ]; then<br />
install -Dm644 "$srcdir/src/$_gourl/$f" \<br />
"$pkgdir/usr/share/licenses/$pkgname/$f"<br />
fi<br />
done<br />
}<br />
<br />
# vim:set ts=2 sw=2 et:<br />
<br />
I used {{ic|prepare()}} for the steps which were in {{ic|build()}} as it's run before {{ic|pkgver()}}. I don't think it's unreasonable to call the "go get" step preparation so I don't think it's a concern. --[[User:Baelish|Baelish]] ([[User talk:Baelish|talk]]) 19:52, 27 July 2018 (UTC)<br />
<br />
== Go Modules and Caching ==<br />
<br />
I was recently building a package and noticed that it errored in CI but worked locally. It turned out I had a cached version of a dependency that had been removed upstream. This made me wonder if it wouldn't be good practice to recommend setting: GOCACHE="${srcdir}/cache" or similar before downloading packages so that doing a clean build also destroys the cache.<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:44, 22 January 2019 (UTC)<br />
<br />
I'm not sure if this is desirable for most users. When I'm not building in a chroot, I want makepkg to use my local cache to build with. This saves me time when building. <br />
<br />
The problem you experienced is a side-effect of the golang community relying on third party hosting for dependencies that aren't immutable, like github. Projects like [https://github.com/gomods/athens Athens] address this by providing a proxy that caches any dependency ever requested of it. If run locally using such a proxy also saves a lot of time downloading dependencies if running in a clean chroot. It can also be set up to run for your CI so that a changed repo doesn't stop your build anymore. This requires exporting GOPROXY in your build to point at the proxy endpoint.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:39, 9 May 2019 (UTC)<br />
<br />
== Go Modules, -git packages, and pkgver ==<br />
<br />
For anyone maintaining -git packages of Go tools, the following can be put in pkgver() to output the version that Go Modules uses if you don't have any semver compatible tags. This way, your packages can have a version that someone requiring your tool can drop in their go.mod file:<br />
<br />
git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:v0.0.0_%cd_%h --no-patch HEAD<br />
<br />
—[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:56, 22 January 2019 (UTC)<br />
<br />
EDIT: and if you want to automatically pick up the tag if they start using them, something like this will work:<br />
<br />
TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0" | sed s/-/_/g)<br />
SUFFIX=$(git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:%cd_%h --no-patch HEAD)<br />
printf '%s_%s' "$TAG" "$SUFFIX"<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 15:55, 22 January 2019 (UTC)<br />
<br />
== More detail about best practices for LDFLAGS and patching makefiles ==<br />
<br />
The page currently suggests patching the Makefile or "Export GOFLAGS - This is less desirable as we are dropping flags from LDFLAGS."<br />
<br />
Could someone explain how LDFLAGS should be used when running go build? That would help me understand why the use of GOFLAGS is less desirable, and how I might patch an existing Makefile to make use of LDFLAGS.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:20, 9 May 2019 (UTC)<br />
<br />
Additionally, can anyone comment on whether CGO_LDFLAGS should be set for projects that use CGO?<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:40, 9 May 2019 (UTC)<br />
<br />
== Add section suggesting the use of GOPROXY when building packages in a clean chroot ==<br />
<br />
When building in a clean chroot, `go build` must fetch all dependencies again. Using a go modules proxy locally like [https://github.com/gomods/athens Athens] dramatically speeds up subsequent builds by allowing all dependencies to be cached for future builds in the clean chroot. I find this extremely helpful when working on my golang PKGBUILDs.</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=Talk:Go_package_guidelines&diff=572811Talk:Go package guidelines2019-05-09T04:40:46Z<p>Alaskanarcher: Add question about CGO_LDFLAGS</p>
<hr />
<div>==GOPATH has been removed from /etc/profile.d/go.sh==<br />
/etc/profile.d/go.sh since go 1.2-1 doesn't contain GOPATH and breaks PKGBUILD templates.<br />
https://projects.archlinux.org/svntogit/community.git/commit/trunk?h=packages/go&id=fa3a43875223f38c577093e2b6dc14c604a78b08<br />
I really don't know if /usr/lib/go/src is the right place for sources or they should go to /usr/lib/go/site/src.<br />
In either case, $GOPATH is not reset after sourcing /etc/profile.d/go.sh, so src ends up in $srcdir.<br />
[[User:Vulpesvelox|Vulpesvelox]] ([[User talk:Vulpesvelox|talk]]) 10:44, 24 January 2014 (UTC)<br />
<br />
==Use of GOPATH==<br />
<br />
In the {{ic|check()}} function of the sample PKGBUILD using {{ic|go get}}, the GOPATH is set to {{ic|"$GOPATH:$srcdir"}}. This fails if GOPATH is empty.<br />
Beside, I don't see the point of setting this to a possibly existing $GOPATH when the code.<br />
I think PKGBUILDs should ''not'' rely on the the user's repository, that is, it should be sandboxed. If the users already have a repo, they probably don't need the PKGBUILD in the first place.<br />
All they need is run {{ic|go get -u blah/blah}} and same for testing.<br />
What do you think?<br />
-- [[User:Ambrevar|Ambrevar]] ([[User talk:Ambrevar|talk]]) 07:39, 1 May 2016 (UTC)<br />
<br />
== Generate version using pkgver() ==<br />
<br />
I tweaked one of the examples on this page to include a automatically generated version. Perhaps worth considering:<br />
<br />
<br />
# Maintainer: Harold Drost <baelish@bluecell.net><br />
<br />
pkgname=okta-kubectl-auth-git<br />
pkgver=0.1.1.r2.gd731ef2<br />
pkgrel=1<br />
pkgdesc="Okta auth plugin for kubectl"<br />
arch=("x86_64")<br />
makedepends=("go")<br />
url="https://github.com/jetstack/okta-kubectl-auth"<br />
license=("Apache")<br />
options=('!strip' '!emptydirs')<br />
_gourl="github.com/jetstack/okta-kubectl-auth"<br />
<br />
prepare() {<br />
GOPATH="$srcdir" go get -fix -v -x ${_gourl}<br />
}<br />
<br />
pkgver() {<br />
cd "$srcdir/src/$_gourl"<br />
git describe --long --tags | sed 's/\([^-]*-g\)/r\1/;s/-/./g'<br />
}<br />
<br />
check() {<br />
GOPATH="$GOPATH:$srcdir" go test -v -x ${_gourl}<br />
}<br />
<br />
package() {<br />
mkdir -p "$pkgdir/usr/bin"<br />
install -p -m755 "$srcdir/bin/"* "$pkgdir/usr/bin"<br />
<br />
# Package license (if available)<br />
for f in LICENSE COPYING LICENSE.* COPYING.*; do<br />
if [ -e "$srcdir/src/$_gourl/$f" ]; then<br />
install -Dm644 "$srcdir/src/$_gourl/$f" \<br />
"$pkgdir/usr/share/licenses/$pkgname/$f"<br />
fi<br />
done<br />
}<br />
<br />
# vim:set ts=2 sw=2 et:<br />
<br />
I used {{ic|prepare()}} for the steps which were in {{ic|build()}} as it's run before {{ic|pkgver()}}. I don't think it's unreasonable to call the "go get" step preparation so I don't think it's a concern. --[[User:Baelish|Baelish]] ([[User talk:Baelish|talk]]) 19:52, 27 July 2018 (UTC)<br />
<br />
== Go Modules and Caching ==<br />
<br />
I was recently building a package and noticed that it errored in CI but worked locally. It turned out I had a cached version of a dependency that had been removed upstream. This made me wonder if it wouldn't be good practice to recommend setting: GOCACHE="${srcdir}/cache" or similar before downloading packages so that doing a clean build also destroys the cache.<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:44, 22 January 2019 (UTC)<br />
<br />
I'm not sure if this is desirable for most users. When I'm not building in a chroot, I want makepkg to use my local cache to build with. This saves me time when building. <br />
<br />
The problem you experienced is a side-effect of the golang community relying on third party hosting for dependencies that aren't immutable, like github. Projects like [https://github.com/gomods/athens Athens] address this by providing a proxy that caches any dependency ever requested of it. If run locally using such a proxy also saves a lot of time downloading dependencies if running in a clean chroot. It can also be set up to run for your CI so that a changed repo doesn't stop your build anymore. This requires exporting GOPROXY in your build to point at the proxy endpoint.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:39, 9 May 2019 (UTC)<br />
<br />
== Go Modules, -git packages, and pkgver ==<br />
<br />
For anyone maintaining -git packages of Go tools, the following can be put in pkgver() to output the version that Go Modules uses if you don't have any semver compatible tags. This way, your packages can have a version that someone requiring your tool can drop in their go.mod file:<br />
<br />
git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:v0.0.0_%cd_%h --no-patch HEAD<br />
<br />
—[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:56, 22 January 2019 (UTC)<br />
<br />
EDIT: and if you want to automatically pick up the tag if they start using them, something like this will work:<br />
<br />
TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0" | sed s/-/_/g)<br />
SUFFIX=$(git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:%cd_%h --no-patch HEAD)<br />
printf '%s_%s' "$TAG" "$SUFFIX"<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 15:55, 22 January 2019 (UTC)<br />
<br />
== More detail about best practices for LDFLAGS and patching makefiles ==<br />
<br />
The page currently suggests patching the Makefile or "Export GOFLAGS - This is less desirable as we are dropping flags from LDFLAGS."<br />
<br />
Could someone explain how LDFLAGS should be used when running go build? That would help me understand why the use of GOFLAGS is less desirable, and how I might patch an existing Makefile to make use of LDFLAGS.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:20, 9 May 2019 (UTC)<br />
<br />
Additionally, can anyone comment on whether CGO_LDFLAGS should be set for projects that use CGO?<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:40, 9 May 2019 (UTC)</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=Talk:Go_package_guidelines&diff=572810Talk:Go package guidelines2019-05-09T04:39:27Z<p>Alaskanarcher: Respond to initial suggestion of setting GOCACHE</p>
<hr />
<div>==GOPATH has been removed from /etc/profile.d/go.sh==<br />
/etc/profile.d/go.sh since go 1.2-1 doesn't contain GOPATH and breaks PKGBUILD templates.<br />
https://projects.archlinux.org/svntogit/community.git/commit/trunk?h=packages/go&id=fa3a43875223f38c577093e2b6dc14c604a78b08<br />
I really don't know if /usr/lib/go/src is the right place for sources or they should go to /usr/lib/go/site/src.<br />
In either case, $GOPATH is not reset after sourcing /etc/profile.d/go.sh, so src ends up in $srcdir.<br />
[[User:Vulpesvelox|Vulpesvelox]] ([[User talk:Vulpesvelox|talk]]) 10:44, 24 January 2014 (UTC)<br />
<br />
==Use of GOPATH==<br />
<br />
In the {{ic|check()}} function of the sample PKGBUILD using {{ic|go get}}, the GOPATH is set to {{ic|"$GOPATH:$srcdir"}}. This fails if GOPATH is empty.<br />
Beside, I don't see the point of setting this to a possibly existing $GOPATH when the code.<br />
I think PKGBUILDs should ''not'' rely on the the user's repository, that is, it should be sandboxed. If the users already have a repo, they probably don't need the PKGBUILD in the first place.<br />
All they need is run {{ic|go get -u blah/blah}} and same for testing.<br />
What do you think?<br />
-- [[User:Ambrevar|Ambrevar]] ([[User talk:Ambrevar|talk]]) 07:39, 1 May 2016 (UTC)<br />
<br />
== Generate version using pkgver() ==<br />
<br />
I tweaked one of the examples on this page to include a automatically generated version. Perhaps worth considering:<br />
<br />
<br />
# Maintainer: Harold Drost <baelish@bluecell.net><br />
<br />
pkgname=okta-kubectl-auth-git<br />
pkgver=0.1.1.r2.gd731ef2<br />
pkgrel=1<br />
pkgdesc="Okta auth plugin for kubectl"<br />
arch=("x86_64")<br />
makedepends=("go")<br />
url="https://github.com/jetstack/okta-kubectl-auth"<br />
license=("Apache")<br />
options=('!strip' '!emptydirs')<br />
_gourl="github.com/jetstack/okta-kubectl-auth"<br />
<br />
prepare() {<br />
GOPATH="$srcdir" go get -fix -v -x ${_gourl}<br />
}<br />
<br />
pkgver() {<br />
cd "$srcdir/src/$_gourl"<br />
git describe --long --tags | sed 's/\([^-]*-g\)/r\1/;s/-/./g'<br />
}<br />
<br />
check() {<br />
GOPATH="$GOPATH:$srcdir" go test -v -x ${_gourl}<br />
}<br />
<br />
package() {<br />
mkdir -p "$pkgdir/usr/bin"<br />
install -p -m755 "$srcdir/bin/"* "$pkgdir/usr/bin"<br />
<br />
# Package license (if available)<br />
for f in LICENSE COPYING LICENSE.* COPYING.*; do<br />
if [ -e "$srcdir/src/$_gourl/$f" ]; then<br />
install -Dm644 "$srcdir/src/$_gourl/$f" \<br />
"$pkgdir/usr/share/licenses/$pkgname/$f"<br />
fi<br />
done<br />
}<br />
<br />
# vim:set ts=2 sw=2 et:<br />
<br />
I used {{ic|prepare()}} for the steps which were in {{ic|build()}} as it's run before {{ic|pkgver()}}. I don't think it's unreasonable to call the "go get" step preparation so I don't think it's a concern. --[[User:Baelish|Baelish]] ([[User talk:Baelish|talk]]) 19:52, 27 July 2018 (UTC)<br />
<br />
== Go Modules and Caching ==<br />
<br />
I was recently building a package and noticed that it errored in CI but worked locally. It turned out I had a cached version of a dependency that had been removed upstream. This made me wonder if it wouldn't be good practice to recommend setting: GOCACHE="${srcdir}/cache" or similar before downloading packages so that doing a clean build also destroys the cache.<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:44, 22 January 2019 (UTC)<br />
<br />
I'm not sure if this is desirable for most users. When I'm not building in a chroot, I want makepkg to use my local cache to build with. This saves me time when building. <br />
<br />
The problem you experienced is a side-effect of the golang community relying on third party hosting for dependencies that aren't immutable, like github. Projects like [https://github.com/gomods/athens Athens] address this by providing a proxy that caches any dependency ever requested of it. If run locally using such a proxy also saves a lot of time downloading dependencies if running in a clean chroot. It can also be set up to run for your CI so that a changed repo doesn't stop your build anymore. This requires exporting GOPROXY in your build to point at the proxy endpoint.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:39, 9 May 2019 (UTC)<br />
<br />
== Go Modules, -git packages, and pkgver ==<br />
<br />
For anyone maintaining -git packages of Go tools, the following can be put in pkgver() to output the version that Go Modules uses if you don't have any semver compatible tags. This way, your packages can have a version that someone requiring your tool can drop in their go.mod file:<br />
<br />
git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:v0.0.0_%cd_%h --no-patch HEAD<br />
<br />
—[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:56, 22 January 2019 (UTC)<br />
<br />
EDIT: and if you want to automatically pick up the tag if they start using them, something like this will work:<br />
<br />
TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0" | sed s/-/_/g)<br />
SUFFIX=$(git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:%cd_%h --no-patch HEAD)<br />
printf '%s_%s' "$TAG" "$SUFFIX"<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 15:55, 22 January 2019 (UTC)<br />
<br />
== More detail about best practices for LDFLAGS and patching makefiles ==<br />
<br />
The page currently suggests patching the Makefile or "Export GOFLAGS - This is less desirable as we are dropping flags from LDFLAGS."<br />
<br />
Could someone explain how LDFLAGS should be used when running go build? That would help me understand why the use of GOFLAGS is less desirable, and how I might patch an existing Makefile to make use of LDFLAGS.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:20, 9 May 2019 (UTC)</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=Talk:Go_package_guidelines&diff=572809Talk:Go package guidelines2019-05-09T04:22:00Z<p>Alaskanarcher: removed un-needed username</p>
<hr />
<div>==GOPATH has been removed from /etc/profile.d/go.sh==<br />
/etc/profile.d/go.sh since go 1.2-1 doesn't contain GOPATH and breaks PKGBUILD templates.<br />
https://projects.archlinux.org/svntogit/community.git/commit/trunk?h=packages/go&id=fa3a43875223f38c577093e2b6dc14c604a78b08<br />
I really don't know if /usr/lib/go/src is the right place for sources or they should go to /usr/lib/go/site/src.<br />
In either case, $GOPATH is not reset after sourcing /etc/profile.d/go.sh, so src ends up in $srcdir.<br />
[[User:Vulpesvelox|Vulpesvelox]] ([[User talk:Vulpesvelox|talk]]) 10:44, 24 January 2014 (UTC)<br />
<br />
==Use of GOPATH==<br />
<br />
In the {{ic|check()}} function of the sample PKGBUILD using {{ic|go get}}, the GOPATH is set to {{ic|"$GOPATH:$srcdir"}}. This fails if GOPATH is empty.<br />
Beside, I don't see the point of setting this to a possibly existing $GOPATH when the code.<br />
I think PKGBUILDs should ''not'' rely on the the user's repository, that is, it should be sandboxed. If the users already have a repo, they probably don't need the PKGBUILD in the first place.<br />
All they need is run {{ic|go get -u blah/blah}} and same for testing.<br />
What do you think?<br />
-- [[User:Ambrevar|Ambrevar]] ([[User talk:Ambrevar|talk]]) 07:39, 1 May 2016 (UTC)<br />
<br />
== Generate version using pkgver() ==<br />
<br />
I tweaked one of the examples on this page to include a automatically generated version. Perhaps worth considering:<br />
<br />
<br />
# Maintainer: Harold Drost <baelish@bluecell.net><br />
<br />
pkgname=okta-kubectl-auth-git<br />
pkgver=0.1.1.r2.gd731ef2<br />
pkgrel=1<br />
pkgdesc="Okta auth plugin for kubectl"<br />
arch=("x86_64")<br />
makedepends=("go")<br />
url="https://github.com/jetstack/okta-kubectl-auth"<br />
license=("Apache")<br />
options=('!strip' '!emptydirs')<br />
_gourl="github.com/jetstack/okta-kubectl-auth"<br />
<br />
prepare() {<br />
GOPATH="$srcdir" go get -fix -v -x ${_gourl}<br />
}<br />
<br />
pkgver() {<br />
cd "$srcdir/src/$_gourl"<br />
git describe --long --tags | sed 's/\([^-]*-g\)/r\1/;s/-/./g'<br />
}<br />
<br />
check() {<br />
GOPATH="$GOPATH:$srcdir" go test -v -x ${_gourl}<br />
}<br />
<br />
package() {<br />
mkdir -p "$pkgdir/usr/bin"<br />
install -p -m755 "$srcdir/bin/"* "$pkgdir/usr/bin"<br />
<br />
# Package license (if available)<br />
for f in LICENSE COPYING LICENSE.* COPYING.*; do<br />
if [ -e "$srcdir/src/$_gourl/$f" ]; then<br />
install -Dm644 "$srcdir/src/$_gourl/$f" \<br />
"$pkgdir/usr/share/licenses/$pkgname/$f"<br />
fi<br />
done<br />
}<br />
<br />
# vim:set ts=2 sw=2 et:<br />
<br />
I used {{ic|prepare()}} for the steps which were in {{ic|build()}} as it's run before {{ic|pkgver()}}. I don't think it's unreasonable to call the "go get" step preparation so I don't think it's a concern. --[[User:Baelish|Baelish]] ([[User talk:Baelish|talk]]) 19:52, 27 July 2018 (UTC)<br />
<br />
== Go Modules and Caching ==<br />
<br />
I was recently building a package and noticed that it errored in CI but worked locally. It turned out I had a cached version of a dependency that had been removed upstream. This made me wonder if it wouldn't be good practice to recommend setting: GOCACHE="${srcdir}/cache" or similar before downloading packages so that doing a clean build also destroys the cache.<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:44, 22 January 2019 (UTC)<br />
<br />
== Go Modules, -git packages, and pkgver ==<br />
<br />
For anyone maintaining -git packages of Go tools, the following can be put in pkgver() to output the version that Go Modules uses if you don't have any semver compatible tags. This way, your packages can have a version that someone requiring your tool can drop in their go.mod file:<br />
<br />
git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:v0.0.0_%cd_%h --no-patch HEAD<br />
<br />
—[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:56, 22 January 2019 (UTC)<br />
<br />
EDIT: and if you want to automatically pick up the tag if they start using them, something like this will work:<br />
<br />
TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0" | sed s/-/_/g)<br />
SUFFIX=$(git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:%cd_%h --no-patch HEAD)<br />
printf '%s_%s' "$TAG" "$SUFFIX"<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 15:55, 22 January 2019 (UTC)<br />
<br />
== More detail about best practices for LDFLAGS and patching makefiles ==<br />
<br />
The page currently suggests patching the Makefile or "Export GOFLAGS - This is less desirable as we are dropping flags from LDFLAGS."<br />
<br />
Could someone explain how LDFLAGS should be used when running go build? That would help me understand why the use of GOFLAGS is less desirable, and how I might patch an existing Makefile to make use of LDFLAGS.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:20, 9 May 2019 (UTC)</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=Talk:Go_package_guidelines&diff=572808Talk:Go package guidelines2019-05-09T04:20:03Z<p>Alaskanarcher: /* More detail about best practices for LDFLAGS and patching makefiles */ new section</p>
<hr />
<div>==GOPATH has been removed from /etc/profile.d/go.sh==<br />
/etc/profile.d/go.sh since go 1.2-1 doesn't contain GOPATH and breaks PKGBUILD templates.<br />
https://projects.archlinux.org/svntogit/community.git/commit/trunk?h=packages/go&id=fa3a43875223f38c577093e2b6dc14c604a78b08<br />
I really don't know if /usr/lib/go/src is the right place for sources or they should go to /usr/lib/go/site/src.<br />
In either case, $GOPATH is not reset after sourcing /etc/profile.d/go.sh, so src ends up in $srcdir.<br />
[[User:Vulpesvelox|Vulpesvelox]] ([[User talk:Vulpesvelox|talk]]) 10:44, 24 January 2014 (UTC)<br />
<br />
==Use of GOPATH==<br />
<br />
In the {{ic|check()}} function of the sample PKGBUILD using {{ic|go get}}, the GOPATH is set to {{ic|"$GOPATH:$srcdir"}}. This fails if GOPATH is empty.<br />
Beside, I don't see the point of setting this to a possibly existing $GOPATH when the code.<br />
I think PKGBUILDs should ''not'' rely on the the user's repository, that is, it should be sandboxed. If the users already have a repo, they probably don't need the PKGBUILD in the first place.<br />
All they need is run {{ic|go get -u blah/blah}} and same for testing.<br />
What do you think?<br />
-- [[User:Ambrevar|Ambrevar]] ([[User talk:Ambrevar|talk]]) 07:39, 1 May 2016 (UTC)<br />
<br />
== Generate version using pkgver() ==<br />
<br />
I tweaked one of the examples on this page to include a automatically generated version. Perhaps worth considering:<br />
<br />
<br />
# Maintainer: Harold Drost <baelish@bluecell.net><br />
<br />
pkgname=okta-kubectl-auth-git<br />
pkgver=0.1.1.r2.gd731ef2<br />
pkgrel=1<br />
pkgdesc="Okta auth plugin for kubectl"<br />
arch=("x86_64")<br />
makedepends=("go")<br />
url="https://github.com/jetstack/okta-kubectl-auth"<br />
license=("Apache")<br />
options=('!strip' '!emptydirs')<br />
_gourl="github.com/jetstack/okta-kubectl-auth"<br />
<br />
prepare() {<br />
GOPATH="$srcdir" go get -fix -v -x ${_gourl}<br />
}<br />
<br />
pkgver() {<br />
cd "$srcdir/src/$_gourl"<br />
git describe --long --tags | sed 's/\([^-]*-g\)/r\1/;s/-/./g'<br />
}<br />
<br />
check() {<br />
GOPATH="$GOPATH:$srcdir" go test -v -x ${_gourl}<br />
}<br />
<br />
package() {<br />
mkdir -p "$pkgdir/usr/bin"<br />
install -p -m755 "$srcdir/bin/"* "$pkgdir/usr/bin"<br />
<br />
# Package license (if available)<br />
for f in LICENSE COPYING LICENSE.* COPYING.*; do<br />
if [ -e "$srcdir/src/$_gourl/$f" ]; then<br />
install -Dm644 "$srcdir/src/$_gourl/$f" \<br />
"$pkgdir/usr/share/licenses/$pkgname/$f"<br />
fi<br />
done<br />
}<br />
<br />
# vim:set ts=2 sw=2 et:<br />
<br />
I used {{ic|prepare()}} for the steps which were in {{ic|build()}} as it's run before {{ic|pkgver()}}. I don't think it's unreasonable to call the "go get" step preparation so I don't think it's a concern. --[[User:Baelish|Baelish]] ([[User talk:Baelish|talk]]) 19:52, 27 July 2018 (UTC)<br />
<br />
== Go Modules and Caching ==<br />
<br />
I was recently building a package and noticed that it errored in CI but worked locally. It turned out I had a cached version of a dependency that had been removed upstream. This made me wonder if it wouldn't be good practice to recommend setting: GOCACHE="${srcdir}/cache" or similar before downloading packages so that doing a clean build also destroys the cache.<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:44, 22 January 2019 (UTC)<br />
<br />
== Go Modules, -git packages, and pkgver ==<br />
<br />
For anyone maintaining -git packages of Go tools, the following can be put in pkgver() to output the version that Go Modules uses if you don't have any semver compatible tags. This way, your packages can have a version that someone requiring your tool can drop in their go.mod file:<br />
<br />
git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:v0.0.0_%cd_%h --no-patch HEAD<br />
<br />
—[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 05:56, 22 January 2019 (UTC)<br />
<br />
EDIT: and if you want to automatically pick up the tag if they start using them, something like this will work:<br />
<br />
TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0" | sed s/-/_/g)<br />
SUFFIX=$(git show --abbrev-commit --abbrev=12 --date='format:%G%m%d%H%M%S' --pretty=format:%cd_%h --no-patch HEAD)<br />
printf '%s_%s' "$TAG" "$SUFFIX"<br />
<br />
[[User:SamWhited|SamWhited]] ([[User talk:SamWhited|talk]]) 15:55, 22 January 2019 (UTC)<br />
<br />
== More detail about best practices for LDFLAGS and patching makefiles ==<br />
<br />
The page currently suggests patching the Makefile or "Export GOFLAGS - This is less desirable as we are dropping flags from LDFLAGS."<br />
<br />
Could someone explain how LDFLAGS should be used when running go build? That would help me understand why the use of GOFLAGS is less desirable, and how I might patch an existing Makefile to make use of LDFLAGS.<br />
<br />
[[User:Alaskanarcher|Alaskanarcher]] ([[User talk:Alaskanarcher|talk]]) 04:20, 9 May 2019 (UTC) alaskanarcher</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=Xrandr&diff=511889Xrandr2018-02-24T08:18:09Z<p>Alaskanarcher: xrasengan is now in the AUR so I changed the link to the AUR package.</p>
<hr />
<div>{{Lowercase title}}<br />
[[Category:X server]]<br />
[[ru:Xrandr]]<br />
[[ja:Xrandr]]<br />
[[zh-hans:Xrandr]]<br />
{{Related articles start}}<br />
{{Related|Xorg}}<br />
{{Related|Multihead}}<br />
{{Related articles end}}<br />
<br />
''xrandr'' is an official configuration utility to the [[Wikipedia:RandR|RandR]] [[Wikipedia:X Window System|X Window System]] extension. It can be used to set the size, orientation or reflection of the outputs for a screen. For configuring multiple monitors see the [[Multihead]] page.<br />
<br />
== Installation ==<br />
<br />
[[Install]] {{Pkg|xorg-xrandr}}. Graphical front ends such as {{Pkg|arandr}}, {{Pkg|lxrandr}} or {{Pkg|lxrandr-gtk3}} are also available.<br />
<br />
== Testing configuration ==<br />
<br />
When run without any option, ''xrandr'' shows the names of different outputs available on the system ({{ic|VGA-1}}, {{ic|HDMI-1}}, etc.) and resolutions available on each, with a '''*''' after the current one and a '''+''' after the preferred one :<br />
<br />
{{hc|xrandr|<br />
Screen 0: minimum 320 x 200, current 3200 x 1080, maximum 8192 x 8192<br />
VGA-1 disconnected (normal left inverted right x axis y axis)<br />
HDMI-1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 531mm x 299mm<br />
1920x1080 59.93 + 60.00* 50.00 59.94 <br />
1920x1080i 60.00 50.00 59.94 <br />
1680x1050 59.88 <br />
…<br />
}}<br />
{{Note|If your resolution is not present in the above list, see [[#Adding undetected resolutions]]}}<br />
<br />
You can use ''xrandr'' to set different resolution (must be present in the above list) on some output:<br />
<br />
$ xrandr --output HDMI-1 --mode 1920x1080<br />
<br />
When multiple refresh rates are present in the list, it may be changed by the {{ic|--rate}} option, either at the same time or independently. For example:<br />
<br />
$ xrandr --output HDMI-1 --mode 1920x1080 --rate 60<br />
<br />
The {{ic|--auto}} option will turn the specified output on if it is off and set the preferred (maximum) resolution:<br />
<br />
$ xrandr --output HDMI-1 --auto<br />
<br />
It is possible to specify multiple outputs in one command, e.g. to turn off {{ic|HDMI-1}} and turn on {{ic|HDMI-2}} with preferred resolution:<br />
<br />
$ xrandr --output HDMI-1 --off --output HDMI-2 --auto<br />
<br />
{{Note|<br />
* Changes you make using ''xrandr'' will only last through the current session.<br />
* ''xrandr'' has a lot more capabilities - see {{man|1|xrandr}} for details.<br />
}}<br />
<br />
== Configuration ==<br />
<br />
''xrandr'' is just a simple interface to the RandR extension and has no configuration file. However, there are multiple ways of achieving persistent configuration:<br />
<br />
# The RandR extension can be configured via [[Xorg#Configuration|X configuration files]], see [[Multihead#RandR]] for details. This method provides only static configuration.<br />
# If you need dynamic configuration, you need to execute ''xrandr'' commands each time X server starts. See [[Autostarting#Graphical]] for details. This method has the disadvantage of occurring fairly late in the startup process, thus it will not alter the resolution of the [[display manager]] if you use one.<br />
# Custom scripts calling ''xrandr'' can be bound to events (for example when external monitor is plugged in), see [[acpid]] for details. The [[#Scripts]] section provides you with some example scripts that might be useful for this purpose.<br />
<br />
{{Tip|Both KDM and GDM have startup scripts that are executed when X is initiated. For GDM, these are in {{ic|/etc/gdm/}}, while for KDM this is done at {{ic|/usr/share/config/kdm/Xsetup}} and for SDDM at {{ic|/usr/share/sddm/scripts/Xsetup}}. This method requires root access and mucking around in system config files, but will take effect earlier in the startup process than using xprofile.}}<br />
<br />
=== Scripts ===<br />
<br />
==== Toggle external monitor ====<br />
<br />
This script toggles between an external monitor (specified by {{ic|$extern}}) and a default monitor (specified by {{ic|$intern}}), so that only one monitor is active at a time.<br />
<br />
The default monitor should be connected when running the script, which is always true for a laptop.<br />
<br />
{{bc|<nowiki><br />
#!/bin/bash<br />
intern=LVDS1<br />
extern=VGA1<br />
<br />
if xrandr | grep "$extern disconnected"; then<br />
xrandr --output "$extern" --off --output "$intern" --auto<br />
else<br />
xrandr --output "$intern" --off --output "$extern" --auto<br />
fi<br />
</nowiki>}}<br />
<br />
{{Note|To leave the external monitor enabled, replace the ''else'' clause with {{ic|xrandr --output "$intern" --primary --auto --output "$extern" --right-of "$intern" --auto}}.}}<br />
<br />
==== Manage 2-monitors ====<br />
<br />
{{AUR|mons}} is a POSIX-compliant shell script to quickly manage 2-monitors display.<br />
<br />
It provides well-known modes like computer, duplicate, extend and projector mode as well as selecting and positioning one or two monitors among those plugged in (for more details, see [https://github.com/Ventto/mons mons]).<br />
<br />
==== Example 3 ====<br />
<br />
{{Accuracy|1=Basic shell mistakes: relying on quoting errors to relay arguments (instead of using arrays), convoluted grep+sed pipes (instead of awk), echo -e (instead of printf), ancient backtick format (instead of {{ic|$()}})}}<br />
<br />
This script iterates through connected monitors, selects currently active monitor, turns next one on and the others off:<br />
<br />
{{bc|<nowiki><br />
# get info from xrandr<br />
connectedOutputs=$(xrandr | grep " connected" | sed -e "s/\([A-Z0-9]\+\) connected.*/\1/")<br />
activeOutput=$(xrandr | grep -E " connected (primary )?[1-9]+" | sed -e "s/\([A-Z0-9]\+\) connected.*/\1/")<br />
<br />
# initialize variables<br />
execute="xrandr "<br />
default="xrandr "<br />
i=1<br />
switch=0<br />
<br />
for display in $connectedOutputs<br />
do<br />
# build default configuration<br />
if [ $i -eq 1 ]<br />
then<br />
default=$default"--output $display --auto "<br />
else<br />
default=$default"--output $display --off "<br />
fi<br />
<br />
# build "switching" configuration<br />
if [ $switch -eq 1 ]<br />
then<br />
execute=$execute"--output $display --auto "<br />
switch=0<br />
else<br />
execute=$execute"--output $display --off "<br />
fi<br />
<br />
# check whether the next output should be switched on<br />
if [ $display = $activeOutput ]<br />
then<br />
switch=1<br />
fi<br />
<br />
i=$(( $i + 1 ))<br />
done<br />
<br />
# check if the default setup needs to be executed then run it<br />
echo "Resulting Configuration:"<br />
if [ -z "$(echo $execute | grep "auto")" ]<br />
then<br />
echo "Command: $default"<br />
`$default`<br />
else<br />
echo "Command: $execute"<br />
`$execute`<br />
fi<br />
echo -e "\n$(xrandr)"<br />
</nowiki>}}<br />
<br />
==== Avoid X crash with xrasengan ====<br />
<br />
Use this workaround to turn on connected outputs that may be in suspend mode and hence shown as disconnected, as is often the case of DisplayPort monitors:<br />
<br />
{{bc|<nowiki><br />
declare -i count=2<br />
declare -i seconds=1<br />
<br />
while ((count)); do<br />
xrandr >/dev/null<br />
sleep $seconds<br />
((count--))<br />
done<br />
</nowiki>}}<br />
<br />
{{AUR|xrasengan}} is an xrandr wrapper with this workaround built in.<br />
<br />
$ xrasengan --force -on DisplayPort-0 -off HDMI-0<br />
<br />
With the {{ic|--force}} option, ''xrasengan'' will update status of all outputs before HDMI-0 is turned off, avoiding an X crash if they were the only connected/active outputs.<br />
<br />
To force reload current settings, ''xrasengan'' provides a {{ic|--try-reload-active-layout}} option, which uses {{ic|--force}} and ''unxrandr'' from the {{Pkg|arandr}} package to assemble the command line:<br />
<br />
$ xrasengan --try-reload-active-layout<br />
<br />
This can be used in systemd unit or in a keyboard binding to avoid blank screen when resuming DisplayPort monitors from suspend.<br />
<br />
== Troubleshooting ==<br />
<br />
=== Adding undetected resolutions ===<br />
<br />
Due to buggy hardware or drivers, your monitor's correct resolutions may not always be detected by xrandr. For example, the EDID data block queried from the monitor may be incorrect. However, we can add the desired resolutions to xrandr. Also, this same procedure can be used to add refresh rates you know are supported, but not enabled by your driver.<br />
<br />
First we run {{ic|gtf}} or {{ic|cvt}} to get the '''Modeline''' for the resolution we want:<br />
<br />
{{hc|$ cvt 1280 1024|<br />
# 1280x1024 59.89 Hz (CVT 1.31M4) hsync: 63.67 kHz; pclk: 109.00 MHz<br />
Modeline "1280x1024_60.00" 109.00 1280 1368 1496 1712 1024 1027 1034 1063 -hsync +vsync<br />
}}<br />
<br />
{{Tip|1=For some LCD screens (e.g. Samsung 2343NW, Acer XB280HK), the command {{ic|cvt -r}} (= with reduced blanking) is to be used.}}<br />
<br />
{{Note|If the Intel video driver {{pkg|xf86-video-intel}} is used, it may report the desired resolution along with its properties in {{ic|/var/log/Xorg.0.log}} — use that first if it is different from the output of {{ic|gtf}} or {{ic|cvt}}. For instance, the log and its use with xrandr:<br />
[ 45.063] (II) intel(0): clock: 241.5 MHz Image Size: 597 x 336 mm<br />
[ 45.063] (II) intel(0): h_active: 2560 h_sync: 2600 h_sync_end 2632 h_blank_end 2720 h_border: 0<br />
[ 45.063] (II) intel(0): v_active: 1440 v_sync: 1443 v_sync_end 1448 v_blanking: 1481 v_border: 0<br />
<br />
xrandr --newmode "2560x1440" 241.50 2560 2600 2632 2720 1440 1443 1448 1481 -hsync +vsync<br />
}}<br />
<br />
Then we create a new xrandr mode. Note that the Modeline keyword needs to be ommited.<br />
<br />
$ xrandr --newmode "1280x1024_60.00" 109.00 1280 1368 1496 1712 1024 1027 1034 1063 -hsync +vsync<br />
<br />
After creating it we need an extra step to add this new mode to our current output (VGA1). We use just the name of the mode, since the parameters have been set previously.<br />
<br />
$ xrandr --addmode VGA1 1280x1024_60.00<br />
<br />
Now we change the resolution of the screen to the one we just added:<br />
<br />
$ xrandr --output VGA1 --mode 1280x1024_60.00<br />
<br />
Note that these settings only take effect during this session. <br />
<br />
If you are not sure about the resolution you will test, you may add a {{ic|sleep 5}} and a safe resolution command line following, like this:<br />
<br />
$ xrandr --output VGA1 --mode 1280x1024_60.00 && sleep 5 && xrandr --newmode "1024x768-safe" 65.00 1024 1048 1184 1344 768 771 777 806 -HSync -VSync && xrandr --addmode VGA1 1024x768-safe && xrandr --output VGA1 --mode 1024x768-safe<br />
<br />
Also, change {{ic|VGA1}} to correct output name.<br />
<br />
==== EDID checksum is invalid ====<br />
<br />
If the previous method results in an {{ic|*ERROR* EDID checksum is invalid}} error during boot, see [[KMS#Forcing modes and EDID]] and [http://askubuntu.com/questions/201081/how-can-i-make-linux-behave-better-when-edid-is-unavailable].<br />
<br />
Or {{ic|xrandr --addmode}} might give you the error {{ic|X Error of failed request: BadMatch}}. NVIDIA users should read [[NVIDIA/Troubleshooting#xrandr BadMatch]]. {{ic|BadMatch}} could indicate an invalid EDID checksum. To verify that this is the case, run X in verbose mode (e.g. {{ic|startx -- -logverbose 6}}) and check your Xorg log for messages about a bad EDID.<br />
<br />
==== Screen resolution reverts back after a blink ====<br />
<br />
If you use [[GNOME]] and your monitor doesn't have an EDID, above [[#Adding undetected resolutions]] might not work, with your screen just blinking once, after {{ic|xrandr --output}}.<br />
<br />
Poke around with {{ic|~/.config/monitors.xml}}, or delete the file completely, and then reboot.<br />
<br />
It is better explained in [http://unix.stackexchange.com/questions/184941/gnome-prevents-high-resolution-vga-without-edid-info-over-vga this] article.<br />
<br />
=== Permanently adding undetected resolutions ===<br />
<br />
Once a suitable resolution is found using {{ic|xrandr}}, the mode can be permanently added by creating an entry in {{ic|/etc/X11/xorg.conf.d/}}:<br />
<br />
{{hc|/etc/X11/xorg.conf.d/10-monitor.conf|<br />
Section "Monitor"<br />
Identifier "VGA1"<br />
Modeline "1280x1024_60.00" 109.00 1280 1368 1496 1712 1024 1027 1034 1063 -hsync +vsync<br />
Option "PreferredMode" "1280x1024_60.00"<br />
EndSection<br />
<br />
Section "Screen"<br />
Identifier "Screen0"<br />
Monitor "VGA1"<br />
DefaultDepth 24<br />
SubSection "Display"<br />
Modes "1280x1024_60.00"<br />
EndSubSection<br />
EndSection<br />
<br />
Section "Device"<br />
Identifier "Device0"<br />
Driver "intel"<br />
EndSection}}<br />
<br />
Replace {{ic|intel}} with the right driver, e.g. {{ic|nvidia}}. When the X server is restarted, you should be able to set the new resolution.<br />
<br />
If this doesn't work for you, try removing the Screen and Device sections and just leaving the Monitor section. [https://bbs.archlinux.org/viewtopic.php?id=225134]<br />
<br />
=== Resolution lower than expected ===<br />
<br />
{{Tip|Try [[#Adding undetected resolutions]] first, if it doesn't work, you may try this method.}}<br />
<br />
If your video card is recognized but the resolution is lower than you expect, you may try this.<br />
<br />
Background: ATI X1550 based video card and two LCD monitors DELL 2408(up to 1920x1200) and Samsung 206BW(up to 1680x1050). Upon first login after installation, the resolution default to 1152x864. xrandr does not list any resolution higher than 1152x864. You may want to try editing /etc/X11/xorg.conf, add a section about virtual screen, logout, login and see if this helps. If not then read on.<br />
<br />
Change xorg.conf<br />
{{hc|/etc/X11/xorg.conf|<br />
Section "Screen"<br />
...<br />
SubSection "Display"<br />
Virtual 3600 1200<br />
EndSubSection<br />
EndSection<br />
}}<br />
<br />
About the numbers: DELL on the left and Samsung on the right. So the virtual width is of sum of both LCD width 3600=1920+1680; Height then is figured as the max of them, which is max(1200,1050)=1200. If you put one LCD above the other, use this calculation instead: (max(width1, width2), height1+height2).<br />
<br />
=== Correction of overscan tv resolutions ===<br />
<br />
With a flat panel TV, [[w:overscan]] looks like the picture is "zoomed in" so the edges are cut off.<br />
<br />
Check your TV if there is a parameter to change. If not, apply an {{ic|underscan}} and change border values. <br />
The required {{ic|underscan vborder}} and {{ic|underscan hborder}} values can be different for you, just check it and change it by more or less.<br />
<br />
{{ic|$ xrandr --output HDMI-0 --set underscan on --set "underscan vborder" 25 --set "underscan hborder" 40 }}<br />
<br />
=== Full RGB in HDMI ===<br />
<br />
It may occur that the [[Intel]] driver will not configure correctly the output of the HDMI monitor. It will set a limited color range (16-235) using the [https://patchwork.kernel.org/patch/1972181/ Broadcast RGB property], and the black will not look black, it will be grey.<br />
<br />
See [[Intel graphics#Weathered colors (color range problem)]].<br />
<br />
== See also ==<br />
* https://wiki.ubuntu.com/X/Config/Resolution<br />
* [[debian:XStrikeForce/HowToRandR12|Debian Wiki - RandR 1.2 tutorial]]<br />
* [http://www.thinkwiki.org/wiki/Xorg_RandR_1.2 Xorg RandR 1.2 on ThinkWiki]<br />
* [http://www.x.org/wiki/FAQVideoModes#ObtainingmodelinesfromWindowsprogramPowerStrip FAQVideoModes - more information about modelines]</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=Mobile_broadband_modem&diff=458708Mobile broadband modem2016-12-07T10:37:36Z<p>Alaskanarcher: After further testing on a clean install I found that only libmbim was necessary to bring up my modem</p>
<hr />
<div>[[Category:Modems]]<br />
[[ja:USB 3G モデム]]<br />
[[ru:USB 3G Modem]]<br />
{{Related articles start}}<br />
{{Related|wvdial}}<br />
{{Related|Direct Modem Connection}}<br />
{{Related|3G and GPRS modems with pppd}}<br />
{{Related|:Category:Modems}}<br />
{{Related articles end}}<br />
<br />
{{Poor writing|does not conform to [[Help:Style]]}}<br />
A number of mobile telephone networks around the world offer mobile internet connections over UMTS (or EDGE or GSM) using a portable USB modem device.<br />
<br />
==Remove the PIN==<br />
First of all use your SIM card in a normal phone and disable the PIN request if present. If the SIM card asks the PIN wvdial will not work.<br />
<br />
Failing that, you can also use ''mmcli'' to unlock the SIM card:<br />
<br />
# mmcli -i ''SIMNUMBER'' --pin=XXXX<br />
<br />
where ''SIMNUMBER'' can be found using {{ic|mmcli -L}} and {{ic|mmcli -m X}}.<br />
<br />
==Device identification==<br />
Examine the output of<br />
$ lsusb<br />
which will show the vendor and product IDs of the device. Note that some devices will show ''two'' different product IDs at different times as explained below.<br />
<br />
==Mode switching==<br />
<br />
Often these devices will have two modes (1) USB flash memory storage (2) USB Modem. The first mode, sometimes known as ZeroCD, is often used to deliver an internet communications program for another operating system and is generally of no interest to Linux users. Additionally some have a slot into which the user can insert an additional flash memory card.<br />
<br />
A useful utility for switching these devices into modem mode is {{Pkg|usb_modeswitch}}, available in the [[official repositories]].<br />
<br />
Udev rules are supplied with the package in {{ic|/lib/udev/rules.d/40-usb_modeswitch.rules}}. This contains entries for many devices, which it will switch to modem mode upon insertion.<br />
<br />
When a device is switched, its product ID may change to a different value. The vendor ID will remain unchanged. This can be seen in the output of {{ic|lsusb}}.<br />
<br />
Some devices are supported in the USB serial kernel module called "option" (after the Option devices, but not limited to just those) and may be used without usb_modeswitch.<br />
<br />
Udev itself included a utility called {{ic|/lib/udev/modem-modeswitch}}. In udev 157 this was renamed to {{ic|/lib/udev/mobile-action-modeswitch}} and morphed into a tool that only switches Mobile Action cables. For other devices use {{ic|usb_modeswitch}}.<br />
<br />
{{Note|You can find an alternative way to do this base on eject command [[ZTE_MF110/MF190#Switch_from_CD_mode_to_modem_mode_on_the_device|here]].}}<br />
<br />
==Connection==<br />
<br />
=== Network Manager ===<br />
<br />
After installing {{Pkg|usbutils}} and {{Pkg|usb_modeswitch}}, you just need to install {{Pkg|modemmanager}} to make the modem work with [[NetworkManager]].<br />
<br />
Make sure both NetworkManager and ModemManager services are running, see [[systemd#Using units]] for details.<br />
<br />
Make sure {{Pkg|mobile-broadband-provider-info}} and {{Pkg|nm-connection-editor}} are installed.<br />
<br />
A system restart might be necessary for ModemManager to detect the USB modem. After you restart the NetworkManager-applet and plug the modem in again NetworkManager should recognize the modem in the menu without further configuration. Setting up the modem in NetworkManager is self-explanatory, you should only need the login-information provided by your network provider.<br />
<br />
Whilst running ModemManager gammu will not work. SMS and Ussd codes can be still used with {{Pkg|modem-manager-gui}}.<br />
<br />
=== pppd ===<br />
[[pppd]] can be used to configure 3g connections. Step by step instruction is available on [[3G and GPRS modems with pppd]]. Optionally, {{Aur|pppconfig}} can be used to simplify the pppd configuration using dialog interface.<br />
<br />
=== wvdial ===<br />
See main article: [[wvdial]]<br />
<br />
=== netctl ===<br />
Netctl can be used to establish a connection using a USB modem. An example configuration file provided by {{Pkg|netctl}} is located at {{ic|/etc/netctl/examples/mobile_ppp}}. Minimally you will probably have to specify<br />
{{hc|/etc/netctl/mobile_ppp|<br />
<nowiki>Interface=cdc-wdmX<br />
Connection=mobile_ppp<br />
PhoneNumber=XxxxXXXX<br />
AccessPointName=Broadband</nowiki><br />
}}<br />
<br />
See the [[netctl]] article and [https://github.com/joukewitteveen/netctl/blob/master/docs/netctl.profile.5.txt netctl.profile] for more information.<br />
<br />
=== libmbim ===<br />
Install {{ic|libmbim}} from the official repositories. To bring up the modem you can use {{ic|mbim-network}} which is a wrapper for {{ic|mmcli}} calls. First create a profile for mbim-network.<br />
{{hc|/etc/mbim-network.conf|<br />
<nowiki>APN=Broadband</nowiki>}}<br />
Now connect to the network with {{bc|# mbim-network /dev/cdc-wdmX start}}. Then bring up the interface and get an ip address:<br />
{{bc|<br />
# ip link set wwanY up<br />
# dhcpcd wwanY<br />
}}<br />
<br />
=== sakis3g ===<br />
There may be the chance that the modem stick is supported by [http://www.sakis3g.com/ sakis3g] which is an all in one command line script and automates all the steps above. Install {{AUR|sakis3g}} from the [[AUR]].<br />
<br />
=== Low connection speed ===<br />
Someone claims that the connection speed under Linux is lower than Windows: [https://bbs.archlinux.org/viewtopic.php?id=111513]<br />
<br />
A short summary for possible solutions which are not fully verified.<br />
In most of conditions, the low speed is caused by bad receiver signals and too many people in cell. But you still could use the following method to try to improve the connection speed.<br />
<br />
====QoS parameter====<br />
The {{ic|AT+CGEQMIN}} and {{ic|AT+CGEQREQ}} commands can be used to set the QoS. It is also possible to decrease and limit the connect speed. <br />
Add the following {{ic|Init}} command in {{ic|/etc/wvdial.conf}}:<br />
Init6 = AT+CGEQMIN=1,4,64,640,64,640<br />
Init7 = AT+CGEQREQ=1,4,64,640,64,640<br />
<br />
====Baud parameter====<br />
Baud parameter in wvdial.conf could be used to increase the connection speed.<br />
<br />
Baud = 460800<br />
<br />
But the official Huawei E261 windows application set the Baud=9600 under Windows Vista.<br />
More verifications are needed to double check this point.<br />
<br />
===Monitor used bandwith===<br />
Frequently a 3G connection obtained via a mobile phone operator comes with restricted bandwidth, so that you are only allowed to use a certain bandwidth per time (e.g. 1GB per month). While it is quite straight-forward to know which type of network applications are pretty bandwidth extensive (e.g. video streaming, gaming, torrent, etc.), it may be difficult to keep an overview about overall consumed bandwidth. <br />
<br />
A number of tools are available to help with that. Two console tools are {{pkg|vnstat}}, which allows to keep track of bandwith over time, and {{pkg|iftop}} to monitor bandwidth of individual sessions. If you are a [[KDE]] user, {{Pkg|knemo}} might help. All are available in the [[official repositories]].<br />
<br />
==Reading SMS==<br />
This was tested on a Huawei EM770W (GTM382E) 3g card integrated into an Acer Aspire AS3810TG laptop.<br />
<br />
{{bc|<br />
$ pacman -S gnokii<br />
$ mkdir -p $XDG_CONFIG_HOME/gnokii<br />
}}<br />
<br />
Usually the configuration directory is {{ic|~/.config/gnokii}}. <br />
<br />
{{bc|<br />
$ cp /etc/gnokiirc ~/.config/gnokii/config<br />
}}<br />
<br />
edit {{ic|~/.config/gnokii/config}} as follows:<br />
<br />
{{bc|1=<br />
port = /dev/ttyUSB0<br />
}}<br />
<br />
You may have to use a different port depending on your configuration, for example {{ic|/dev/ttyUSB1}} or something else:<br />
<br />
{{bc|1=<br />
model = AT<br />
connection = serial<br />
}}<br />
<br />
You need to be part of the {{ic|uucp}} group to use {{ic|/dev/ttyUSB0}}, for example if your user is called "x": <br />
{{bc|<br />
# gpasswd -a x uucp<br />
$ newgrp uucp<br />
}}<br />
The ''newgrp'' command allows you to take advantage of the new group assignment immediately without having to logout/login.<br />
<br />
Then launch gnokii:<br />
{{bc|<br />
$ xgnokii<br />
}}<br />
Click on the "SMS" icon button, a window opens up. Then click: "messages->activate sms reading". Your messages will show up in the window.<br />
===command line script===<br />
A small command line script using gnokii to read SMS on your SIM card (not phone memory) without having to start a GUI:<br />
$ gnokii --getsms SM 0 end 2>&1|grep Text -A1 -B3|grep -v Text<br />
What it does:<br />
{{bc|<nowiki><br />
gnokii # invoke gnokii<br />
--getsms SM 0 end # read SMS from SM-memory location (=SIM card) starting at 0 and reading all occupied memory locations ("end")<br />
2>&1 # connect STDERR to STDOUT to make sure the output from the --getsms command can be piped to grep<br />
|grep Text # pipe output from gnokii to grep, anchoring at output containing "Text"<br />
-A1 -B3 # print one line after the matched pattern and three lines before the matched pattern<br />
|grep -v Text # grep result to another grep to exclude the "Text" line (-v for inverting the pattern)<br />
</nowiki>}}<br />
Granted this does not work very well if your SMS contains the word "Text", but you may adapt the script to your liking.<br />
<br />
== Fix image quality ==<br />
If you are getting low quality images while browsing the web over a mobile broadband connection with the hints "shift+r improves the quality of this image" and "shift+a improves the quality of all images on this page", follow these instructions:<br />
<br />
[[Install]] {{Pkg|tinyproxy}}, available in the [[official repositories]].<br />
<br />
Edit /etc/tinyproxy/tinyproxy.conf and insert the following two lines:<br />
AddHeader "Pragma" "No-Cache"<br />
AddHeader "Cache-Control" "No-Cache"<br />
Start tinyproxy:<br />
systemctl start tinyproxy<br />
Configure your browser to use localhost:8888 as a proxy server and you are all done. This is especially useful if you are using, for example, Google Chrome which, unlike Firefox, does not allow you to modify the Pragma and Cache-Control headers.</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=Mobile_broadband_modem&diff=458700Mobile broadband modem2016-12-07T07:23:37Z<p>Alaskanarcher: Added detail about mbim-network profile /etc/mbim-network.conf</p>
<hr />
<div>[[Category:Modems]]<br />
[[ja:USB 3G モデム]]<br />
[[ru:USB 3G Modem]]<br />
{{Related articles start}}<br />
{{Related|wvdial}}<br />
{{Related|Direct Modem Connection}}<br />
{{Related|3G and GPRS modems with pppd}}<br />
{{Related|:Category:Modems}}<br />
{{Related articles end}}<br />
<br />
{{Poor writing|does not conform to [[Help:Style]]}}<br />
A number of mobile telephone networks around the world offer mobile internet connections over UMTS (or EDGE or GSM) using a portable USB modem device.<br />
<br />
==Remove the PIN==<br />
First of all use your SIM card in a normal phone and disable the PIN request if present. If the SIM card asks the PIN wvdial will not work.<br />
<br />
Failing that, you can also use ''mmcli'' to unlock the SIM card:<br />
<br />
# mmcli -i ''SIMNUMBER'' --pin=XXXX<br />
<br />
where ''SIMNUMBER'' can be found using {{ic|mmcli -L}} and {{ic|mmcli -m X}}.<br />
<br />
==Device identification==<br />
Examine the output of<br />
$ lsusb<br />
which will show the vendor and product IDs of the device. Note that some devices will show ''two'' different product IDs at different times as explained below.<br />
<br />
==Mode switching==<br />
<br />
Often these devices will have two modes (1) USB flash memory storage (2) USB Modem. The first mode, sometimes known as ZeroCD, is often used to deliver an internet communications program for another operating system and is generally of no interest to Linux users. Additionally some have a slot into which the user can insert an additional flash memory card.<br />
<br />
A useful utility for switching these devices into modem mode is {{Pkg|usb_modeswitch}}, available in the [[official repositories]].<br />
<br />
Udev rules are supplied with the package in {{ic|/lib/udev/rules.d/40-usb_modeswitch.rules}}. This contains entries for many devices, which it will switch to modem mode upon insertion.<br />
<br />
When a device is switched, its product ID may change to a different value. The vendor ID will remain unchanged. This can be seen in the output of {{ic|lsusb}}.<br />
<br />
Some devices are supported in the USB serial kernel module called "option" (after the Option devices, but not limited to just those) and may be used without usb_modeswitch.<br />
<br />
Udev itself included a utility called {{ic|/lib/udev/modem-modeswitch}}. In udev 157 this was renamed to {{ic|/lib/udev/mobile-action-modeswitch}} and morphed into a tool that only switches Mobile Action cables. For other devices use {{ic|usb_modeswitch}}.<br />
<br />
{{Note|You can find an alternative way to do this base on eject command [[ZTE_MF110/MF190#Switch_from_CD_mode_to_modem_mode_on_the_device|here]].}}<br />
<br />
==Connection==<br />
<br />
=== Network Manager ===<br />
<br />
After installing {{Pkg|usbutils}} and {{Pkg|usb_modeswitch}}, you just need to install {{Pkg|modemmanager}} to make the modem work with [[NetworkManager]].<br />
<br />
Make sure both NetworkManager and ModemManager services are running, see [[systemd#Using units]] for details.<br />
<br />
Make sure {{Pkg|mobile-broadband-provider-info}} and {{Pkg|nm-connection-editor}} are installed.<br />
<br />
A system restart might be necessary for ModemManager to detect the USB modem. After you restart the NetworkManager-applet and plug the modem in again NetworkManager should recognize the modem in the menu without further configuration. Setting up the modem in NetworkManager is self-explanatory, you should only need the login-information provided by your network provider.<br />
<br />
Whilst running ModemManager gammu will not work. SMS and Ussd codes can be still used with {{Pkg|modem-manager-gui}}.<br />
<br />
=== pppd ===<br />
[[pppd]] can be used to configure 3g connections. Step by step instruction is available on [[3G and GPRS modems with pppd]]. Optionally, {{Aur|pppconfig}} can be used to simplify the pppd configuration using dialog interface.<br />
<br />
=== wvdial ===<br />
See main article: [[wvdial]]<br />
<br />
=== netctl ===<br />
Netctl can be used to establish a connection using a USB modem. To bring up the modem you can use {{ic|mbim-network}}. First create a profile for mbim-network.<br />
{{hc|/etc/mbim-network.conf|<br />
<nowiki>APN=Broadband</nowiki>}}<br />
Now connect to the network with {{bc|# mbim-network /dev/cdc-wdmX start}}. Then bring up the interface and get an ip address:<br />
{{bc|<br />
# ip link set wwanY up<br />
# dhcpcd wwanY<br />
}}<br />
An example configuration file provided by {{Pkg|netctl}} is located at {{ic|/etc/netctl/examples/mobile_ppp}}. Minimally you will probably have to specify<br />
{{hc|/etc/netctl/mobile_ppp|<br />
<nowiki>Interface=cdc-wdmX<br />
Connection=mobile_ppp<br />
PhoneNumber=XxxxXXXX<br />
AccessPointName=Broadband</nowiki><br />
}}<br />
<br />
See the [[netctl]] article and [https://github.com/joukewitteveen/netctl/blob/master/docs/netctl.profile.5.txt netctl.profile] for more information.<br />
<br />
=== sakis3g ===<br />
There may be the chance that the modem stick is supported by [http://www.sakis3g.com/ sakis3g] which is an all in one command line script and automates all the steps above. Install {{AUR|sakis3g}} from the [[AUR]].<br />
<br />
=== Low connection speed ===<br />
Someone claims that the connection speed under Linux is lower than Windows: [https://bbs.archlinux.org/viewtopic.php?id=111513]<br />
<br />
A short summary for possible solutions which are not fully verified.<br />
In most of conditions, the low speed is caused by bad receiver signals and too many people in cell. But you still could use the following method to try to improve the connection speed.<br />
<br />
====QoS parameter====<br />
The {{ic|AT+CGEQMIN}} and {{ic|AT+CGEQREQ}} commands can be used to set the QoS. It is also possible to decrease and limit the connect speed. <br />
Add the following {{ic|Init}} command in {{ic|/etc/wvdial.conf}}:<br />
Init6 = AT+CGEQMIN=1,4,64,640,64,640<br />
Init7 = AT+CGEQREQ=1,4,64,640,64,640<br />
<br />
====Baud parameter====<br />
Baud parameter in wvdial.conf could be used to increase the connection speed.<br />
<br />
Baud = 460800<br />
<br />
But the official Huawei E261 windows application set the Baud=9600 under Windows Vista.<br />
More verifications are needed to double check this point.<br />
<br />
===Monitor used bandwith===<br />
Frequently a 3G connection obtained via a mobile phone operator comes with restricted bandwidth, so that you are only allowed to use a certain bandwidth per time (e.g. 1GB per month). While it is quite straight-forward to know which type of network applications are pretty bandwidth extensive (e.g. video streaming, gaming, torrent, etc.), it may be difficult to keep an overview about overall consumed bandwidth. <br />
<br />
A number of tools are available to help with that. Two console tools are {{pkg|vnstat}}, which allows to keep track of bandwith over time, and {{pkg|iftop}} to monitor bandwidth of individual sessions. If you are a [[KDE]] user, {{Pkg|knemo}} might help. All are available in the [[official repositories]].<br />
<br />
==Reading SMS==<br />
This was tested on a Huawei EM770W (GTM382E) 3g card integrated into an Acer Aspire AS3810TG laptop.<br />
<br />
{{bc|<br />
$ pacman -S gnokii<br />
$ mkdir -p $XDG_CONFIG_HOME/gnokii<br />
}}<br />
<br />
Usually the configuration directory is {{ic|~/.config/gnokii}}. <br />
<br />
{{bc|<br />
$ cp /etc/gnokiirc ~/.config/gnokii/config<br />
}}<br />
<br />
edit {{ic|~/.config/gnokii/config}} as follows:<br />
<br />
{{bc|1=<br />
port = /dev/ttyUSB0<br />
}}<br />
<br />
You may have to use a different port depending on your configuration, for example {{ic|/dev/ttyUSB1}} or something else:<br />
<br />
{{bc|1=<br />
model = AT<br />
connection = serial<br />
}}<br />
<br />
You need to be part of the {{ic|uucp}} group to use {{ic|/dev/ttyUSB0}}, for example if your user is called "x": <br />
{{bc|<br />
# gpasswd -a x uucp<br />
$ newgrp uucp<br />
}}<br />
The ''newgrp'' command allows you to take advantage of the new group assignment immediately without having to logout/login.<br />
<br />
Then launch gnokii:<br />
{{bc|<br />
$ xgnokii<br />
}}<br />
Click on the "SMS" icon button, a window opens up. Then click: "messages->activate sms reading". Your messages will show up in the window.<br />
===command line script===<br />
A small command line script using gnokii to read SMS on your SIM card (not phone memory) without having to start a GUI:<br />
$ gnokii --getsms SM 0 end 2>&1|grep Text -A1 -B3|grep -v Text<br />
What it does:<br />
{{bc|<nowiki><br />
gnokii # invoke gnokii<br />
--getsms SM 0 end # read SMS from SM-memory location (=SIM card) starting at 0 and reading all occupied memory locations ("end")<br />
2>&1 # connect STDERR to STDOUT to make sure the output from the --getsms command can be piped to grep<br />
|grep Text # pipe output from gnokii to grep, anchoring at output containing "Text"<br />
-A1 -B3 # print one line after the matched pattern and three lines before the matched pattern<br />
|grep -v Text # grep result to another grep to exclude the "Text" line (-v for inverting the pattern)<br />
</nowiki>}}<br />
Granted this does not work very well if your SMS contains the word "Text", but you may adapt the script to your liking.<br />
<br />
== Fix image quality ==<br />
If you are getting low quality images while browsing the web over a mobile broadband connection with the hints "shift+r improves the quality of this image" and "shift+a improves the quality of all images on this page", follow these instructions:<br />
<br />
[[Install]] {{Pkg|tinyproxy}}, available in the [[official repositories]].<br />
<br />
Edit /etc/tinyproxy/tinyproxy.conf and insert the following two lines:<br />
AddHeader "Pragma" "No-Cache"<br />
AddHeader "Cache-Control" "No-Cache"<br />
Start tinyproxy:<br />
systemctl start tinyproxy<br />
Configure your browser to use localhost:8888 as a proxy server and you are all done. This is especially useful if you are using, for example, Google Chrome which, unlike Firefox, does not allow you to modify the Pragma and Cache-Control headers.</div>Alaskanarcherhttps://wiki.archlinux.org/index.php?title=Mobile_broadband_modem&diff=458695Mobile broadband modem2016-12-07T07:13:26Z<p>Alaskanarcher: The netctl article does not cover anything about mobile_ppp connections. I used this method to bring up my ATT AirCard 340U USB modem because I don't normally use NetworkManager.</p>
<hr />
<div>[[Category:Modems]]<br />
[[ja:USB 3G モデム]]<br />
[[ru:USB 3G Modem]]<br />
{{Related articles start}}<br />
{{Related|wvdial}}<br />
{{Related|Direct Modem Connection}}<br />
{{Related|3G and GPRS modems with pppd}}<br />
{{Related|:Category:Modems}}<br />
{{Related articles end}}<br />
<br />
{{Poor writing|does not conform to [[Help:Style]]}}<br />
A number of mobile telephone networks around the world offer mobile internet connections over UMTS (or EDGE or GSM) using a portable USB modem device.<br />
<br />
==Remove the PIN==<br />
First of all use your SIM card in a normal phone and disable the PIN request if present. If the SIM card asks the PIN wvdial will not work.<br />
<br />
Failing that, you can also use ''mmcli'' to unlock the SIM card:<br />
<br />
# mmcli -i ''SIMNUMBER'' --pin=XXXX<br />
<br />
where ''SIMNUMBER'' can be found using {{ic|mmcli -L}} and {{ic|mmcli -m X}}.<br />
<br />
==Device identification==<br />
Examine the output of<br />
$ lsusb<br />
which will show the vendor and product IDs of the device. Note that some devices will show ''two'' different product IDs at different times as explained below.<br />
<br />
==Mode switching==<br />
<br />
Often these devices will have two modes (1) USB flash memory storage (2) USB Modem. The first mode, sometimes known as ZeroCD, is often used to deliver an internet communications program for another operating system and is generally of no interest to Linux users. Additionally some have a slot into which the user can insert an additional flash memory card.<br />
<br />
A useful utility for switching these devices into modem mode is {{Pkg|usb_modeswitch}}, available in the [[official repositories]].<br />
<br />
Udev rules are supplied with the package in {{ic|/lib/udev/rules.d/40-usb_modeswitch.rules}}. This contains entries for many devices, which it will switch to modem mode upon insertion.<br />
<br />
When a device is switched, its product ID may change to a different value. The vendor ID will remain unchanged. This can be seen in the output of {{ic|lsusb}}.<br />
<br />
Some devices are supported in the USB serial kernel module called "option" (after the Option devices, but not limited to just those) and may be used without usb_modeswitch.<br />
<br />
Udev itself included a utility called {{ic|/lib/udev/modem-modeswitch}}. In udev 157 this was renamed to {{ic|/lib/udev/mobile-action-modeswitch}} and morphed into a tool that only switches Mobile Action cables. For other devices use {{ic|usb_modeswitch}}.<br />
<br />
{{Note|You can find an alternative way to do this base on eject command [[ZTE_MF110/MF190#Switch_from_CD_mode_to_modem_mode_on_the_device|here]].}}<br />
<br />
==Connection==<br />
<br />
=== Network Manager ===<br />
<br />
After installing {{Pkg|usbutils}} and {{Pkg|usb_modeswitch}}, you just need to install {{Pkg|modemmanager}} to make the modem work with [[NetworkManager]].<br />
<br />
Make sure both NetworkManager and ModemManager services are running, see [[systemd#Using units]] for details.<br />
<br />
Make sure {{Pkg|mobile-broadband-provider-info}} and {{Pkg|nm-connection-editor}} are installed.<br />
<br />
A system restart might be necessary for ModemManager to detect the USB modem. After you restart the NetworkManager-applet and plug the modem in again NetworkManager should recognize the modem in the menu without further configuration. Setting up the modem in NetworkManager is self-explanatory, you should only need the login-information provided by your network provider.<br />
<br />
Whilst running ModemManager gammu will not work. SMS and Ussd codes can be still used with {{Pkg|modem-manager-gui}}.<br />
<br />
=== pppd ===<br />
[[pppd]] can be used to configure 3g connections. Step by step instruction is available on [[3G and GPRS modems with pppd]]. Optionally, {{Aur|pppconfig}} can be used to simplify the pppd configuration using dialog interface.<br />
<br />
=== wvdial ===<br />
See main article: [[wvdial]]<br />
<br />
=== netctl ===<br />
Netctl can be used to establish a connection using a USB modem. To bring up the modem you can use: <br />
{{bc|# mbim-network /dev/cdc-wdmX start}} <br />
Then bring up the interface and get an ip address:<br />
{{bc|<br />
# ip link set wwanY up<br />
# dhcpcd wwanY<br />
}}<br />
An example configuration file provided by {{Pkg|netctl}} is located at {{ic|/etc/netctl/examples/mobile_ppp}}. Minimally you will probably have to specify<br />
{{hc|/etc/netctl/mobile_ppp|<br />
<nowiki>Interface=cdc-wdmX<br />
Connection=mobile_ppp<br />
PhoneNumber=XxxxXXXX<br />
AccessPointName=Broadband</nowiki><br />
}}<br />
<br />
See the [[netctl]] article and [https://github.com/joukewitteveen/netctl/blob/master/docs/netctl.profile.5.txt netctl.profile] for more information.<br />
<br />
=== sakis3g ===<br />
There may be the chance that the modem stick is supported by [http://www.sakis3g.com/ sakis3g] which is an all in one command line script and automates all the steps above. Install {{AUR|sakis3g}} from the [[AUR]].<br />
<br />
=== Low connection speed ===<br />
Someone claims that the connection speed under Linux is lower than Windows: [https://bbs.archlinux.org/viewtopic.php?id=111513]<br />
<br />
A short summary for possible solutions which are not fully verified.<br />
In most of conditions, the low speed is caused by bad receiver signals and too many people in cell. But you still could use the following method to try to improve the connection speed.<br />
<br />
====QoS parameter====<br />
The {{ic|AT+CGEQMIN}} and {{ic|AT+CGEQREQ}} commands can be used to set the QoS. It is also possible to decrease and limit the connect speed. <br />
Add the following {{ic|Init}} command in {{ic|/etc/wvdial.conf}}:<br />
Init6 = AT+CGEQMIN=1,4,64,640,64,640<br />
Init7 = AT+CGEQREQ=1,4,64,640,64,640<br />
<br />
====Baud parameter====<br />
Baud parameter in wvdial.conf could be used to increase the connection speed.<br />
<br />
Baud = 460800<br />
<br />
But the official Huawei E261 windows application set the Baud=9600 under Windows Vista.<br />
More verifications are needed to double check this point.<br />
<br />
===Monitor used bandwith===<br />
Frequently a 3G connection obtained via a mobile phone operator comes with restricted bandwidth, so that you are only allowed to use a certain bandwidth per time (e.g. 1GB per month). While it is quite straight-forward to know which type of network applications are pretty bandwidth extensive (e.g. video streaming, gaming, torrent, etc.), it may be difficult to keep an overview about overall consumed bandwidth. <br />
<br />
A number of tools are available to help with that. Two console tools are {{pkg|vnstat}}, which allows to keep track of bandwith over time, and {{pkg|iftop}} to monitor bandwidth of individual sessions. If you are a [[KDE]] user, {{Pkg|knemo}} might help. All are available in the [[official repositories]].<br />
<br />
==Reading SMS==<br />
This was tested on a Huawei EM770W (GTM382E) 3g card integrated into an Acer Aspire AS3810TG laptop.<br />
<br />
{{bc|<br />
$ pacman -S gnokii<br />
$ mkdir -p $XDG_CONFIG_HOME/gnokii<br />
}}<br />
<br />
Usually the configuration directory is {{ic|~/.config/gnokii}}. <br />
<br />
{{bc|<br />
$ cp /etc/gnokiirc ~/.config/gnokii/config<br />
}}<br />
<br />
edit {{ic|~/.config/gnokii/config}} as follows:<br />
<br />
{{bc|1=<br />
port = /dev/ttyUSB0<br />
}}<br />
<br />
You may have to use a different port depending on your configuration, for example {{ic|/dev/ttyUSB1}} or something else:<br />
<br />
{{bc|1=<br />
model = AT<br />
connection = serial<br />
}}<br />
<br />
You need to be part of the {{ic|uucp}} group to use {{ic|/dev/ttyUSB0}}, for example if your user is called "x": <br />
{{bc|<br />
# gpasswd -a x uucp<br />
$ newgrp uucp<br />
}}<br />
The ''newgrp'' command allows you to take advantage of the new group assignment immediately without having to logout/login.<br />
<br />
Then launch gnokii:<br />
{{bc|<br />
$ xgnokii<br />
}}<br />
Click on the "SMS" icon button, a window opens up. Then click: "messages->activate sms reading". Your messages will show up in the window.<br />
===command line script===<br />
A small command line script using gnokii to read SMS on your SIM card (not phone memory) without having to start a GUI:<br />
$ gnokii --getsms SM 0 end 2>&1|grep Text -A1 -B3|grep -v Text<br />
What it does:<br />
{{bc|<nowiki><br />
gnokii # invoke gnokii<br />
--getsms SM 0 end # read SMS from SM-memory location (=SIM card) starting at 0 and reading all occupied memory locations ("end")<br />
2>&1 # connect STDERR to STDOUT to make sure the output from the --getsms command can be piped to grep<br />
|grep Text # pipe output from gnokii to grep, anchoring at output containing "Text"<br />
-A1 -B3 # print one line after the matched pattern and three lines before the matched pattern<br />
|grep -v Text # grep result to another grep to exclude the "Text" line (-v for inverting the pattern)<br />
</nowiki>}}<br />
Granted this does not work very well if your SMS contains the word "Text", but you may adapt the script to your liking.<br />
<br />
== Fix image quality ==<br />
If you are getting low quality images while browsing the web over a mobile broadband connection with the hints "shift+r improves the quality of this image" and "shift+a improves the quality of all images on this page", follow these instructions:<br />
<br />
[[Install]] {{Pkg|tinyproxy}}, available in the [[official repositories]].<br />
<br />
Edit /etc/tinyproxy/tinyproxy.conf and insert the following two lines:<br />
AddHeader "Pragma" "No-Cache"<br />
AddHeader "Cache-Control" "No-Cache"<br />
Start tinyproxy:<br />
systemctl start tinyproxy<br />
Configure your browser to use localhost:8888 as a proxy server and you are all done. This is especially useful if you are using, for example, Google Chrome which, unlike Firefox, does not allow you to modify the Pragma and Cache-Control headers.</div>Alaskanarcher