Difference between revisions of "ISCSI Boot"

From ArchWiki
Jump to: navigation, search
(Device not found)
m (add ja link)
 
(18 intermediate revisions by 9 users not shown)
Line 1: Line 1:
 +
{{DISPLAYTITLE:iSCSI Boot}}
 
[[Category:Getting and installing Arch]]
 
[[Category:Getting and installing Arch]]
 
[[Category:Networking]]
 
[[Category:Networking]]
 
[[Category:Storage]]
 
[[Category:Storage]]
{{Expansion}}
+
[[ja:ISCSI ブート]]
{{Box YELLOW||The actual boot will use an extra drive / usb drive or pxe boot?}}
+
{{Related articles start}}
 +
{{Related|iSCSI Target}}
 +
{{Related|iSCSI Initiator}}
 +
{{Related articles end}}
  
{{Article summary start}}
+
You can install Arch on an [[iSCSI Target]]. This allows you to boot from an iscsi target using a diskless machine. No physical disk is required unless you need an ipxe boot USB (because your NIC isn't iBFT capable or you don't want to setup a TFTP server).
{{Article summary text|How to install Arch on an iSCSI target.}}
+
{{Article summary heading|Related}}
+
{{Article summary wiki|iSCSI Target}}
+
{{Article summary wiki|iSCSI Initiator}}
+
{{Article summary end}}
+
 
+
You can install Arch on an [[iSCSI Target]].
+
This howto will guide you through the process.
+
  
 
== Server / Target Setup ==
 
== Server / Target Setup ==
Line 22: Line 18:
 
== Client / Initiator Setup ==
 
== Client / Initiator Setup ==
  
=== Install over iSCSI ===
+
=== Overview ===
  
Download Arch Linux ISO image [https://www.archlinux.org/download/] and boot Arch Linux using the ISO image.  After Arch Linux is booted, either use net as the install source or manually ifconfig and dhcp.
+
1. install {{Pkg|open-iscsi}} package in installer system
  
Now before you continue to "Prepare Hard Drive(s)" install open-iscsi and connect to target.
+
2. connect to iSCSI target and create partitions on logical drive of target.
  
Be sure to your servers IP address, Name, etc.
+
3. install Arch Linux system in usual way
  
<pre>
+
4. install {{Pkg|open-iscsi}} package in installed system
wget https://aur.archlinux.org/packages/op/open-iscsi/open-iscsi.tar.gz
+
  
tar xzvf open-iscsi.tar.gz
+
5. create initial RAM disk image containing open-iscsi modules.
  
cd open-iscsi
+
{{Note|In addition to the above, you have to prepare sanboot-able infrastructure which is necessary to load boot loader (GRUB, etc.) from remote disc. Some network interface cards support sanboot. If you don't have such cards, you can use [http://ipxe.org/ iPXE], [http://etherboot.org/wiki/start gPXE], and so on.}}
  
pacman -Sy
+
=== Install over iSCSI ===
  
pacman -S patch make gcc
+
Download Arch Linux ISO image [https://www.archlinux.org/download/] and boot Arch Linux using the ISO image.  After Arch Linux is booted, either use net as the install source or manually ifconfig and dhcp.
  
makepkg --asroot
+
1. Unfortunately ISO install image does not include modules for iSCSI, you have to install and setup them at first.
  
pacman -U open-iscsi...pkg.tar.xz
+
Before you continue to "Partition the disks", [[install]] the {{Pkg|open-iscsi}} package from the [[official repositories]] and connect to target.
  
modprobe iscsi_tcp
+
In the following, server's(target's) IP address is 192.168.1.100, client's(initiator's) IP address is 192.168.1.101, iSCSI initiator name is "iSCSI.Initiator.Name" and target name is "iSCSI.Target.Name". You should, of course, be sure to your network configration and so on.
  
iscsistart -i iqn.2010-06.ClientName:desc -t iqn.2010-06.ServerName:desc -g 1 -a 192.168.1.100
+
<pre>
 +
pacman -Sy
 +
pacman -S open-iscsi
 +
modprobe iscsi_tcp
 
</pre>
 
</pre>
  
During the above procedures, if "pacman -Sy" asks you to upgrade pacman, answer "no" otherwise pacman may fail to install patch, make, and gcc, which are essential to the compilation of the open-iscsi source.
+
2. Get a list of targets
  
(Optional) If you want to make sure that your iSCSI target is up and running, you may start "iscsid" and check whether the iSCSI target is available.
+
iscsiadm -m discovery -t st -p 192.168.1.100
  
<pre>
+
Sample output (your output may differ depending on the portal ip and target name)
# iscsid
+
 
# iscsiadm -m discovery -t sendtargets -p <server-IP>
+
192.168.1.100:3260,-1 iqn.2011-03.example.org.istgt:arch
</pre>
+
 
 +
Connect to the target
 +
 
 +
iscsiadm -m node -T iqn.2011-03.example.org.istgt:arch -p 192.168.1.100 -l
  
Continue to prepare the hard drive, using the iSCSI target drive.  It is suggested that you manually create a partition that uses the entire iSCSI target drive.
+
Now your local host connects to the drive of target host (see dmesg output).
  
{{Note|When you "Manually Configure block devices, filesystems and mountpoints" make sure that you use '''UUID'''. This way there won't be any issues booting if the number of devices changes (/dev/sda /dev/sdb ...) by adding or removing hard drives, usb thumb drives, etc... or booting it on different machines.}}
 
 
{{Note|It is recommended that you '''NOT''' include swap on the iSCSI drive when creating the partitions, you can just '''ignore''' the warning.}}
 
{{Note|It is recommended that you '''NOT''' include swap on the iSCSI drive when creating the partitions, you can just '''ignore''' the warning.}}
  
Before you "Configure System" you must install open-iscsi to the "future" root filesystem, and download and install the hook for booting from iSCSI.  This should be done after all selected packages are installed.  
+
3. You can create a partition table and partitions in the same way as a local drive. And continue to install Arch Linux in the usual way.
  
Create these hook files, replacing the IP, etc. You may create these hook files after open-iscsi is installed to the "future" filesystem.
+
4. [[Install]] the {{Pkg|open-iscsi}} package in the "future" root file system.
  
1. /mnt/usr/lib/initcpio/install/iscsi: It should look like the following.
+
5. Before doing [[mkinitcpio]] in the "future" root file system, you have to prepare the following two files.
  
<pre>
+
i) {{ic|/mnt/usr/lib/initcpio/install/iscsi}}
# vim: set ft=sh:
+
  
 +
{{bc|1=
 
build ()
 
build ()
 
{
 
{
 
     local mod
 
     local mod
     for mod in iscsi_tcp libiscsi libiscsi_tcp scsi_transport_iscsi crc32c; do
+
     for mod in iscsi_ibft iscsi_tcp libiscsi libiscsi_tcp scsi_transport_iscsi crc32c; do
 
         add_module "$mod"
 
         add_module "$mod"
 
     done
 
     done
  
 
     add_checked_modules "/drivers/net"
 
     add_checked_modules "/drivers/net"
     add_binary "/sbin/iscsistart"
+
     add_binary "/usr/bin/iscsistart"
 
     add_runscript
 
     add_runscript
 
}
 
}
Line 91: Line 91:
 
HELPEOF
 
HELPEOF
 
}
 
}
</pre>
+
}}
  
2. /mnt/usr/lib/initcpio/hooks/iscsi:  It should look like the following.
+
==== Using an iBFT Compatible Rom ====
  
<pre>
+
If using a iBFT compatible NIC or boot device (such as ipxe) you can use auto configuration to set the network configuration and iscsi target.
# vim: set ft=sh:
+
  
 +
ii) {{ic|/mnt/usr/lib/initcpio/hooks/iscsi}}
 +
 +
{{bc|1=
 +
run_hook ()
 +
{
 +
    modprobe iscsi_tcp
 +
    modprobe iscsi_ibft
 +
 +
    echo "Network configuration based on iBFT"
 +
    iscsistart -N || echo "Unable to configure network"
 +
 +
    echo "iSCSI auto connect based on iBFT"
 +
    iscsistart -b || echo "Unable to auto connect"
 +
}
 +
}}
 +
 +
==== Manually Setting the iSCSI Target ====
 +
 +
If you are not using an iBFT compatible boot rom you must explicitly setup the network and the iscsi target manually.
 +
 +
ii) {{ic|/mnt/usr/lib/initcpio/hooks/iscsi}}
 +
 +
{{bc|1=
 
run_hook ()
 
run_hook ()
 
{
 
{
 
     modprobe iscsi_tcp
 
     modprobe iscsi_tcp
 
     ifconfig eth0 192.168.1.101 netmask 255.255.255.0 broadcast 192.168.1.255
 
     ifconfig eth0 192.168.1.101 netmask 255.255.255.0 broadcast 192.168.1.255
     sleep 10
+
     sleep 2
     iscsistart -i iqn.2010-06.ClientName:desc -t iqn.2010-06.ServerName:desc -g 1 -a 192.168.1.100
+
     iscsistart -i iSCSI.Initiator.Name -t iSCSI.Target.Name -g 1 -a 192.168.1.100
 
}
 
}
</pre>
+
}}
  
If you want to use dhcp for the above script, you may try to replace the "ifconfig" line with "dhcpcd eth0", but make sure that dhcpcd is installed.
+
==== Using DHCP ====
  
Now, you can install open-iscsi into the "future" root filesystem.
+
If you want to use dhcp for the above script, you may use the following hook, but make sure that {{Pkg|dhcpcd}} is installed and is added to the BINARY line in [[mkinitcpio|/etc/mkinitcpio.conf]]
  
<pre>
+
ii) {{ic|/mnt/usr/lib/initcpio/hooks/iscsi}}
mv open-iscsi...pkg.tar.xz /mnt/root
+
  
chroot /mnt/ /bin/bash
+
{{bc|1=
 +
run_hook ()
 +
{
 +
    modprobe iscsi_tcp
 +
    mkdir -p /var/lib/dhcpcd
 +
    dhcpcd eth0
 +
    sleep 2
 +
    iscsistart -i iSCSI.Initiator.Name -t iSCSI.Target.Name -g 1 -a 192.168.1.100
 +
}
 +
}}
  
cd /root/
+
Add "iscsi" to the HOOKS line in [[mkinitcpio|/etc/mkinitcpio.conf]].
  
pacman -U open-iscsi...pkg.tar.xz
+
Then run "mkinitcpio -p linux" and new {{ic|/boot/initramfs-linux.img}} and {{ic|/boot/initramfs-linux-fallback.img}} will be generated.
</pre>
+
  
Now you can create the above hook files by pasting the above scripts.
+
{{Note|If you plan on booting this installation of Arch on machines with nic cards that require different modules, remove "autodetect" from HOOKS}}
 +
{{Note|Rebuilding the initial ramdisk will take some time if autodetect is removed from HOOKS}}
  
<pre>
+
Now your new system can mount the file systems from iSCSI target drive after reboot.
nano /lib/initcpio/install/iscsi
+
  
nano /lib/initcpio/hooks/iscsi
+
== Configuring Open iSCSI initiator on the diskless system ==
  
exit
+
The initiator on the diskless system must be configured to make it resistant to network problems or even the restart of the target system, the open-iscsi [https://github.com/open-iscsi/open-iscsi/blob/master/README README] describes optimal iSCSI settings for iSCSI root
</pre>
+
  
Now you are ready to "Configure System."  Most of the configuration files can remain unchanged.
+
When accessing the root partition directly through a iSCSI disk, the iSCSI timers should be set so that iSCSI layer has several chances to try to re-establish a session and so that commands are not quickly requeued to the SCSI layer. Basically you want the opposite of when using dm-multipath.
  
Add "iscsi" to the HOOKS in /etc/mkinitcpio.conf.  The "mkinitcpio" will generate a new kernel image after you are done with "Configure System."
+
For this setup, you can turn off iSCSI pings by setting:
  
{{Note|If you plan on booting this installation of Arch on machines with nic cards that require different modules, remove "autodetect" from HOOKS}}
+
node.conn[0].timeo.noop_out_interval = 0
{{Note|Rebuilding the initial ramdisk will take some time if autodetect is removed from HOOKS}}
+
node.conn[0].timeo.noop_out_timeout = 0
  
When configuring grub find the lines for Arch that are
+
And you can turn the replacement_timer to a very long value:
  
<pre>root (hdX,0)</pre>
+
node.session.timeo.replacement_timeout = 86400
and change to
+
 
<pre>root (hd0,0)</pre>
+
If a network problem is detected by the initiator, the running commands are failed immediately. There is one exception to this and that is when the SCSI layer's error handler is running. To check if the SCSI error handler is running iscsiadm can be run as:
 +
 
 +
iscsiadm -m session -P 3
 +
 
 +
You will then see:
 +
 
 +
Host Number: X State: Recovery
 +
When the SCSI EH is running, commands will not be failed until node.session.timeo.replacement_timeout seconds.
 +
 
 +
To modify the timer that starts the SCSI EH, you can either write directly to the device's sysfs file:
 +
 
 +
echo X > /sys/block/sdX/device/timeout
 +
 
 +
where X is in seconds or on most distros you can modify the udev rule.
 +
 
 +
To modify the timeout using a udev rule create /etc/udev/rules.d/50-iscsi.rules, and add the following lines:
 +
 
 +
  ACTION=="add", SUBSYSTEM=="scsi" , ATTR{type}=="0|7|14", \
 +
        RUN+="/bin/sh -c 'echo 90 > /sys$$DEVPATH/timeout'"
 +
 
 +
Change the echo 90 part of the line to the value that you want.
 +
 
 +
The default timeout for normal File System commands is 30 seconds when udev is not being used.
 +
 
 +
Since low level I/O commands will go through the IO scheduler on the target it is recommended for performance to use elevator=noop on the diskless system, see [[Improving performance]] to set the elevator.
  
 
== Troubleshooting ==
 
== Troubleshooting ==
  
 
=== Device not found ===
 
=== Device not found ===
If you are having problems with detecting your eth0 interface, simply install {{Pkg|mkinitcpio-nfs-utils}} package from the [[Official_Repositories|official repositories]] and {{Pkg|dhcpcd}} on your iSCSI drive and add '''net''' HOOK to ''/etc/mkinitcpio.conf''.
 
 
Also add '''ip=::::::dhcp''' to your kernel parameters and now you can use ''dhcpcd eth0'' as described before.
 
  
For more informations, please refer to [https://wiki.archlinux.org/index.php/Mkinitcpio#Using_net].
+
If you are having problems with detecting your eth0 interface you may need to explicitly install the kernel module for your NIC in the MODULES line in [[mkinitcpio|/etc/mkinitcpio.conf]].

Latest revision as of 11:59, 24 November 2016

Related articles

You can install Arch on an iSCSI Target. This allows you to boot from an iscsi target using a diskless machine. No physical disk is required unless you need an ipxe boot USB (because your NIC isn't iBFT capable or you don't want to setup a TFTP server).

Server / Target Setup

You can set up an iSCSI target with any hosting server OS. Follow the procedure outlined in iSCSI Target if you use Arch Linux as the hosting server OS.

Client / Initiator Setup

Overview

1. install open-iscsi package in installer system

2. connect to iSCSI target and create partitions on logical drive of target.

3. install Arch Linux system in usual way

4. install open-iscsi package in installed system

5. create initial RAM disk image containing open-iscsi modules.

Note: In addition to the above, you have to prepare sanboot-able infrastructure which is necessary to load boot loader (GRUB, etc.) from remote disc. Some network interface cards support sanboot. If you don't have such cards, you can use iPXE, gPXE, and so on.

Install over iSCSI

Download Arch Linux ISO image [1] and boot Arch Linux using the ISO image. After Arch Linux is booted, either use net as the install source or manually ifconfig and dhcp.

1. Unfortunately ISO install image does not include modules for iSCSI, you have to install and setup them at first.

Before you continue to "Partition the disks", install the open-iscsi package from the official repositories and connect to target.

In the following, server's(target's) IP address is 192.168.1.100, client's(initiator's) IP address is 192.168.1.101, iSCSI initiator name is "iSCSI.Initiator.Name" and target name is "iSCSI.Target.Name". You should, of course, be sure to your network configration and so on.

pacman -Sy
pacman -S open-iscsi
modprobe iscsi_tcp

2. Get a list of targets

iscsiadm -m discovery -t st -p 192.168.1.100

Sample output (your output may differ depending on the portal ip and target name)

192.168.1.100:3260,-1 iqn.2011-03.example.org.istgt:arch

Connect to the target

iscsiadm -m node -T iqn.2011-03.example.org.istgt:arch -p 192.168.1.100 -l

Now your local host connects to the drive of target host (see dmesg output).

Note: It is recommended that you NOT include swap on the iSCSI drive when creating the partitions, you can just ignore the warning.

3. You can create a partition table and partitions in the same way as a local drive. And continue to install Arch Linux in the usual way.

4. Install the open-iscsi package in the "future" root file system.

5. Before doing mkinitcpio in the "future" root file system, you have to prepare the following two files.

i) /mnt/usr/lib/initcpio/install/iscsi

build ()
{
    local mod
    for mod in iscsi_ibft iscsi_tcp libiscsi libiscsi_tcp scsi_transport_iscsi crc32c; do
        add_module "$mod"
    done

    add_checked_modules "/drivers/net"
    add_binary "/usr/bin/iscsistart"
    add_runscript
}

help ()
{
cat <<HELPEOF
  This hook allows you to boot from an iSCSI target.
HELPEOF
}

Using an iBFT Compatible Rom

If using a iBFT compatible NIC or boot device (such as ipxe) you can use auto configuration to set the network configuration and iscsi target.

ii) /mnt/usr/lib/initcpio/hooks/iscsi


Manually Setting the iSCSI Target

If you are not using an iBFT compatible boot rom you must explicitly setup the network and the iscsi target manually.

ii) /mnt/usr/lib/initcpio/hooks/iscsi

run_hook ()
{
    modprobe iscsi_tcp
    ifconfig eth0 192.168.1.101 netmask 255.255.255.0 broadcast 192.168.1.255
    sleep 2
    iscsistart -i iSCSI.Initiator.Name -t iSCSI.Target.Name -g 1 -a 192.168.1.100
}

Using DHCP

If you want to use dhcp for the above script, you may use the following hook, but make sure that dhcpcd is installed and is added to the BINARY line in /etc/mkinitcpio.conf

ii) /mnt/usr/lib/initcpio/hooks/iscsi

run_hook ()
{
    modprobe iscsi_tcp
    mkdir -p /var/lib/dhcpcd
    dhcpcd eth0
    sleep 2
    iscsistart -i iSCSI.Initiator.Name -t iSCSI.Target.Name -g 1 -a 192.168.1.100
}

Add "iscsi" to the HOOKS line in /etc/mkinitcpio.conf.

Then run "mkinitcpio -p linux" and new /boot/initramfs-linux.img and /boot/initramfs-linux-fallback.img will be generated.

Note: If you plan on booting this installation of Arch on machines with nic cards that require different modules, remove "autodetect" from HOOKS
Note: Rebuilding the initial ramdisk will take some time if autodetect is removed from HOOKS

Now your new system can mount the file systems from iSCSI target drive after reboot.

Configuring Open iSCSI initiator on the diskless system

The initiator on the diskless system must be configured to make it resistant to network problems or even the restart of the target system, the open-iscsi README describes optimal iSCSI settings for iSCSI root

When accessing the root partition directly through a iSCSI disk, the iSCSI timers should be set so that iSCSI layer has several chances to try to re-establish a session and so that commands are not quickly requeued to the SCSI layer. Basically you want the opposite of when using dm-multipath.

For this setup, you can turn off iSCSI pings by setting:

node.conn[0].timeo.noop_out_interval = 0
node.conn[0].timeo.noop_out_timeout = 0

And you can turn the replacement_timer to a very long value:

node.session.timeo.replacement_timeout = 86400

If a network problem is detected by the initiator, the running commands are failed immediately. There is one exception to this and that is when the SCSI layer's error handler is running. To check if the SCSI error handler is running iscsiadm can be run as:

iscsiadm -m session -P 3

You will then see:

Host Number: X State: Recovery When the SCSI EH is running, commands will not be failed until node.session.timeo.replacement_timeout seconds.

To modify the timer that starts the SCSI EH, you can either write directly to the device's sysfs file:

echo X > /sys/block/sdX/device/timeout

where X is in seconds or on most distros you can modify the udev rule.

To modify the timeout using a udev rule create /etc/udev/rules.d/50-iscsi.rules, and add the following lines:

  ACTION=="add", SUBSYSTEM=="scsi" , ATTR{type}=="0|7|14", \
       RUN+="/bin/sh -c 'echo 90 > /sys$$DEVPATH/timeout'"

Change the echo 90 part of the line to the value that you want.

The default timeout for normal File System commands is 30 seconds when udev is not being used.

Since low level I/O commands will go through the IO scheduler on the target it is recommended for performance to use elevator=noop on the diskless system, see Improving performance to set the elevator.

Troubleshooting

Device not found

If you are having problems with detecting your eth0 interface you may need to explicitly install the kernel module for your NIC in the MODULES line in /etc/mkinitcpio.conf.