Arch Linux AMIs for Amazon Web Services

From ArchWiki

This article or section needs language, wiki syntax or style improvements. See Help:Style for reference.

Reason: Don't write in first person, this is not a personal blog. (Discuss in Talk:Arch Linux AMIs for Amazon Web Services)

Public community Arch AMIs

Note: Arch Linux currently does not offer official Amazon Machine Images. The ones listed here are created by the community.

AMIs

Arch Linux AMIs are listed here: http://arch-ami-list.drzee.net/

AMIs are build twice a month (the 1st and the 15th - 2:00am UTC) and are available for all regions that to do not explicitly require 'Opt-in' - see Region List. If an AMI is needed in a region where its currently not available, an AMI can be copied to that region.

The AMIs are EBS HVM AMIs and are available with two different kernels:

  • std - using the Standard Arch Linux Kernel from the default Arch repositories configured with the necessary modules for EC2 usage. The scheduler and I/O is not optimized for Cloud usage reducing performance.
  • ec2 - using an EC2 optimized kernel of the Standard Arch Linux Kernel created by UplinkLabs and hosted in a dedicated repository: see https://git.uplinklabs.net/steven/ec2-packages.git

Both kernels have been tested on many different EC2 instance types (t2, t3, t3a, m/r/c5, m/r/c6 and advanced hardware with GPUs) and are booting fine.

Note: As of 2023-06-12, there is confirmation that EC2 optimized kernel from UplinkLabs hangs upon boot when running on an instance family using the XEN hypervisor (e.g. t2-micro). The XEN kernel modules are not included in the latest kernel which is why the boot fails. Please use only instance families using the new Nitro Hypervisor - see: Instances build on Nitro. These will still work with the EC2 optimized kernel.
Note: ec2 repository ships other packages alongside linux-ec2 kernel. On default, ec2 is set as the top most repository in pacman.conf. As a result, packages from ec2 may have a higher priority than those from the Official repositories. This can lead to some inconsistent behaviors depending on actual versions of packages.

AMIs with LTS kernels are not build.

REST API to List AMIs

An REST API is available to get a JSON of available AMIs:

Note: The API endpoint now uses a custom domain name which should remain static.

First Run

Note: The mirror list baked into the AMI was generated at time of image build and is using servers in Germany.

After booting the AMI it is recommended/required to execute the following steps to initialize pacman and select fast local repositories:

# pacman-key --init
# pacman-key --populate
# reflector --country "ISO 3166-1 Alpha-2 Country Code" --protocol https,http --score 20 --sort rate --save /etc/pacman.d/mirrorlist
# pacman -Syu

The Reflector package is preinstalled in the AMIs.

It is recommended to set a proper configuration for reflector in /etc/xdg/reflector/reflector.conf and enable the timer services to regularly refresh the mirror list. For details see the Reflector package documentation.

Alternative provide your own mirrorlist and do not use the reflector package.

Build process

Note: This section provides a short description on how the AMIs are build.

The entire build process runs on AWS and is fully automated.

Overall the automated build procedure is managed by a AWS Step Function that is executed at regular intervals using a Amazon EventBridge timed event.

The step function will initiate the build process and uses a combination of native calls and AWS Lambda functions for more complex elements.

A new set of AMIs is build, by booting an EC2 instance with the previous AMI and using it as the work or build machine. The build machine is bootstrapped with a special build script that essentially uses pacstrap and some additional steps to build the image, the basics are outlined below in the next section.

Following the build of the AMI the new AMI is test booted on an EC2 instance to verify that it start up correctly. If successful the AMI is distributed to the regions and registered in a DynamoDB database. The database can be queried using the API REST endpoint. Old AMIs are deleted from the regions and the DynamoDB database.

Credits

Thanks to Steven from UplinkLabs for helping to understand the build process and test the initial quality of the images. Also thanks to Mathcom for an excellent shells script to help me get started (unfortunately the link to that has been removed), which accelerate putting the basic build process together.

You may send comments and suggestions (without any promise that they will be looked at) to: arch-ami 'at' drzee.net

Wishlist

  • Create official EC2/Cloud optimized kernels in the Standard Arch Linux repositories.
  • Add the AWS CLI v2 to the Standard Arch Linux repositories (v1 is available, but may be discontinued in the future) - Thanks to yan12125 for adding aws-cli-v2

Building Arch AMIs

This article or section needs language, wiki syntax or style improvements. See Help:Style for reference.

Reason: Long code blocks are not the appropriate format for a wiki, this is not a code forge. The script needs to be maintained elsewhere or avoided altogether. (Discuss in Talk:Arch Linux AMIs for Amazon Web Services)
Note: This was slapped together from the raw bash shell used in the automated process. It may require some tweaking to work fully. Feel free improve the formatting and/or readability by editing the page - thanks for helping out! If there is something that could be improve in the created AMI please let me know!

You can also build your own Arch Linux AMI. The basic steps are (raw bash shell command):

#Make an image file
mnt=<mountpoint>
dd if=/dev/zero of=<file> bs=1 count=0 seek=8G

#Setup the loop device for the image file
image=`losetup -fP --show <file>`
 
#We setup Legoacy Boot via GRUB - not UEFI (most AWS hardware can do both and the older only Legacy)
parted -s $image -- mklabel msdos
parted -s $image -- mkpart primary 0% 100%
parted -s $image -- toggle 1 boot

partition=`lsblk -po kname -n $image | grep -v "^$image$"`
 
mkfs.ext4 $partition
 
mount $partition $mnt

#Start the build process
#We remove any ignore package settings in pacman.conf on the build system or this will negative impact pacstrap operation
sed -i '/^IgnorePkg/ s/./#&/' /etc/pacman.conf
pacstrap $mnt base grub mkinitcpio
genfstab -U -p $mnt >> $mnt/etc/fstab

#Setup the mirror list - copy in the mirror list from the build system
#We make a copy of the original mirror list in the image and will restore it back later
cp $mnt/etc/pacman.d/mirrorlist $mnt/etc/pacman.d/mirrorlist.bk01
/bin/cp /etc/pacman.d/mirrorlist $mnt/etc/pacman.d/mirrorlist
 
sed -ri '/^\[core\]/aSigLevel = PackageRequired' $mnt/etc/pacman.conf
sed -ri '/^\[extra\]/aSigLevel = PackageRequired' $mnt/etc/pacman.conf
sed -ri '/^\[community\]/aSigLevel = PackageRequired' $mnt/etc/pacman.conf
 
#Enable ParallelDownloads in pacman to make it faster
sed -i '/ParallelDownloads/s/^#//g' $mnt/etc/pacman.conf

arch-chroot $mnt /bin/bash -c "pacman-key --init"
arch-chroot $mnt /bin/bash -c "pacman-key --populate archlinux"
arch-chroot $mnt /bin/bash -c "pacman -Syy"
arch-chroot $mnt /bin/bash -c "pacman --needed --noconfirm -S archlinux-keyring"
 
#Setup with en-US and en-GB locale
sed -i 's/^#en_GB/en_GB/' $mnt/etc/locale.gen 
sed -i 's/^#en_US/en_US/' $mnt/etc/locale.gen
 
#Generate the locales
arch-chroot $mnt /bin/bash -c "locale-gen"
 
#Set more locales for consoles
cat > $mnt/etc/vconsole.conf << "EOF"
KEYMAP=us
FONT=LatArCyrHeb-14
EOF

cat > $mnt/etc/locale.conf << "EOF"
LANG=en_US.utf8
EOF
 
#Set time to UTC
ln -sf ../usr/share/zoneinfo/UTC $mnt/etc/localtime
 
#List of default minimal packages we want in the AMI
arch-chroot $mnt /bin/bash -c "pacman --needed --noconfirm -S systemd-sysvcompat dosfstools e2fsprogs exfatprogs ntfs-3g xfsprogs man which lsof reflector rsync vi python3 audit irqbalance openssh haveged cloud-init cloud-utils aws-cli-v2 jq"
 
#Normal Kernel
arch-chroot $mnt /bin/bash -c "pacman --needed --noconfirm -S linux linux-headers" 

#Audit Rules
mkdir $mnt/etc/audit/rules.d

# Set audit rules
cat > $mnt/etc/audit/rules.d/audit.rules <<"EOF"
# From:
# https://security.blogoverflow.com/2013/01/a-brief-introduction-to-auditd/
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.
# First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 1024
-a always,exit -S adjtimex -S settimeofday -S stime -k time-change
-a always,exit -S clock_settime -k time-change
-a always,exit -S sethostname -S setdomainname -k system-locale
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k identity
-w /var/run/utmp -p wa -k session
-w /var/log/wtmp -p wa -k session
-w /var/log/btmp -p wa -k session
-w /etc/selinux/ -p wa -k MAC-policy

# Disable adding any additional rules. 
# Note that adding new rules will require a reboot
-e 2
EOF

chmod -R o-rwx $mnt/etc/audit
 
#The cloud-init locale module creates an invalid format in /etc/locale.gen so we disable it
sed -ri '/- locale/s/^/#/' $mnt/etc/cloud/cloud.cfg

# enable syslog logger for cloud-init
sed -ri 's/# (.*log_syslog.*)$/ \1/g' $mnt/etc/cloud/cloud.cfg.d/05_logging.cfg
sed -ri 's/( - \[ \*log_base, \*log_file)/#\1/g' $mnt/etc/cloud/cloud.cfg.d/05_logging.cfg

# Disable other unnecessary or broken modules
sed -ri '/ ntp/d' $mnt/etc/cloud/cloud.cfg #<-Not supported on Arch
 
# Some Kernel Module loading config
echo "blacklist floppy" > $mnt/etc/modprobe.d/blacklist-floppy.conf

# Include modules that may be needed in a variety
# of hypervisors, depending on where the guest is run.
MODULES=""

# Support power-off requests. - ipmi is Intelligent Platform Management Interface, used to manage a machine outside the OS.
MODULES+="button ipmi-msghandler ipmi-poweroff"

# Support nvme, Non-Volatile Memory Express, a controller spec for SSDs
MODULES+=" nvme"

# Support the KVM, kernel-based virtual machine
MODULES+=" virtio virtio-blk virtio-net virtio-pci virtio-ring"

# Support the Xen virtual machine
MODULES+=" xen-blkfront xen-netfront xen-pcifront xen-privcmd"

# Support SR-IOV, single root i/o virtualization
MODULES+=" ixgbevf"

# Support for AWS EC2 ENA, Elastic Network Adapter
MODULES+=" ena"

sed -ri "s/^MODULES=.*/MODULES=($MODULES)/g" $mnt/etc/mkinitcpio.conf
sed -ri "s/^FILES=.*/FILES=(\/etc\/modprobe.d\/blacklist-floppy.conf)/g" $mnt/etc/mkinitcpio.conf

img=/boot/vmlinuz-linux
init_img=/boot/initramfs-linux.img
preset_name=linux.preset

rm $mnt/etc/mkinitcpio.d/*.preset

# Disable module auto-detection and setup the presets
cat > $mnt/etc/mkinitcpio.d/$preset_name <<EOF
# mkinitcpio preset file for linux
ALL_config="/etc/mkinitcpio.conf"
ALL_kver=$img
PRESETS=('default')
default_image=$init_img
# Turn off autodetect:
default_options="-S autodetect"
EOF

#Boot target
ln -sf ../../../../usr/lib/systemd/system/multi-user.target $mnt/etc/systemd/system/default.target
 
# Finally, run mkinitcpio.
# Reads /etc/mkinitcpio.conf, /etc/mkinitcpio.d/*.preset.
# As specified in .conf, writes /boot/initramfs-linux.img
arch-chroot $mnt /bin/bash -c "mkinitcpio -P"
 
#Setup GRUB boot load
arch-chroot $mnt /bin/bash -c "grub-install --target=i386-pc --recheck ${image}"

sed -ri 's/GRUB_TIMEOUT=5/GRUB_TIMEOUT=1/' $mnt/etc/default/grub
sed -ri 's/^#GRUB_TERMINAL_OUTPUT/GRUB_TERMINAL_OUTPUT/' $mnt/etc/default/grub
sed -ri "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"console=ttyS0 earlyprint=serial,ttyS0,keep loglevel=7 nomodeset\"/g" $mnt/etc/default/grub
sed -ri '/^GRUB_TIMEOUT=/a GRUB_DISABLE_SUBMENU=y' $mnt/etc/default/grub

arch-chroot $mnt /bin/bash -c "grub-mkconfig -o /boot/grub/grub.cfg"
 
# Set SSH port
sed -ri '1i Port 22' $mnt/etc/ssh/sshd_config
# Disable password authentication. It doesn't make sense in a cloud setting.
sed -ri 's/^#PasswordAuthentication yes/PasswordAuthentication no/' $mnt/etc/ssh/sshd_config
 
# Configure the initial network
cat > $mnt/etc/systemd/network/20.ethernet << "EOF"
[Match]
Name = en* eth*
[Network]
DHCP = yes
[DHCP]
UseMTU = yes
UseDNS = yes
UseDomains = yes
EOF

#Setup resolv.conf correct to work with GPG and possibly other tools relying on resolve.conf
rm $mnt/etc/resolv.conf
ln -s /run/systemd/resolve/stub-resolv.conf $mnt/etc/resolv.conf

#Enable services
arch-chroot $mnt /bin/bash -c "systemctl enable systemd-timesyncd.service"
arch-chroot $mnt /bin/bash -c "systemctl enable nscd.service"
arch-chroot $mnt /bin/bash -c "systemctl enable auditd.service"
arch-chroot $mnt /bin/bash -c "systemctl enable haveged.service"
arch-chroot $mnt /bin/bash -c "systemctl enable irqbalance.service"
arch-chroot $mnt /bin/bash -c "systemctl enable cloud-init.service"
arch-chroot $mnt /bin/bash -c "systemctl enable cloud-config.service"
arch-chroot $mnt /bin/bash -c "systemctl enable cloud-final.service"
arch-chroot $mnt /bin/bash -c "systemctl enable systemd-networkd"
arch-chroot $mnt /bin/bash -c "systemctl enable systemd-resolved"
arch-chroot $mnt /bin/bash -c "systemctl enable sshd.service"

# Do some cleanup
# Copy in the original mirror list again
mv $mnt/etc/pacman.d/mirrorlist.bk01 $mnt/etc/pacman.d/mirrorlist
# Clear package cache, repo DB cache and logs
find $mnt/var/cache/pacman/pkg -type f -print0 | xargs -0 rm -fv
find $mnt/var/lib/pacman/sync -type f -print0 | xargs -0 rm -fv
find $mnt/var/log -type f -print0 | xargs -0 rm -fv
 
#Clear arch key ring
rm -Rf $mnt/etc/pacman.d/gnupg

#Trim the FS to minmize space
fstrim $mnt

sleep 5

#Umount and remove loop device
umount $partition
losetup -d $image

Now we have an image file and need to convert that to an AMI.

For that we need to boot up an EC2 instance (any linux operating system is fine), copy the image file over to it (if we not already did the above using an EC2 instance), create and mount a second EBS volume and copy the content of the image to the volume (using something like dd - or even better ddpt). Finally we snapshot the volume and turn the snapshot into the AMI.

Using AWS CLI tool for that the process looks like (in Bash):

# Create the volume used for the image target and attach it
volume_id=`aws ec2 create-volume --region $aws_region --availability-zone $aws_az --no-encrypted --volume-type gp3 --size 8 --tag-specifications 'ResourceType=volume,Tags=[{Key=arch_ami,Value='${ami_name}'}]' | jq -r ."VolumeId"`

#Wait for the volume to be ready
aws ec2 wait volume-available --region $aws_region --volume-ids $volume_id

aws ec2 attach-volume --region $aws_region --device /dev/xvdf --instance $instance_id --volume-id $volume_id

#Slep a moment to let the volume settle and devices get created
sleep 10


#Copy the image to the volume using ddpt (needs to be installed from AUR)
target=<target block device>
sudo ddpt if=$img_file of=$target bs=512 conv=sparse,fsync oflag=sparse,strunc 

#Detach the volume 
aws ec2 detach-volume --region $aws_region --volume-id $volume_id
 
#Detach will take a few seconds
sleep 10

#Take the snapshot
snap_id=`aws ec2 create-snapshot --region $aws_region --volume-id $volume_id --tag-specifications 'ResourceType=snapshot,Tags=[{Key=arch_ami,Value='${ami_name}'}]' | jq -r ."SnapshotId"`

#wait for snapshot completion
aws ec2 wait snapshot-completed --region $aws_region --snapshot-ids $snap_id

#Delete the volume
aws ec2 delete-volume --region $aws_region --volume-id $volume_id

#Register the AMI in the current region
ami_id=`aws ec2 register-image --region $aws_region --architecture x86_64 --ena-support --block-device-mappings "DeviceName=/dev/sda1,Ebs={SnapshotId=$snap_id}" --name $ami_name --description "$ami_description" --root-device-name /dev/sda1 --virtualization-type hvm | jq -r .ImageId`

And that's it. You now have a private AMI that you can use to boot new instances. To make it public you just need to change the access permissions and also remember to allow the public read access to the underlying snapshot. If you don't do the latter others can still use the AMI, but they can not make copies of it and move it to other regions.