Arch Linux AMIs for Amazon Web Services: Difference between revisions

Latest revision as of 16:16, 1 May 2024

Public community Arch AMIs

Note: Arch Linux currently does not offer official Amazon Machine Images. The ones listed here are created by the community.

AMIs

Arch Linux AMIs are listed here: http://arch-ami-list.drzee.net/

AMIs are build twice a month (the 1st and the 15th - 2:00am UTC) and are available for all regions that to do not explicitly require 'Opt-in' - see Region List. If an AMI is needed in a region where its currently not available, an AMI can be copied to that region.

The AMIs are EBS HVM AMIs and are available with two different kernels:

std - using the Standard Arch Linux Kernel from the default Arch repositories configured with the necessary modules for EC2 usage. The scheduler and I/O is not optimized for Cloud usage reducing performance.
ec2 - using an EC2 optimized kernel of the Standard Arch Linux Kernel created by UplinkLabs and hosted in a dedicated repository: see https://git.uplinklabs.net/steven/ec2-packages.git

Both kernels have been tested on many different EC2 instance types (t2, t3, t3a, m/r/c5, m/r/c6 and advanced hardware with GPUs) and are booting fine.

Note: As of 2023-06-12, there is confirmation that EC2 optimized kernel from UplinkLabs hangs upon boot when running on an instance family using the XEN hypervisor (e.g. t2-micro). The XEN kernel modules are not included in the latest kernel which is why the boot fails. Please use only instance families using the new Nitro Hypervisor - see: Instances build on Nitro. These will still work with the EC2 optimized kernel.

Note: ec2 repository ships other packages alongside linux-ec2 kernel. On default, ec2 is set as the top most repository in pacman.conf. As a result, packages from ec2 may have a higher priority than those from the Official repositories. This can lead to some inconsistent behaviors depending on actual versions of packages.

AMIs with LTS kernels are not build.

REST API to List AMIs

An REST API is available to get a JSON of available AMIs:

Note: The API endpoint now uses a custom domain name which should remain static.

Get all AMIs: https://arch-ami-api.drzee.net/
Get list of latest AMI in each region: https://arch-ami-api.drzee.net/latest (this produces the same list as http://arch-ami-list.drzee.net/ but in JSON)
Get all AMIs in region: https://arch-ami-api.drzee.net/region - replace region with the desired region: eu-north-1, eu-west-1, us-east-1 etc.
Get all AMIs in region for CPU architecture: https://arch-ami-api.drzee.net/region/arch - replace arch with x86_64
Get all AMIs in region for CPU architecture and kernel-type: https://arch-ami-api.drzee.net/region/arch/type - replace type with std or ec2
Get Latest AMI in region for in region for CPU architecture and kernel-type: https://arch-ami-api.drzee.net/region/arch/type/latest - replace region, arch and type

First Run

Note: The mirror list baked into the AMI was generated at time of image build and is using servers in Germany.

After booting the AMI it is recommended/required to execute the following steps to initialize pacman and select fast local repositories:

# pacman-key --init
# pacman-key --populate
# reflector --country "ISO 3166-1 Alpha-2 Country Code" --protocol https,http --score 20 --sort rate --save /etc/pacman.d/mirrorlist
# pacman -Syu

The Reflector package is preinstalled in the AMIs.

It is recommended to set a proper configuration for reflector in /etc/xdg/reflector/reflector.conf and enable the timer services to regularly refresh the mirror list. For details see the Reflector package documentation.

Alternative provide your own mirrorlist and do not use the reflector package.

Build process

Note: This section provides a short description on how the AMIs are build.

The entire build process runs on AWS and is fully automated.

Overall the automated build procedure is managed by a AWS Step Function that is executed at regular intervals using a Amazon EventBridge timed event.

The step function will initiate the build process and uses a combination of native calls and AWS Lambda functions for more complex elements.

A new set of AMIs is build, by booting an EC2 instance with the previous AMI and using it as the work or build machine. The build machine is bootstrapped with a special build script that essentially uses pacstrap and some additional steps to build the image, the basics are outlined below in the next section.

Following the build of the AMI the new AMI is test booted on an EC2 instance to verify that it start up correctly. If successful the AMI is distributed to the regions and registered in a DynamoDB database. The database can be queried using the API REST endpoint. Old AMIs are deleted from the regions and the DynamoDB database.

Credits

Thanks to Steven from UplinkLabs for helping to understand the build process and test the initial quality of the images. Also thanks to Mathcom for an excellent shells script to help me get started (unfortunately the link to that has been removed), which accelerate putting the basic build process together.

You may send comments and suggestions (without any promise that they will be looked at) to: arch-ami 'at' drzee.net

Wishlist

Create official EC2/Cloud optimized kernels in the Standard Arch Linux repositories.
Add the AWS CLI v2 to the Standard Arch Linux repositories (v1 is available, but may be discontinued in the future)
- Unfortunately AWS CLI v2 was removed from the Extra Repo due to issues with Python 3.12 - images are again build with aws-cli v1

Building Arch AMIs

You can also build your own Arch Linux AMI. See [1] for details.

@@ Line 2: / Line 2: @@
 [[es:Arch Linux AMIs for Amazon Web Services]]
 [[ja:Amazon Web Services の Arch Linux AMI]]
+[[pt:Arch Linux AMIs for Amazon Web Services]]
+[[zh-hans:Arch Linux AMIs for Amazon Web Services]]
 == Public community Arch AMIs ==
@@ Line 7: / Line 10: @@
 === AMIs ===
 '''Arch Linux AMIs are listed here:''' http://arch-ami-list.drzee.net/
@@ Line 15: / Line 17: @@
 The AMIs are EBS HVM AMIs and are available with two different kernels:
 *'''std''' - using the Standard Arch Linux Kernel from the default Arch repositories configured with the necessary modules for EC2 usage. The scheduler and I/O is not optimized for Cloud usage reducing performance.
-*'''ec2''' - using an EC2 optimized kernel of the Standard Arch Linux Kernel created by UplinkLabs and hosted in a dedicated repository: https://arch-linux-ami.s3.amazonaws.com/repo/$arch{{Dead link|2022|09|17|status=403}} (see also https://git.uplinklabs.net/steven/ec2-packages.git)
+*'''ec2''' - using an EC2 optimized kernel of the Standard Arch Linux Kernel created by UplinkLabs and hosted in a dedicated repository: see https://git.uplinklabs.net/steven/ec2-packages.git
+Both kernels have been tested on many different EC2 instance types (t2, t3, t3a, m/r/c5, m/r/c6 and advanced hardware with GPUs) and are booting fine.
-Both kernels have been tested on many different EC2 instance types (t2, t3, t3a, m/r/c5, m/r/c6 and advanced hardware with GPUs) and are booting fine.
+{{Note|As of 2023-06-12, there is confirmation that EC2 optimized kernel from UplinkLabs hangs upon boot when running on an instance family using the XEN hypervisor (e.g. {{ic|t2-micro}}). The XEN kernel modules are not included in the latest kernel which is why the boot fails. Please use only instance families using the new Nitro Hypervisor - see: [https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/instance-types.html#ec2-nitro-instances Instances build on Nitro]. These will still work with the EC2 optimized kernel.}}
+{{Note|'''ec2''' repository ships other packages alongside '''linux-ec2''' kernel. On default, '''ec2''' is set as the top most repository in {{ic|pacman.conf}}. As a result, packages from '''ec2''' may have a higher priority than those from the [[Official repositories]]. This can lead to some inconsistent behaviors depending on actual versions of packages.}}
 AMIs with LTS kernels are not build.
@@ Line 25: / Line 31: @@
 An REST API is available to get a JSON of available AMIs:
-{{Note|The API endpoint may change without warning.}}
+{{Note|The API endpoint now uses a custom domain name which should remain static.}}
+* Get all AMIs: {{ic|https://arch-ami-api.drzee.net/}}
+* Get list of latest AMI in each region: {{ic|https://arch-ami-api.drzee.net/latest}} (this produces the same list as http://arch-ami-list.drzee.net/ but in JSON)
+* Get all AMIs in region: {{ic|https://arch-ami-api.drzee.net/region}} - replace {{ic|region}} with the desired region: {{ic|eu-north-1}}, {{ic|eu-west-1}}, {{ic|us-east-1}} etc.
+* Get all AMIs in region for CPU architecture: {{ic|https://arch-ami-api.drzee.net/region/arch}} - replace {{ic|arch}} with {{ic|x86_64}}
+* Get all AMIs in region for CPU architecture and kernel-type: {{ic|https://arch-ami-api.drzee.net/region/arch/type}} - replace {{ic|type}} with {{ic|std}} or {{ic|ec2}}
+* Get Latest AMI in region for in region for CPU architecture and kernel-type: {{ic|https://arch-ami-api.drzee.net/region/arch/type/latest}} - replace {{ic|region}}, {{ic|arch}} and {{ic|type}}
-* Get all AMIs: {{ic|https://5nplxwo1k1.execute-api.eu-central-1.amazonaws.com/prod}}
+=== First Run ===
-* Get list of latest AMI in each region: {{ic|https://5nplxwo1k1.execute-api.eu-central-1.amazonaws.com/prod/latest}} (this produces the same list as http://arch-ami-list.drzee.net/ but in JSON)
-* Get all AMIs in region: {{ic|https://5nplxwo1k1.execute-api.eu-central-1.amazonaws.com/prod/region}} - replace {{ic|region}} with the desired region: {{ic|eu-north-1}}, {{ic|eu-west-1}}, {{ic|us-east-1}} etc.
-* Get all AMIs in region for CPU architecture: {{ic|https://5nplxwo1k1.execute-api.eu-central-1.amazonaws.com/prod/region/arch}} - replace {{ic|arch}} with {{ic|x86_64}}
-* Get all AMIs in region for CPU architecture and kernel-type: {{ic|https://5nplxwo1k1.execute-api.eu-central-1.amazonaws.com/prod/region/arch/type}} - replace {{ic|type}} with {{ic|std}} or {{ic|ec2}}
-* Get Latest AMI in region for in region for CPU architecture and kernel-type: {{ic|https://5nplxwo1k1.execute-api.eu-central-1.amazonaws.com/prod/region/arch/type/latest}} - replace {{ic|region}}, {{ic|arch}} and {{ic|type}}
-=== First Run ===
+{{Note|The mirror list baked into the AMI was generated at time of image build and is using servers in Germany.}}
-After booting the AMI it is recommended/required to execute the following steps to initialize pacman and select fast repos:
+After booting the AMI it is recommended/required to execute the following steps to initialize pacman and select fast local repositories:
   # pacman-key --init
@@ Line 44: / Line 51: @@
 The [[Reflector]] package is preinstalled in the AMIs.
+It is recommended to set a proper configuration for reflector in {{ic|/etc/xdg/reflector/reflector.conf}} and enable the timer services to regularly refresh the mirror list. For details see the [[Reflector]] package documentation.
+Alternative provide your own mirrorlist and do not use the reflector package.
 === Build process ===
@@ Line 55: / Line 66: @@
 The step function will initiate the build process and uses a combination of native calls and AWS Lambda functions for more complex elements.
-A new set of AMIs is build, by booting an EC2 instance with the previous AMI and using it as the ''work or build machine''. The ''build machine'' is bootstrapped with a special build script that largely follows the steps outlined here: http://mathcom.com/arch.aws.ami.html with some minor modifications, tweaks and additions. Step 7.4 is skipped as we do not need a dedicated firewall.
+A new set of AMIs is build, by booting an EC2 instance with the previous AMI and using it as the ''work or build machine''. The ''build machine'' is bootstrapped with a special build script that essentially uses pacstrap and some additional steps to build the image, the basics are outlined below in the next section.
 Following the build of the AMI the new AMI is ''test booted'' on an EC2 instance to verify that it start up correctly. If successful the AMI is distributed to the regions and registered in a DynamoDB database. The database can be queried using the API REST endpoint. Old AMIs are deleted from the regions and the DynamoDB database.
 === Credits ===
-Thanks to '''Steven''' from UplinkLabs for helping to understand the build process and test the initial quality of the images. Also thanks to '''Mathcom''' for the [http://mathcom.com/arch.aws.ami.html guide], which accelerate putting the basic build process together.
+Thanks to '''Steven''' from UplinkLabs for helping to understand the build process and test the initial quality of the images. Also thanks to '''Mathcom''' for an excellent shells script to help me get started (unfortunately the link to that has been removed), which accelerate putting the basic build process together.
 You may send comments and suggestions (without any promise that they will be looked at) to: '''arch-ami 'at' drzee.net'''
@@ Line 69: / Line 80: @@
 * Create official EC2/Cloud optimized kernels in the Standard Arch Linux repositories.
 * Add the AWS CLI v2 to the Standard Arch Linux repositories (v1 is available, but may be discontinued in the future)
+** Unfortunately AWS CLI v2 was removed from the Extra Repo due to issues with Python 3.12 - images are again build with aws-cli v1
 == Building Arch AMIs ==
-You can also build your own Arch Linux AMI. Here are some guides:
+You can also build your own Arch Linux AMI. See [http://arch-ami-list.drzee.net/ami_build_howto.html] for details.
-* {{AUR|linux-ec2}} compiles the Arch Linux kernel for AWS with Xen modules enabled and the XSAVE patch applied. Note that at least some instance types will also work with the stock Arch Linux kernel.
-* [https://www.uplinklabs.net/projects/arch-linux-on-ec2/ Uplink Labs] has a manual on the build process.
-* Another tutorial on building your own AMIs can be found at https://gitlab.com/anemos-io/archlinux-ec2
-* Mathcom publishes a detailed guide that uses only core Arch tools, at http://mathcom.com/arch.aws.ami.html
-== AMI Images from Uplink Labs ==
-{{Out of date|The AMIs provided by Uplink Labs are no longer maintained}}
-Uplink Labs creates new images approximately twice a month. Images are being built
-for a number of regions, and cover the following configurations:
-* ebs hvm x86_64 lts
-* s3  hvm x86_64 lts
-* ebs hvm x86_64 stable
-* s3  hvm x86_64 stable
-AMI links and more information are available at https://www.uplinklabs.net/projects/arch-linux-on-ec2/ .