Rebuilderd

Rebuilderd is an independent verification system for binary packages. It allows verification of pre-compiled packages by repeating the build step in an identical environment and then verifies that the package is identical. Right now only Arch Linux packages can be verified.

Note: rebuilderd is still experimental and a large number of builds are not reproducible yet. A failed rebuild does not necessarily mean that the package is malicious.

If you run a rebuilder or consider running a rebuilder feel free to join the #archlinux-reproducible IRC channel.

Installation

Install the rebuilderd package.

Setup

Single machine

On a single machine you just enable rebuilderd.service and the worker service(s): rebuilderd-worker@workername.service.

Afterwards you can verify rebuilderd is correctly running with a single worker by running:

# rebuildctl status

If you get an error that no authentication cookie could be found, you might need to add yourself to the rebuilderd user group.

Continue with the next section to sync packages that our worker can rebuild.

Rebuilder network

To securely run a rebuilder network we need to configure a secret key for administration with rebuildctl, and another secret key that is used by our workers to sign up. The secrets can be generated with pwgen -1s 32.

[http]
# it is highly recommended to bind to localhost and setup nginx as a reverse proxy for https
bind_addr = "127.0.0.1:8080"

[auth]
# set the generated secret for administration here
cookie = "INSECURE-CHANGE-ME"

[worker]
# set the generated secret for our workers here
signup_secret = "INSECURE-CHANGE-ME"

Start and enable rebuilderd.service.

Next we head over to our worker machines to set them up. Install rebuilderd as well and edit /etc/rebuilderd-worker.conf:

## The rebuilderd to connect to
endpoint = "https://rebuilder.example.com"
# change this to the generated secret for our workers
signup_secret = "INSECURE-CHANGE-ME"

Start and enable rebuilderd-worker@workername.service.

Syncing packages to rebuild

The rebuilder setup will not do anything by default until you explicitly configure where to sync packages from. The sync profiles are configured in /etc/rebuilderd-sync.conf. The profile names are supposed to be unique. You configure it to only build packages of a specific maintainer with the maintainer option. By default it is importing packages of all maintainers.

## rebuild all of core
[profile."archlinux-core"]
distro = "archlinux"
suite = "core"
architecture = "x86_64"
source = "https://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch"

## rebuild community packages of specific maintainers, or whitelist packages by name.
## If no filter is set, all packages are imported, if both filters are set the package only
## has to match one of them to be included.
#[profile."archlinux-community"]
#distro = "archlinux"
#suite = "community"
#architecture = "x86_64"
#source = "https://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch"
#maintainers = ["somebody"]
#pkgs = ["some-pkg", "python-*"]
#excludes = ["tensorflow*"]

Afterwards you can enable rebuilderd-sync@archlinux-core.timer to automatically sync the profile.

Ansible playbook

An Ansible playbook with a rebuilderd role can be found in Arch Linux's infastructure repository. Use this as inspiration as the Ansible repository is heavily dependent on how the Arch Linux infrastructure is set up.

Tips and tricks

Requeueing failed builds

There is currently no builtin way to requeue, you can run the following command to add all failed builds to the queue again:

# rebuildctl pkgs ls --distro archlinux --suite core --status BAD --json | jq -r '.[].name' | xargs -L1 rebuildctl queue push archlinux core

Troubleshooting

Failed to allocate scope: Unit build.scope already exists

This is happening if multiple versions of repro are running at the same time. This is going to be fixed in a future version.