User:Grawlinson/Packaging notes

Systemd

Units

Use the upstream unit files whenever they exist
Try not to do anything Arch-specific. This will maximize chances of not having to change behavior in the future once the unit files are provided by upstream. In particular avoid 1 2 EnvironmentFile=, especially if it points to the Arch-specific /etc/conf.d
Always separate initialization behavior from the actual daemon behavior. If necessary, use a separate unit for the initialization, blocked on a ConditionFoo from systemd.unit(5). An example of this is sshd.service and sshdgenkeys.service.

Not using an EnvironmentFile= is OK if:

Either the daemon has its own configuration file where the same settings can be specified
The default service file "just works" in the most common case. Users who want to change the behavior should then override the default service file. If it is not possible to provide a sane default service file, it should be discussed on a case-by-case basis

A few comments about service files, assuming current behavior should be roughly preserved, and fancy behavior avoided:

If your service requires the network to be configured before it starts, use After=network.target. Do not use Wants=network.target or Requires=network.target
Use Type=forking, unless you know it's not necessary
- Many daemons use the exit of the first process to signal that they are ready, so to minimize problems, it is safest to use this mode
- To make sure that systemd is able to figure out which process is the main process, tell the daemon to write a pidfile and point systemd to it using PIDFile=
- If the daemon in question is dbus-activated, socket-activated, or specifically supports Type=notify, that's a different matter, but currently only the case for a minority of daemons
Arch's rc scripts do not support dependencies, but with systemd they should be added add where necessary
- The most typical case is that A requires the service B to be running before A is started. In that case add Requires=B and After=B to A.
- If the dependency is optional then add Wants=B and After=B instead
- Dependencies are typically placed on services and not on targets

If you want to get fancy, you should know what you are doing.

Note: Keep in mind that values to keys such as ExecStart and ExecStop are not run within a shell, but only passed to execv

Users and groups

Instead of creating users/groups in PKGBUILD/$pkgname.install, ship a sysusers.d(5) config file in /usr/lib/sysusers.d.
A pacman hook included in systemd will run systemd-sysusers foo.conf upon install to ensure the necessary users/groups are created right away, not just on the next boot.

Group Memberships

When assigning the primary group for an user in sysusers.conf, it will not be added to /etc/group unless the auxilary relationship is explicitly stated as shown below. See upstream bug report for explanation.

sysusers.conf

# Create group 'example'
g example

# Create 3 users 'example{1..3}' with primary group 'example'
u example1 -:example
u example2 -:example
u example3 -:example

# For users to appear in /etc/group, the auxilary relationship must be explicitly stated.
m example1 example
m example2 example
m example3 example

Temporary files and directories

Instead of creating necessary runtime directories and files when a service is started (as some rc scripts do), ship a tmpfiles.d(5) config file in /usr/lib/tmpfiles.d.
A pacman hook included in systemd will run systemd-tmpfiles --create foo.conf upon install to ensure the necessary runtime files are created right away, not just on the next boot

Tip: This feature can be used for a whole lot of other things, e.g. for writing to arbitrary files, even in /sys

Man pages

Some binaries include a hidden 'man' subcommand that generates man pages. Since it is not known where the binaries get the date from (internal string, external invocation of date, etc), it is simpler to replace the date with a known value to ensure reproducibility.

PKGBUILD

# ensure reproducibility of man page
# i'm not 100% sure where the man sub-command gets the date from, i assume
# it uses $TODAYS_DATE, so this should make the man page reproducible
local _commit_date=$(git show --no-patch --format=%cd --date=format:%Y-%m-%d)

sed \
  -i page.1 \
  -e "s/\"[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\"/\"$_commit_date\"/"

Shell Completions

Shells generally have system-wide directories to store completions. The following table is a summary of where completion files should reside in, and what they should be called.

Note: There should only be optional dependencies on bash-completion or zsh-completions when completion files exist in these packages.

Shell	Directory	File
bash	`/usr/share/bash-completion/completions`	`binary_name`
fish	`/usr/share/fish/vendor_completions.d`	`binary_name.fish`
zsh	`/usr/share/zsh/site-functions`	`_binary_name`

Other shells:

Elvish does not have a system-wide directory for completions yet[1].

Language Specific

For language specific dependencies, e.g. Python (python-*) and Ruby (ruby-*), it is slightly easier to list dependencies like this:

PKGBUILD

_deps=('cachecontrol' 'cachy' 'cleo' 'html5lib' 'lockfile'
       'packaging' 'pkginfo' 'poetry-core' 'requests' 'requests-toolbelt'
       'shellingham' 'tomlkit' 'keyring' 'pexpect' 'virtualenv')
depends=("${_deps[@]/#/python-}")

Go

Upstream project without go modules

PKGBUILD

url=https://github.com/upstream_user/upstream_project

prepare() {
  cd "$pkgname-$pkgver"
  go mod init "${url#https://}"
  go mod tidy
}

Skip specific tests with `go test`

PKGBUILD

test() {
  cd "$pkgname-$pkgver"
  go test -v $(go list ./... | grep -v "filetoremovehere")
}

Download dependencies before building

PKGBUILD

prepare() {
  cd "$pkgname-$pkgver"
  go mod vendor
}

Node.js

Ensure `npm` shuts up

PKGBUILD

build() {
  cd "$pkgname-$pkgver"

  local NPM_FLAGS=(--no-audit --no-fund --no-update-notifier)
  npm install --cache "$srcdir/npm-cache" "${NPM_FLAGS[@]}"
  npm run dist:lin "${NPM_FLAGS[@]}"
}

Python

PEP-517 install method

Simple method for packages that comply with PEP-517. Will not work with packages using setup.py and other weirdness.

Using python-build, python-installer and the build-system present in pyproject.toml.

PKGBUILD

makedepends=('python-build' 'python-installer' 'python-$BUILD_SYSTEM')

build() {
  cd "$_pkgname-$pkgver"

  python -m build \
    --wheel \
    --no-isolation
}

package() {
  cd "$_pkgname-$pkgver"

  python -m installer \
    --destdir="$pkgdir" \
    dist/*.whl
}

Remove strict dependency requirements

PKGBUILD

prepare() {
  cd "$_name-$pkgver"
  sed -i \
    -e 's:,[[:space:]]\?<=\?[[:space:]]\?[[:digit:]|.]*::g' \
    -e 's:==:>=:g' \
    requirements.txt
}

Skip specific tests with `pytest`

PKGBUILD

test() {
  cd "$_name-$pkgver"
  pytest --deselect test/test_snapshot.py::test_snapshots[html-snapshot11]
}

Ensure projects with `src`-like structure pass tests

Note: This is because Python looks for the package module/folder in PYTHONPATH.

PKGBUILD

test() {
  cd "$_name-$pkgver"
  PYTHONPATH="$PWD/src:$PYTHONPATH" pytest
}

Ruby

Example `package` function

PKGBUILD

package() {
  local _gemdir="$(ruby -e'puts Gem.default_dir')"

  gem install \
    --verbose \
    --ignore-dependencies \
    --no-user-install \
    --install-dir "$pkgdir/$_gemdir" \
    --bindir "$pkgdir/usr/bin" \
    "$_gemname-$pkgver.gem"

  # delete cache
  cd "$pkgdir/$_gemdir"
  rm -rf cache

  # delete unnecessary files & folders
  rm -vrf "extensions/$CARCH-linux/$(basename $_gemdir)/$_gemname-$pkgver/gem_make.out"
  cd "gems/$_gemname-$pkgver"
  find . -type f -name ".*" -delete
  rm -rf .github .rspec appveyor.yml

  # move documentation
  install -vd "$pkgdir/usr/share/doc/$pkgname"
  mv README.md examples "$pkgdir/usr/share/doc/$pkgname"

  # move license
  install -vd "$pkgdir/usr/share/licenses/$pkgname"
  mv LICENSE "$pkgdir/usr/share/licenses/$pkgname"
}

Reproducible packaging

When the gem requires native extensions to be compiled, unnecessary build artifacts are introduced to the package.

Note: gem.build_complete does not need to be deleted, as it is possibly referenced as part of the Gemfile/Bundler specification.

The following artifacts are generated:

gem_make.out
mkmf.log
page-Makefile.ri and other similar pages. These are only generated when ruby-rdoc is a dependency, and --no-document is not passed to gem install.

By default, ruby-rdoc is overzealous in which files it chooses to generate documentation for. This is an extension to the above package function that attempts to emulate the documentation generated by gem install while preserving reproducibility.

PKGBUILD#package

# generate reproducible documentation
install -vd "$pkgdir/$_gemdir/doc/$_gemname-$pkgver"
cd "$pkgdir/$_gemdir/gems/$_gemname-$pkgver"
rdoc \
  --format ri \
  --output "$pkgdir$_gemdir/doc/$_gemname-$pkgver/ri" \
  ./lib
# delete unnecessary rdoc metadata file
rm -f "$pkgdir$_gemdir/doc/$_gemname-$pkgver/ri/created.rid"

Rust

Reproducible packaging

Do not specify cargo as makedepends, instead explicitly state either rust or rustup. Only use rustup when upstream state the toolchain version in a file (rust-toolchain.toml or rust-toolchain). This is because rustup does not pin the compiler version by default.

Skip specific tests

cargo test --locked --target-dir=target -- --skip test_markdown_rendering

Ecosystem specific

CMake

ctest only works if tests are integrated into CMakeLists.txt.

PKGBUILD

build() {
  cmake \
    -B build \
    -S "$pkgname-$pkgver" \
    -D CMAKE_INSTALL_PREFIX=/usr \
    ... additional options

  cmake --build build
}

check() {
  cd build

  ctest --output-on-failure
}

package() {
  DESTDIR="$pkgdir" cmake --install build
}

Nomad drivers

Nomad drivers can be installed globally to /usr/lib/nomad/plugins.

Terraform providers

Terraform providers can be installed globally, but do not follow the FHS.

PKGBUILD

package() {
  cd "$pkgname-$pkgver"
  # Terraform unfortunately only accepts non-FHS compliant directories for plugins :(
  # https://www.hashicorp.com/blog/automatic-installation-of-third-party-providers-with-terraform-0-13
  # tl;dr $PLUGIN_DIRECTORY/$SOURCEHOSTNAME/$SOURCENAMESPACE/$NAME/$VERSION/$OS_$ARCH/
  install -vDm755 "build/$pkgname" \
    -t "$pkgdir/usr/share/terraform/plugins/registry.terraform.io/terraform-provider/provider/$pkgver/linux_amd64"
}

Nginx modules

All modules must be compiled dynamically. Upstream has documentation on converting static modules to dynamic modules.

Tip: For nginx-mainline, substitute the package dependencies.

PKGBUILD

pkgname=nginx-mod-module
url='https://github.com/upstream-authors/module-name'
_modname="${url##*/}"
depends=(nginx)
makedepends=(nginx-src)

prepare() {
  mkdir -p build
  ln -sf -t build /usr/src/nginx/{auto,src}
}

build() {
  cd build
  /usr/src/nginx/configure --add-dynamic-module="$srcdir/$_modname-$pkgver"
  make modules
}

package() {
  install -vDm755 -t "$pkgdir/usr/lib/nginx/modules" build/objs/*.so
}

Tips and tricks

License extraction

sed -n '/Monoid is dual licensed/,/OTHER DEALINGS IN THE FONT SOFTWARE./p'
sed -n "/^Copyright.*$/,/^.*DAMAGE.$/p"

Environment variables

Prefer command-line interface flags over environment variables. If environment variables are only required by one command, prefix the command with the environment variable as such: ENV_VAR=true command --flag. Utilising export tends to pollute the system environment.

Systemd

Units

Users and groups

Group Memberships

Temporary files and directories

Man pages

Shell Completions

Language Specific

Go

Upstream project without go modules

Skip specific tests with go test

Download dependencies before building

Node.js

Ensure npm shuts up

Python

PEP-517 install method

Remove strict dependency requirements

Skip specific tests with pytest

Ensure projects with src-like structure pass tests

Ruby

Example package function

Reproducible packaging

Rust

Reproducible packaging

Skip specific tests

Ecosystem specific

CMake

Nomad drivers

Terraform providers

Nginx modules

Tips and tricks

License extraction

Environment variables

Skip specific tests with `go test`

Ensure `npm` shuts up

Skip specific tests with `pytest`

Ensure projects with `src`-like structure pass tests

Example `package` function