User:Grawlinson/Packaging notes

From ArchWiki
Jump to navigation Jump to search

Systemd

Units

  • Use the upstream unit files whenever they exist
  • Try not to do anything Arch-specific. This will maximize chances of not having to change behavior in the future once the unit files are provided by upstream. In particular avoid 12 EnvironmentFile=, especially if it points to the Arch-specific /etc/conf.d
  • Always separate initialization behavior from the actual daemon behavior. If necessary, use a separate unit for the initialization, blocked on a ConditionFoo from systemd.unit(5). An example of this is sshd.service and sshdgenkeys.service.

Not using an EnvironmentFile= is OK if:

  • Either the daemon has its own configuration file where the same settings can be specified
  • The default service file "just works" in the most common case. Users who want to change the behavior should then override the default service file. If it is not possible to provide a sane default service file, it should be discussed on a case-by-case basis

A few comments about service files, assuming current behavior should be roughly preserved, and fancy behavior avoided:

  • If your service requires the network to be configured before it starts, use After=network.target. Do not use Wants=network.target or Requires=network.target
  • Use Type=forking, unless you know it's not necessary
    • Many daemons use the exit of the first process to signal that they are ready, so to minimize problems, it is safest to use this mode
    • To make sure that systemd is able to figure out which process is the main process, tell the daemon to write a pidfile and point systemd to it using PIDFile=
    • If the daemon in question is dbus-activated, socket-activated, or specifically supports Type=notify, that's a different matter, but currently only the case for a minority of daemons
  • Arch's rc scripts do not support dependencies, but with systemd they should be added add where necessary
    • The most typical case is that A requires the service B to be running before A is started. In that case add Requires=B and After=B to A.
    • If the dependency is optional then add Wants=B and After=B instead
    • Dependencies are typically placed on services and not on targets

If you want to get fancy, you should know what you are doing.

Note: Keep in mind that values to keys such as ExecStart and ExecStop are not run within a shell, but only passed to execv

Users and groups

  • Instead of creating users/groups in PKGBUILD/$pkgname.install, ship a sysusers.d(5) config file in /usr/lib/sysusers.d.
  • A pacman hook included in systemd will run systemd-sysusers foo.conf upon install to ensure the necessary users/groups are created right away, not just on the next boot.

Group Memberships

When assigning the primary group for an user in sysusers.conf, it will not be added to /etc/group unless the auxilary relationship is explicitly stated as shown below. See upstream bug report for explanation.

sysusers.conf
# Create group 'example'
g example

# Create 3 users 'example{1..3}' with primary group 'example'
u example1 -:example
u example2 -:example
u example3 -:example

# For users to appear in /etc/group, the auxilary relationship must be explicitly stated.
m example1 example
m example2 example
m example3 example

Temporary files and directories

  • Instead of creating necessary runtime directories and files when a service is started (as some rc scripts do), ship a tmpfiles.d(5) config file in /usr/lib/tmpfiles.d.
  • A pacman hook included in systemd will run systemd-tmpfiles --create foo.conf upon install to ensure the necessary runtime files are created right away, not just on the next boot
Tip: This feature can be used for a whole lot of other things, e.g. for writing to arbitrary files, even in /sys

Shell Completions

Shells generally have system-wide directories to store completions. The following table is a summary of where completion files should reside in, and what they should be called.

Note: There should only be optional dependencies on bash-completion or zsh-completions when completion files exist in these packages.
Shell Directory File
bash /usr/share/bash-completion/completions binary_name
fish /usr/share/fish/vendor_completions.d binary_name.fish
zsh /usr/share/zsh/site-functions _binary_name

Other shells:

  • Elvish does not have a system-wide directory for completions yet[1].

Language Specific

For language specific dependencies, e.g. Python (python-*) and Ruby (ruby-*), it is slightly easier to list dependencies like this:

PKGBUILD
_deps=('cachecontrol' 'cachy' 'cleo' 'html5lib' 'lockfile'
       'packaging' 'pkginfo' 'poetry-core' 'requests' 'requests-toolbelt'
       'shellingham' 'tomlkit' 'keyring' 'pexpect' 'virtualenv')
depends=("${_deps[@]/#/python-}")

Go

Upstream project without go modules

PKGBUILD
url=https://github.com/upstream_user/upstream_project

prepare() {
  cd "$pkgname-$pkgver"
  go mod init "${url#https://}"
  go mod tidy
}

Skip specific tests with go test

PKGBUILD
test() {
  cd "$pkgname-$pkgver"
  go test -v $(go list ./... | grep -v "filetoremovehere")
}

Download dependencies before building

PKGBUILD
prepare() {
  cd "$pkgname-$pkgver"
  go mod vendor
}

Node.js

Ensure npm shuts up

PKGBUILD
build() {
  cd "$pkgname-$pkgver"

  local NPM_FLAGS=(--no-audit --no-fund --no-update-notifier)
  npm install --cache "$srcdir/npm-cache" "${NPM_FLAGS[@]}"
  npm run dist:lin "${NPM_FLAGS[@]}"
}

Python

Skip specific tests with pytest

PKGBUILD
test() {
  cd "$_name-$pkgver"
  pytest --deselect test/test_snapshot.py::test_snapshots[html-snapshot11]
}


Ensure projects with src-like structure pass tests

Note: This is because Python looks for the package module/folder in PYTHONPATH.
PKGBUILD
test() {
  cd "$_name-$pkgver"
  PYTHONPATH="$PWD/src:$PYTHONPATH" pytest
}

Ruby

Example package function

PKGBUILD
package() {
  local _gemdir="$(ruby -e'puts Gem.default_dir')"

  gem install \
    --verbose \
    --ignore-dependencies \
    --no-user-install \
    --install-dir "$pkgdir/$_gemdir" \
    --bindir "$pkgdir/usr/bin" \
    "$_gemname-$pkgver.gem"

  # delete cache
  cd "$pkgdir/$_gemdir"
  rm -rf cache

  # delete unnecessary files & folders
  rm -vrf "extensions/$CARCH-linux/$(basename $_gemdir)/$_gemname-$pkgver/gem_make.out"
  cd "gems/$_gemname-$pkgver"
  find . -type f -name ".*" -delete
  rm -rf .github .rspec appveyor.yml

  # move documentation
  install -vd "$pkgdir/usr/share/doc/$pkgname"
  mv README.md examples "$pkgdir/usr/share/doc/$pkgname"

  # move license
  install -vd "$pkgdir/usr/share/licenses/$pkgname"
  mv LICENSE "$pkgdir/usr/share/licenses/$pkgname"
}

Reproducible packaging

When the gem requires native extensions to be compiled, unnecessary build artifacts are introduced to the package.

Note: gem.build_complete does not need to be deleted, as it is possibly referenced as part of the Gemfile/Bundler specification.

The following artifacts are generated:

  • gem_make.out
  • mkmf.log
  • page-Makefile.ri and other similar pages. These are only generated when ruby-rdoc is a dependency, and --no-document is not passed to gem install.

By default, ruby-rdoc is overzealous in which files it chooses to generate documentation for. This is an extension to the above package function that attempts to emulate the documentation generated by gem install while preserving reproducibility.

PKGBUILD#package
# generate reproducible documentation
install -vd "$pkgdir/$_gemdir/doc/$_gemname-$pkgver"
cd "$pkgdir/$_gemdir/gems/$_gemname-$pkgver"
rdoc \
  --format ri \
  --output "$pkgdir$_gemdir/doc/$_gemname-$pkgver/ri" \
  ./lib
# delete unnecessary rdoc metadata file
rm -f "$pkgdir$_gemdir/doc/$_gemname-$pkgver/ri/created.rid"

Rust

It is not necessary to specify --target, as the defaults are sane.

Download required dependencies before building, and ensure no subsequent network connections are attempted

PKGBUILD
prepare() {
  cd "$pkgname-$pkgver"
  cargo fetch --locked
}

build() {
  cd "$pkgname-$pkgver"
  cargo build --frozen
}

Reproducible packaging

Do not specify cargo as makedepends, instead explicitly state either rust or rustup. Only use rustup when upstream state the toolchain version in a file (rust-toolchain.toml or rust-toolchain). This is because rustup does not pin the compiler version by default.

Ecosystem specific

Nomad drivers

Nomad drivers can be installed globally to /usr/lib/nomad/plugins.

Terraform providers

Terraform providers can be installed globally, but do not follow the FHS.

PKGBUILD
package() {
  cd "$pkgname-$pkgver"
  # Terraform unfortunately only accepts non-FHS compliant directories for plugins :(
  # https://www.hashicorp.com/blog/automatic-installation-of-third-party-providers-with-terraform-0-13
  # tl;dr $PLUGIN_DIRECTORY/$SOURCEHOSTNAME/$SOURCENAMESPACE/$NAME/$VERSION/$OS_$ARCH/
  install -vDm755 "build/$pkgname" \
    -t "$pkgdir/usr/share/terraform/plugins/registry.terraform.io/terraform-provider/provider/$pkgver/linux_amd64"
}

Nginx modules

All modules must be compiled dynamically. Upstream has documentation on converting static modules to dynamic modules.

Tip: For nginx-mainline, substitute the package dependencies.
PKGBUILD
pkgname=nginx-mod-module
url='https://github.com/upstream-authors/module-name'
_modname="${url##*/}"
depends=(nginx)
makedepends=(nginx-src)

prepare() {
  mkdir -p build
  ln -sf -t build /usr/src/nginx/{auto,src}
}

build() {
  cd build
  /usr/src/nginx/configure --add-dynamic-module="$srcdir/$_modname-$pkgver"
  make modules
}

package() {
  install -vDm755 -t "$pkgdir/usr/lib/nginx/modules" build/objs/*.so
}

Tips and tricks

License extraction

sed -n '/Monoid is dual licensed/,/OTHER DEALINGS IN THE FONT SOFTWARE./p'
sed -n "/^Copyright.*$/,/^.*DAMAGE.$/p"

Environment variables

Prefer command-line interface flags over environment variables. If environment variables are only required by one command, prefix the command with the environment variable as such: ENV_VAR=true command --flag. Utilising export tends to pollute the system environment.