Difference between revisions of "DeveloperWiki:HOWTO Be A Packager"

From ArchWiki
Jump to: navigation, search
(fix nymeria svn url)
(completely restructure this guide, streamline it, remove confusing statements, add helper scripts - intended to guide new TUs/Devs)
 
(18 intermediate revisions by 8 users not shown)
Line 5: Line 5:
 
Please follow them closely.
 
Please follow them closely.
  
[[Arch Packaging Standards]]
+
[[Arch packaging standards]]
  
== How To Use SVN ==
 
  
=== Non-recursive checkout ===
+
== Preparation and Setup ==
  
For core, extra, testing and staging repos:
+
=== Installing the Packages ===
  svn checkout -N svn+ssh://gerolde.archlinux.org/srv/svn-packages
+
 
 +
Make sure you have the packages {{ic|devtools}} and {{ic|namcap}} installed.
  
For community, community-testing, community-staging, multilib, multilib-testing, multilib-staging:
+
=== SSH Config ===
  svn checkout -N svn+ssh://svn-community@nymeria.archlinux.org/srv/repos/svn-community/svn
 
  
 +
If you have multiple SSH keys in your SSH Agent, you will need to make sure that the correct key is being used to contact the Arch servers.
 +
Also, when your local username differs from the one being used on the Arch servers, you need to take care of that too.
  
This creates a directory named "svn-packages" which contains nothing. It does, however, know that it is an svn checkout.
+
Example {{ic|~/.ssh/config}} excerpt:
 +
<pre>
 +
host repos.archlinux.org
 +
hostname repos.archlinux.org
 +
port 22
 +
user foobar
 +
IdentityFile ~/.ssh/id_arch
 +
IdentitiesOnly yes
 +
</pre>
  
=== Checkout a package ===
+
=== makepkg Config ===
  
  cd svn-packages
+
Make sure to configure your {{ic|~/.makepkg.conf}} with the correct {{ic|PACKAGER}} and {{ic|GPGKEY}} variables.
  svn update package-name
+
Wrong signatures or missing {{ic|PACKAGER}} will prevent your packages from entering the repository.
  
This will pull the package you requested into your checkout. From now on, any time you `svn update` at the top level, this will be updated as well.
+
<pre>
 +
PACKAGER="Foo Bar <foo.bar@archlinux.org>"
 +
GPGKEY="0x0123456789abcdef"
 +
</pre>
  
=== Updating all packages ===
+
=== Local SVN Setup using Non-Recursive Checkouts ===
  
  cd svn-packages
+
As SVN provides the ability to do scoped checkouts you can initialize an empty local checkout and later on only fetch the packages that you want.
  svn update
+
To setup the local checkouts run the following commands.
  
=== Adding a package ===
+
For core, extra, testing and staging repos:
  
   cd svn-packages
+
   svn checkout -N svn+ssh://svn-packages@repos.archlinux.org/srv/repos/svn-packages/svn svn-packages
  mkdir new-package
 
  mkdir new-package/{repos,trunk}
 
  $EDITOR new-package/trunk/PKGBUILD
 
  svn add new-package
 
  svn propset svn:keywords "Id" new-package/trunk/PKGBUILD
 
  svn commit
 
  
=== Removing a package ===
+
For community, community-testing, community-staging, multilib, multilib-testing, multilib-staging:
  ssh gerolde.archlinux.org
 
  /arch/db-remove repo-name arch packagename
 
  i.e. /arch/db-remove core i686 openssh
 
  
And if you want to really kill the package, you will need to <pre>svn rm</pre> the entire package directory after the above steps and commit the deletion.
+
  svn checkout -N svn+ssh://svn-community@repos.archlinux.org/srv/repos/svn-community/svn svn-community
  
=== Moving a package between repos ===
+
This creates two directories named {{ic|svn-packages}} and {{ic|svn-community}} which contain nothing but are properly setup as svn repositories.
  ssh gerolde.archlinux.org
 
  /arch/db-move fromrepo torepo packagename
 
  i.e. /arch/db-move testing core openssh
 
  
Alternatively, the move from testing is so common we have helper scripts:
+
=== Helper Scripts (optional) ===
  
  /arch/testing2x openssh bzip2 coreutils
+
==== ch ====
  /arch/testing2x64 openssh bzip2 coreutils
 
  
These scripts only work if the packages on the commandline are either in ''core'' or ''extra''. If a package is only in testing, you have to use ''testing2core'', ''testing2core64'', ''testing2extra'' or ''testing2extra64''.
+
This shell script eases setting up and maintaining the chroots for building the packages immensely.
 +
The script has been developed by [https://www.archlinux.org/people/developers/#bluewind Bluewind] and can be found here: [https://git.server-speed.net/users/flo/bin/plain/ch ch].
  
=== "Tagging" releases ===
+
As this script relies on [[Btrfs]], you have to create a Btrfs volume (20GiB is mostly sufficient), which can either be a file, a logical volume or a dedicated partition.
 +
Furthermore by default this script assumes that the Btrfs for the chroots is mounted at {{ic|/mnt/chroots/arch}}.
  
  cd svn-packages/package-name/trunk
+
Afterwards you can create a 64bit package using:
  archrelease extra-i686
 
  
This makes an svn copy of the trunk entries in a directory named "extra-i686" indicating that this package is in the extra repository for the i686 architecture.
+
  ch build 64
This will be done automatically when using tools such as extrapkg (see below)
 
  
=== Cleaning up your checkout ===
+
(automatically and conveniently invokes makechrootpkg with all required arguments)
  
Since you are now maintaining a non-recursive checkout, you may want to get rid of packages that you are no longer tracking:
+
For further details please take a look at the head of the script, it provides some explanations and usage examples.
  
  svn update package1 package2 --set-depth exclude
 
  
Or if you want an empty toplevel again:
+
== The Workflow ==
  
  svn update --set-depth empty
+
=== Checkout/update your Local Repository ===
 
+
   cd svn-community
== The Process ==
 
=== Checkout/update your local repository ===
 
   cd svn-packages
 
 
   svn update
 
   svn update
=== Update the needed package ===
+
 
 +
=== Adding a new Package ===
 +
This step is only required when adding a new package to the repository for the first time.
 +
  cd svn-community
 +
  mkdir -p new-package/{repos,trunk}
 +
  cd new-package
 +
  $EDITOR trunk/PKGBUILD
 +
  svn add .
 +
  svn propset svn:keywords "Id" trunk/PKGBUILD
 +
  svn commit
 +
 
 +
=== Updating the needed Package ===
 
   svn update some-package
 
   svn update some-package
=== Traverse to the package's trunk directory ===
 
  cd some-package/trunk/
 
  
 
=== Change and build ===
 
=== Change and build ===
 +
  cd some-package/trunk/
 
   $EDITOR PKGBUILD
 
   $EDITOR PKGBUILD
   makepkg
+
   extra-x86_64-build
 +
 
 +
It is '''mandatory''' to build your package using a clean [[DeveloperWiki:Building_in_a_Clean_Chroot|chroot]].
  
It is highly recommended to build your package using a clean [[DeveloperWiki:Building_in_a_Clean_Chroot|chroot]]
+
Or, if you are using the [[DeveloperWiki:HOWTO_Be_A_Packager#ch|ch]] helper, simply do:
 +
  cd some-package/trunk/
 +
  $EDITOR PKGBUILD
 +
  ch clean 64
 +
  ch update 64
 +
  ch build 64
  
=== Run namcap on both PKGBUILD and package ===
+
=== Run namcap on both PKGBUILD and Package ===
 
   namcap PKGBUILD
 
   namcap PKGBUILD
   namcap some-package-1.0-1-i686.pkg.tar.gz
+
   namcap some-package-1.0-1-x86_64.pkg.tar.xz
=== Use devtools to upload and commit ===
+
 
This is repo dependent. For 'extra', you use 'extrapkg'. 'testingpkg' for 'testing', etc
+
=== Run checkpkg on the Package ===
   extrapkg "A commit message"
+
Run in the directory with your freshly built package to get a file list diff compared with the package version currently in the repos.
=== Update the repository ===
+
This can be skipped when adding a new package to the repository for the first time (e.g. by importing it from [[AUR]] to ''community'').
Use 'db-update'. It will find new packages for any repository and it manages both i686 and x86_64 architectures at once, if present. For example:
+
 
  ssh gerolde.archlinux.org
+
=== Use devtools to sign, upload and commit ===
  /arch/db-update
+
Once you are satisfied with the package, you need to make sure all your changes are tracked in the repository.
 +
Run:
 +
 
 +
  svn status
 +
 
 +
in the {{ic|some-package/trunk/}} directory and check the output carefully.
 +
If, for example, you have added a new {{ic|some-package.install}} file, you need to tell svn to track that file, e.g.:
 +
 
 +
  svn add some-package.install
 +
 
 +
Make sure to '''never''' add the binary packages, makepkg logs, etc. to the repository!
 +
 
 +
When you're ready to proceed, you can run the devtools script to sign, upload and commit your work.
 +
This is repo dependent.
 +
For ''extra'' you use {{ic|extrapkg}}, {{ic|communitypkg}} for ''community'', etc.
 +
 
 +
   extrapkg "update to 1.2.3, add post_upgrade hook to fix permissions"
 +
 
 +
Please try to write concise commit messages.
 +
If the package is simply an upstream change, that is fine, but if anything more complex changes, please inform us by writing an appropriate commit message.
 +
 
 +
=== Update the Repository ===
 +
Use {{ic|db-update}}.
 +
It will find new packages for any repository and it manages both {{ic|i686}} and {{ic|x86_64}} architectures at once, if present.
 +
 
 +
For example:
  
== Staging Directories ==
+
  ssh repos.archlinux.org "/packages/db-update"
  
Staging directories are needed on gerolde.archlinux.org for uploading of packages. The following structure is NOT automatically created. You must do it yourself:
+
or:
  
    ~/staging/
+
  ssh repos.archlinux.org "/community/db-update"
    |-- core/
+
 
    |-- extra/
+
== Other Operations ==
    `-- testing/
+
 
 +
=== Removing a Package ===
 +
  ssh repos.archlinux.org "/packages/db-remove repo-name arch packagename"
 +
i.e.:
 +
  ssh repos.archlinux.org "/packages/db-remove core i686 openssh"
 +
 
 +
And if you want to really kill the package, you will need to {{ic|svn rm}} the entire package directory after the above steps and commit the deletion.
 +
 
 +
Sometime the previous command yields:
 +
    svn: E155035: "'/path/to/pkg/<PKG>' is the root of a working copy and cannot be deleted"
 +
 
 +
You can remotely remove it with:
 +
    svn rm svn+ssh://svn-packages@repos.archlinux.org/srv/repos/svn-packages/svn/<PKG>
 +
 
 +
=== Moving a package between repos ===
 +
  ssh repos.archlinux.org "/packages/db-move fromrepo torepo packagename"
 +
i.e.:
 +
  ssh repos.archlinux.org "/packages/db-move testing core openssh"
 +
 
 +
Alternatively, the move from testing is so common we have helper scripts:
 +
 
 +
  /packages/testing2x openssh bzip2 coreutils
 +
  /packages/testing2x64 openssh bzip2 coreutils
 +
 
 +
These scripts only work if the packages on the commandline are either in ''core'' or ''extra''.
 +
If a package is only in testing, you have to use ''testing2core'', ''testing2core64'', ''testing2extra'' or ''testing2extra64''.
 +
 
 +
=== "Tagging" releases ===
 +
 
 +
Fetch the package dir using {{ic|archco}} or {{ic|communityco}} or from an svn checkout.
 +
Then:
 +
  cd package-name/trunk
 +
  archrelease extra-i686
 +
 
 +
This makes an svn copy of the trunk entries in a directory named {{ic|extra-i686}} indicating that this package is in the extra repository for the ''i686'' architecture.
 +
This will be done automatically when using tools such as {{ic|extrapkg}} (see [[DeveloperWiki:HOWTO_Be_A_Packager#Use_devtools_to_sign.2C_upload_and_commit|Use devtools to sign, upload and commit]]).
  
These directories are searched by the db scripts to find new packages and those slated for removal.
 
  
 
== Miscellaneous Stuff ==
 
== Miscellaneous Stuff ==
Line 120: Line 191:
 
=== SVN $Id$ tags ===
 
=== SVN $Id$ tags ===
  
$Id$ tags are a nice helper for PKGBUILDs and should be added to the top of all PKGBUILDs in a comment.
+
{{ic|$Id$}} tags are a nice helper for PKGBUILDs and should be added to the top of all PKGBUILDs in a comment.
 
However, svn needs an additional push to know that it should modify this line on checkout.
 
However, svn needs an additional push to know that it should modify this line on checkout.
  
 
   svn propset svn:keywords "Id" my-package/trunk/PKGBUILD
 
   svn propset svn:keywords "Id" my-package/trunk/PKGBUILD
  
=== Package checking tools ===
+
=== Cleaning up your checkout ===
  
==== namcap ====
+
Since you are now maintaining a non-recursive checkout, you may want to get rid of packages that you are no longer tracking:
Run on both your PKGBUILD and package to check for common packaging problems.
 
  
==== checkpkg ====
+
  svn update package1 package2 --set-depth exclude
Run (as root) in the directory with your freshly built package to get a file list diff compared with the package version currently in the repos.
 
  
=== Commit messages ===
+
Or if you want an empty toplevel again:
  
Please try to write concise commit messages. If the package is simply an upstream change, that is fine, but if anything more complex changes, please inform us by writing an appropriate commit message.
+
  svn update --set-depth empty
  
 
=== Avoid having to enter your password all the time ===
 
=== Avoid having to enter your password all the time ===
  
When working with ''extrapkg'' and the other devtools, quite a few ssh connections are established, even when using ssh keys and the ssh agent. You can work around that.
+
When working with {{ic|extrapkg}} and the other devtools, quite a few ssh connections are established, even when using ssh keys and the ssh agent.
 +
You can work around that.
  
Add this to your ''$HOME/.ssh/config'':
+
Add this to your {{ic|~/.ssh/config}}:
  
ControlPath /home/<your username>/.ssh/master-%h-%p-%r
+
  ControlPath ~/.ssh/master-%h-%p-%r
+
 
Host gerolde.archlinux.org
+
  Host repos.archlinux.org
  
 
Now, before you start working, open a ssh session with
 
Now, before you start working, open a ssh session with
  
ssh -M gerolde.archlinux.org
+
  ssh -M repos.archlinux.org
  
Enter your password and leave that session open until you are finished. All ssh sessions (including scp and svn+ssh) will now be tunneled through this connection.
+
Enter your password and leave that session open until you are finished.
 +
All ssh sessions (including scp and svn+ssh) will now be tunneled through this connection.

Latest revision as of 09:00, 15 September 2017

Follow Package Guidelines

Package guidelines can be found in the Arch Linux documentation. Please follow them closely.

Arch packaging standards


Preparation and Setup

Installing the Packages

Make sure you have the packages devtools and namcap installed.

SSH Config

If you have multiple SSH keys in your SSH Agent, you will need to make sure that the correct key is being used to contact the Arch servers. Also, when your local username differs from the one being used on the Arch servers, you need to take care of that too.

Example ~/.ssh/config excerpt:

host repos.archlinux.org
	hostname repos.archlinux.org
	port 22
	user foobar
	IdentityFile ~/.ssh/id_arch
	IdentitiesOnly yes

makepkg Config

Make sure to configure your ~/.makepkg.conf with the correct PACKAGER and GPGKEY variables. Wrong signatures or missing PACKAGER will prevent your packages from entering the repository.

PACKAGER="Foo Bar <foo.bar@archlinux.org>"
GPGKEY="0x0123456789abcdef"

Local SVN Setup using Non-Recursive Checkouts

As SVN provides the ability to do scoped checkouts you can initialize an empty local checkout and later on only fetch the packages that you want. To setup the local checkouts run the following commands.

For core, extra, testing and staging repos:

  svn checkout -N svn+ssh://svn-packages@repos.archlinux.org/srv/repos/svn-packages/svn svn-packages

For community, community-testing, community-staging, multilib, multilib-testing, multilib-staging:

  svn checkout -N svn+ssh://svn-community@repos.archlinux.org/srv/repos/svn-community/svn svn-community

This creates two directories named svn-packages and svn-community which contain nothing but are properly setup as svn repositories.

Helper Scripts (optional)

ch

This shell script eases setting up and maintaining the chroots for building the packages immensely. The script has been developed by Bluewind and can be found here: ch.

As this script relies on Btrfs, you have to create a Btrfs volume (20GiB is mostly sufficient), which can either be a file, a logical volume or a dedicated partition. Furthermore by default this script assumes that the Btrfs for the chroots is mounted at /mnt/chroots/arch.

Afterwards you can create a 64bit package using:

  ch build 64

(automatically and conveniently invokes makechrootpkg with all required arguments)

For further details please take a look at the head of the script, it provides some explanations and usage examples.


The Workflow

Checkout/update your Local Repository

  cd svn-community
  svn update

Adding a new Package

This step is only required when adding a new package to the repository for the first time.

  cd svn-community
  mkdir -p new-package/{repos,trunk}
  cd new-package
  $EDITOR trunk/PKGBUILD
  svn add .
  svn propset svn:keywords "Id" trunk/PKGBUILD
  svn commit

Updating the needed Package

  svn update some-package

Change and build

  cd some-package/trunk/
  $EDITOR PKGBUILD
  extra-x86_64-build

It is mandatory to build your package using a clean chroot.

Or, if you are using the ch helper, simply do:

  cd some-package/trunk/
  $EDITOR PKGBUILD
  ch clean 64
  ch update 64
  ch build 64

Run namcap on both PKGBUILD and Package

  namcap PKGBUILD
  namcap some-package-1.0-1-x86_64.pkg.tar.xz

Run checkpkg on the Package

Run in the directory with your freshly built package to get a file list diff compared with the package version currently in the repos. This can be skipped when adding a new package to the repository for the first time (e.g. by importing it from AUR to community).

Use devtools to sign, upload and commit

Once you are satisfied with the package, you need to make sure all your changes are tracked in the repository. Run:

  svn status

in the some-package/trunk/ directory and check the output carefully. If, for example, you have added a new some-package.install file, you need to tell svn to track that file, e.g.:

  svn add some-package.install

Make sure to never add the binary packages, makepkg logs, etc. to the repository!

When you're ready to proceed, you can run the devtools script to sign, upload and commit your work. This is repo dependent. For extra you use extrapkg, communitypkg for community, etc.

  extrapkg "update to 1.2.3, add post_upgrade hook to fix permissions"

Please try to write concise commit messages. If the package is simply an upstream change, that is fine, but if anything more complex changes, please inform us by writing an appropriate commit message.

Update the Repository

Use db-update. It will find new packages for any repository and it manages both i686 and x86_64 architectures at once, if present.

For example:

  ssh repos.archlinux.org "/packages/db-update"

or:

  ssh repos.archlinux.org "/community/db-update"

Other Operations

Removing a Package

  ssh repos.archlinux.org "/packages/db-remove repo-name arch packagename"

i.e.:

  ssh repos.archlinux.org "/packages/db-remove core i686 openssh"

And if you want to really kill the package, you will need to svn rm the entire package directory after the above steps and commit the deletion.

Sometime the previous command yields:

   svn: E155035: "'/path/to/pkg/<PKG>' is the root of a working copy and cannot be deleted"

You can remotely remove it with:

   svn rm svn+ssh://svn-packages@repos.archlinux.org/srv/repos/svn-packages/svn/<PKG>

Moving a package between repos

  ssh repos.archlinux.org "/packages/db-move fromrepo torepo packagename"

i.e.:

  ssh repos.archlinux.org "/packages/db-move testing core openssh"

Alternatively, the move from testing is so common we have helper scripts:

  /packages/testing2x openssh bzip2 coreutils
  /packages/testing2x64 openssh bzip2 coreutils

These scripts only work if the packages on the commandline are either in core or extra. If a package is only in testing, you have to use testing2core, testing2core64, testing2extra or testing2extra64.

"Tagging" releases

Fetch the package dir using archco or communityco or from an svn checkout. Then:

  cd package-name/trunk
  archrelease extra-i686

This makes an svn copy of the trunk entries in a directory named extra-i686 indicating that this package is in the extra repository for the i686 architecture. This will be done automatically when using tools such as extrapkg (see Use devtools to sign, upload and commit).


Miscellaneous Stuff

SVN $Id$ tags

$Id$ tags are a nice helper for PKGBUILDs and should be added to the top of all PKGBUILDs in a comment. However, svn needs an additional push to know that it should modify this line on checkout.

  svn propset svn:keywords "Id" my-package/trunk/PKGBUILD

Cleaning up your checkout

Since you are now maintaining a non-recursive checkout, you may want to get rid of packages that you are no longer tracking:

  svn update package1 package2 --set-depth exclude

Or if you want an empty toplevel again:

  svn update --set-depth empty

Avoid having to enter your password all the time

When working with extrapkg and the other devtools, quite a few ssh connections are established, even when using ssh keys and the ssh agent. You can work around that.

Add this to your ~/.ssh/config:

  ControlPath ~/.ssh/master-%h-%p-%r
  
  Host repos.archlinux.org

Now, before you start working, open a ssh session with

  ssh -M repos.archlinux.org

Enter your password and leave that session open until you are finished. All ssh sessions (including scp and svn+ssh) will now be tunneled through this connection.