Difference between revisions of "DeveloperWiki:HOWTO Be A Packager"

From ArchWiki
Jump to: navigation, search
m (Changed protection level for "DeveloperWiki:HOWTO Be A Packager" ([Move=Allow only administrators] (indefinite)))
(completely restructure this guide, streamline it, remove confusing statements, add helper scripts - intended to guide new TUs/Devs)
Line 7: Line 7:
 
[[Arch packaging standards]]
 
[[Arch packaging standards]]
  
== How To Use SVN ==
 
  
=== Non-recursive checkout ===
+
== Preparation and Setup ==
 +
 
 +
=== Installing the Packages ===
 +
 
 +
Make sure you have the packages {{ic|devtools}} and {{ic|namcap}} installed.
 +
 
 +
=== SSH Config ===
 +
 
 +
If you have multiple SSH keys in your SSH Agent, you will need to make sure that the correct key is being used to contact the Arch servers.
 +
Also, when your local username differs from the one being used on the Arch servers, you need to take care of that too.
 +
 
 +
Example {{ic|~/.ssh/config}} excerpt:
 +
<pre>
 +
host repos.archlinux.org
 +
hostname repos.archlinux.org
 +
port 22
 +
user foobar
 +
IdentityFile ~/.ssh/id_arch
 +
IdentitiesOnly yes
 +
</pre>
 +
 
 +
=== makepkg Config ===
 +
 
 +
Make sure to configure your {{ic|~/.makepkg.conf}} with the correct {{ic|PACKAGER}} and {{ic|GPGKEY}} variables.
 +
Wrong signatures or missing {{ic|PACKAGER}} will prevent your packages from entering the repository.
 +
 
 +
<pre>
 +
PACKAGER="Foo Bar <foo.bar@archlinux.org>"
 +
GPGKEY="0x0123456789abcdef"
 +
</pre>
 +
 
 +
=== Local SVN Setup using Non-Recursive Checkouts ===
 +
 
 +
As SVN provides the ability to do scoped checkouts you can initialize an empty local checkout and later on only fetch the packages that you want.
 +
To setup the local checkouts run the following commands.
  
 
For core, extra, testing and staging repos:
 
For core, extra, testing and staging repos:
 +
 
   svn checkout -N svn+ssh://svn-packages@repos.archlinux.org/srv/repos/svn-packages/svn svn-packages
 
   svn checkout -N svn+ssh://svn-packages@repos.archlinux.org/srv/repos/svn-packages/svn svn-packages
  
 
For community, community-testing, community-staging, multilib, multilib-testing, multilib-staging:
 
For community, community-testing, community-staging, multilib, multilib-testing, multilib-staging:
 +
 
   svn checkout -N svn+ssh://svn-community@repos.archlinux.org/srv/repos/svn-community/svn svn-community
 
   svn checkout -N svn+ssh://svn-community@repos.archlinux.org/srv/repos/svn-community/svn svn-community
  
 +
This creates two directories named {{ic|svn-packages}} and {{ic|svn-community}} which contain nothing but are properly setup as svn repositories.
  
This creates two directories named "svn-packages" and "svn-community" which contains nothing. It does, however, know that it is an svn checkout.
+
=== Helper Scripts (optional) ===
  
=== Checkout a package ===
+
==== ch ====
  
You can use `archco` to fetch a dir in the svn-packages repository or `communityco` to fetch a dir from the svn-community repo. You don't need to be in an svn checkout to do that.
+
This shell script eases setting up and maintaining the chroots for building the packages immensely.
 +
The script has been developed by [https://www.archlinux.org/people/developers/#bluewind Bluewind] and can be found here: [https://git.server-speed.net/users/flo/bin/plain/ch ch].
  
Otherwise you have to cd the svn checkout and exec:
+
As this script relies on [[Btrfs]], you have to create a Btrfs volume (20GiB is mostly sufficient), which can either be a file, a logical volume or a dedicated partition.
  svn update package-name
+
Furthermore by default this script assumes that the Btrfs for the chroots is mounted at {{ic|/mnt/chroots/arch}}.
  
This will pull the package you requested into your checkout. From now on, any time you `svn update` at the top level, this will be updated as well.
+
Afterwards you can create a 64bit package using:
  
=== Updating all packages ===
+
  ch build 64
  
   cd svn-packages
+
(automatically and conveniently invokes makechrootpkg with all required arguments)
 +
 
 +
For further details please take a look at the head of the script, it provides some explanations and usage examples.
 +
 
 +
 
 +
== The Workflow ==
 +
 
 +
=== Checkout/update your Local Repository ===
 +
   cd svn-community
 
   svn update
 
   svn update
  
=== Adding a package ===
+
=== Adding a new Package ===
 
+
This step is only required when adding a new package to the repository for the first time.
   cd svn-packages
+
   cd svn-community
 
   mkdir -p new-package/{repos,trunk}
 
   mkdir -p new-package/{repos,trunk}
 
   cd new-package
 
   cd new-package
Line 44: Line 89:
 
   svn commit
 
   svn commit
  
=== Removing a package ===
+
=== Updating the needed Package ===
   ssh repos.archlinux.org
+
   svn update some-package
  /packages/db-remove repo-name arch packagename
 
  i.e. /packages/db-remove core i686 openssh
 
  
And if you want to really kill the package, you will need to <pre>svn rm</pre> the entire package directory after the above steps and commit the deletion.
+
=== Change and build ===
 +
  cd some-package/trunk/
 +
  $EDITOR PKGBUILD
 +
  extra-x86_64-build
  
Sometime the previous command yields:
+
It is '''mandatory''' to build your package using a clean [[DeveloperWiki:Building_in_a_Clean_Chroot|chroot]].
    svn: E155035: "'/path/to/pkg/<PKG>' is the root of a working copy and cannot be deleted"
+
 
 +
Or, if you are using the [[DeveloperWiki:HOWTO_Be_A_Packager#ch|ch]] helper, simply do:
 +
  cd some-package/trunk/
 +
  $EDITOR PKGBUILD
 +
  ch clean 64
 +
  ch update 64
 +
  ch build 64
 +
 
 +
=== Run namcap on both PKGBUILD and Package ===
 +
  namcap PKGBUILD
 +
  namcap some-package-1.0-1-x86_64.pkg.tar.xz
 +
 
 +
=== Run checkpkg on the Package ===
 +
Run in the directory with your freshly built package to get a file list diff compared with the package version currently in the repos.
 +
This can be skipped when adding a new package to the repository for the first time (e.g. by importing it from [[AUR]] to ''community'').
 +
 
 +
=== Use devtools to sign, upload and commit ===
 +
Once you are satisfied with the package, you need to make sure all your changes are tracked in the repository.
 +
Run:
 +
 
 +
  svn status
 +
 
 +
in the {{ic|some-package/trunk/}} directory and check the output carefully.
 +
If, for example, you have added a new {{ic|some-package.install}} file, you need to tell svn to track that file, e.g.:
 +
 
 +
  svn add some-package.install
  
You can remotely remove it with:
+
Make sure to '''never''' add the binary packages, makepkg logs, etc. to the repository!
    svn rm svn+ssh://svn-packages@repos.archlinux.org/srv/repos/svn-packages/svn/<PKG>
 
  
=== Moving a package between repos ===
+
When you're ready to proceed, you can run the devtools script to sign, upload and commit your work.
  ssh repos.archlinux.org
+
This is repo dependent.
  /packages/db-move fromrepo torepo packagename
+
For ''extra'' you use {{ic|extrapkg}}, {{ic|communitypkg}} for ''community'', etc.
  i.e. /packages/db-move testing core openssh
 
  
Alternatively, the move from testing is so common we have helper scripts:
+
  extrapkg "update to 1.2.3, add post_upgrade hook to fix permissions"
  
  /packages/testing2x openssh bzip2 coreutils
+
Please try to write concise commit messages.
  /packages/testing2x64 openssh bzip2 coreutils
+
If the package is simply an upstream change, that is fine, but if anything more complex changes, please inform us by writing an appropriate commit message.
  
These scripts only work if the packages on the commandline are either in ''core'' or ''extra''. If a package is only in testing, you have to use ''testing2core'', ''testing2core64'', ''testing2extra'' or ''testing2extra64''.
+
=== Update the Repository ===
 +
Use {{ic|db-update}}.
 +
It will find new packages for any repository and it manages both {{ic|i686}} and {{ic|x86_64}} architectures at once, if present.
  
=== "Tagging" releases ===
+
For example:
  
Fetch the package dir using `archco` or `communityco` or from an svn checkout. Then
+
  ssh repos.archlinux.org "/packages/db-update"
  
  cd package-name/trunk
+
or:
  archrelease extra-i686
 
  
This makes an svn copy of the trunk entries in a directory named "extra-i686" indicating that this package is in the extra repository for the i686 architecture.
+
  ssh repos.archlinux.org "/community/db-update"
This will be done automatically when using tools such as extrapkg (see below)
 
  
=== Cleaning up your checkout ===
+
== Other Operations ==
  
Since you are now maintaining a non-recursive checkout, you may want to get rid of packages that you are no longer tracking:
+
=== Removing a Package ===
 +
  ssh repos.archlinux.org "/packages/db-remove repo-name arch packagename"
 +
i.e.:
 +
  ssh repos.archlinux.org "/packages/db-remove core i686 openssh"
  
  svn update package1 package2 --set-depth exclude
+
And if you want to really kill the package, you will need to {{ic|svn rm}} the entire package directory after the above steps and commit the deletion.
  
Or if you want an empty toplevel again:
+
Sometime the previous command yields:
 +
    svn: E155035: "'/path/to/pkg/<PKG>' is the root of a working copy and cannot be deleted"
  
  svn update --set-depth empty
+
You can remotely remove it with:
 +
    svn rm svn+ssh://svn-packages@repos.archlinux.org/srv/repos/svn-packages/svn/<PKG>
  
== The Process ==
+
=== Moving a package between repos ===
=== Checkout/update your local repository ===
+
   ssh repos.archlinux.org "/packages/db-move fromrepo torepo packagename"
   cd svn-packages
+
i.e.:
  svn update
+
   ssh repos.archlinux.org "/packages/db-move testing core openssh"
=== Update the needed package ===
 
  svn update some-package
 
=== Traverse to the package's trunk directory ===
 
   cd some-package/trunk/
 
  
=== Change and build ===
+
Alternatively, the move from testing is so common we have helper scripts:
  $EDITOR PKGBUILD
 
  makechrootpkg
 
  
It is '''mandatory''' to build your package using a clean [[DeveloperWiki:Building_in_a_Clean_Chroot|chroot]]
+
  /packages/testing2x openssh bzip2 coreutils
 +
  /packages/testing2x64 openssh bzip2 coreutils
  
=== Run namcap on both PKGBUILD and package ===
+
These scripts only work if the packages on the commandline are either in ''core'' or ''extra''.
  namcap PKGBUILD
+
If a package is only in testing, you have to use ''testing2core'', ''testing2core64'', ''testing2extra'' or ''testing2extra64''.
  namcap some-package-1.0-1-i686.pkg.tar.gz
 
=== Use devtools to upload and commit ===
 
This is repo dependent. For 'extra', you use 'extrapkg'. 'testingpkg' for 'testing', etc
 
  extrapkg "A commit message"
 
=== Update the repository ===
 
Use 'db-update'. It will find new packages for any repository and it manages both i686 and x86_64 architectures at once, if present. For example:
 
  ssh repos.archlinux.org
 
  /packages/db-update
 
  
== Staging Directories ==
+
=== "Tagging" releases ===
  
Staging directories are needed on repos.archlinux.org for uploading of packages. The following structure is NOT automatically created. You must do it yourself:
+
Fetch the package dir using {{ic|archco}} or {{ic|communityco}} or from an svn checkout.
 +
Then:
 +
  cd package-name/trunk
 +
  archrelease extra-i686
  
    ~/staging/
+
This makes an svn copy of the trunk entries in a directory named {{ic|extra-i686}} indicating that this package is in the extra repository for the ''i686'' architecture.
    |-- core/
+
This will be done automatically when using tools such as {{ic|extrapkg}} (see [[DeveloperWiki:HOWTO_Be_A_Packager#Use_devtools_to_sign.2C_upload_and_commit|Use devtools to sign, upload and commit]]).
    |-- extra/
 
    `-- testing/
 
  
These directories are searched by the db scripts to find new packages and those slated for removal.
 
  
 
== Miscellaneous Stuff ==
 
== Miscellaneous Stuff ==
Line 130: Line 191:
 
=== SVN $Id$ tags ===
 
=== SVN $Id$ tags ===
  
$Id$ tags are a nice helper for PKGBUILDs and should be added to the top of all PKGBUILDs in a comment.
+
{{ic|$Id$}} tags are a nice helper for PKGBUILDs and should be added to the top of all PKGBUILDs in a comment.
 
However, svn needs an additional push to know that it should modify this line on checkout.
 
However, svn needs an additional push to know that it should modify this line on checkout.
  
 
   svn propset svn:keywords "Id" my-package/trunk/PKGBUILD
 
   svn propset svn:keywords "Id" my-package/trunk/PKGBUILD
  
=== Package checking tools ===
+
=== Cleaning up your checkout ===
  
==== namcap ====
+
Since you are now maintaining a non-recursive checkout, you may want to get rid of packages that you are no longer tracking:
Run on both your PKGBUILD and package to check for common packaging problems.
 
  
==== checkpkg ====
+
  svn update package1 package2 --set-depth exclude
Run (as root) in the directory with your freshly built package to get a file list diff compared with the package version currently in the repos.
 
  
=== Commit messages ===
+
Or if you want an empty toplevel again:
  
Please try to write concise commit messages. If the package is simply an upstream change, that is fine, but if anything more complex changes, please inform us by writing an appropriate commit message.
+
  svn update --set-depth empty
  
 
=== Avoid having to enter your password all the time ===
 
=== Avoid having to enter your password all the time ===
  
When working with ''extrapkg'' and the other devtools, quite a few ssh connections are established, even when using ssh keys and the ssh agent. You can work around that.
+
When working with {{ic|extrapkg}} and the other devtools, quite a few ssh connections are established, even when using ssh keys and the ssh agent.
 +
You can work around that.
  
Add this to your ''$HOME/.ssh/config'':
+
Add this to your {{ic|~/.ssh/config}}:
  
ControlPath /home/<your username>/.ssh/master-%h-%p-%r
+
  ControlPath ~/.ssh/master-%h-%p-%r
+
 
Host repos.archlinux.org
+
  Host repos.archlinux.org
  
 
Now, before you start working, open a ssh session with
 
Now, before you start working, open a ssh session with
  
ssh -M repos.archlinux.org
+
  ssh -M repos.archlinux.org
  
Enter your password and leave that session open until you are finished. All ssh sessions (including scp and svn+ssh) will now be tunneled through this connection.
+
Enter your password and leave that session open until you are finished.
 +
All ssh sessions (including scp and svn+ssh) will now be tunneled through this connection.

Revision as of 09:00, 15 September 2017

Follow Package Guidelines

Package guidelines can be found in the Arch Linux documentation. Please follow them closely.

Arch packaging standards


Preparation and Setup

Installing the Packages

Make sure you have the packages devtools and namcap installed.

SSH Config

If you have multiple SSH keys in your SSH Agent, you will need to make sure that the correct key is being used to contact the Arch servers. Also, when your local username differs from the one being used on the Arch servers, you need to take care of that too.

Example ~/.ssh/config excerpt:

host repos.archlinux.org
	hostname repos.archlinux.org
	port 22
	user foobar
	IdentityFile ~/.ssh/id_arch
	IdentitiesOnly yes

makepkg Config

Make sure to configure your ~/.makepkg.conf with the correct PACKAGER and GPGKEY variables. Wrong signatures or missing PACKAGER will prevent your packages from entering the repository.

PACKAGER="Foo Bar <foo.bar@archlinux.org>"
GPGKEY="0x0123456789abcdef"

Local SVN Setup using Non-Recursive Checkouts

As SVN provides the ability to do scoped checkouts you can initialize an empty local checkout and later on only fetch the packages that you want. To setup the local checkouts run the following commands.

For core, extra, testing and staging repos:

  svn checkout -N svn+ssh://svn-packages@repos.archlinux.org/srv/repos/svn-packages/svn svn-packages

For community, community-testing, community-staging, multilib, multilib-testing, multilib-staging:

  svn checkout -N svn+ssh://svn-community@repos.archlinux.org/srv/repos/svn-community/svn svn-community

This creates two directories named svn-packages and svn-community which contain nothing but are properly setup as svn repositories.

Helper Scripts (optional)

ch

This shell script eases setting up and maintaining the chroots for building the packages immensely. The script has been developed by Bluewind and can be found here: ch.

As this script relies on Btrfs, you have to create a Btrfs volume (20GiB is mostly sufficient), which can either be a file, a logical volume or a dedicated partition. Furthermore by default this script assumes that the Btrfs for the chroots is mounted at /mnt/chroots/arch.

Afterwards you can create a 64bit package using:

  ch build 64

(automatically and conveniently invokes makechrootpkg with all required arguments)

For further details please take a look at the head of the script, it provides some explanations and usage examples.


The Workflow

Checkout/update your Local Repository

  cd svn-community
  svn update

Adding a new Package

This step is only required when adding a new package to the repository for the first time.

  cd svn-community
  mkdir -p new-package/{repos,trunk}
  cd new-package
  $EDITOR trunk/PKGBUILD
  svn add .
  svn propset svn:keywords "Id" trunk/PKGBUILD
  svn commit

Updating the needed Package

  svn update some-package

Change and build

  cd some-package/trunk/
  $EDITOR PKGBUILD
  extra-x86_64-build

It is mandatory to build your package using a clean chroot.

Or, if you are using the ch helper, simply do:

  cd some-package/trunk/
  $EDITOR PKGBUILD
  ch clean 64
  ch update 64
  ch build 64

Run namcap on both PKGBUILD and Package

  namcap PKGBUILD
  namcap some-package-1.0-1-x86_64.pkg.tar.xz

Run checkpkg on the Package

Run in the directory with your freshly built package to get a file list diff compared with the package version currently in the repos. This can be skipped when adding a new package to the repository for the first time (e.g. by importing it from AUR to community).

Use devtools to sign, upload and commit

Once you are satisfied with the package, you need to make sure all your changes are tracked in the repository. Run:

  svn status

in the some-package/trunk/ directory and check the output carefully. If, for example, you have added a new some-package.install file, you need to tell svn to track that file, e.g.:

  svn add some-package.install

Make sure to never add the binary packages, makepkg logs, etc. to the repository!

When you're ready to proceed, you can run the devtools script to sign, upload and commit your work. This is repo dependent. For extra you use extrapkg, communitypkg for community, etc.

  extrapkg "update to 1.2.3, add post_upgrade hook to fix permissions"

Please try to write concise commit messages. If the package is simply an upstream change, that is fine, but if anything more complex changes, please inform us by writing an appropriate commit message.

Update the Repository

Use db-update. It will find new packages for any repository and it manages both i686 and x86_64 architectures at once, if present.

For example:

  ssh repos.archlinux.org "/packages/db-update"

or:

  ssh repos.archlinux.org "/community/db-update"

Other Operations

Removing a Package

  ssh repos.archlinux.org "/packages/db-remove repo-name arch packagename"

i.e.:

  ssh repos.archlinux.org "/packages/db-remove core i686 openssh"

And if you want to really kill the package, you will need to svn rm the entire package directory after the above steps and commit the deletion.

Sometime the previous command yields:

   svn: E155035: "'/path/to/pkg/<PKG>' is the root of a working copy and cannot be deleted"

You can remotely remove it with:

   svn rm svn+ssh://svn-packages@repos.archlinux.org/srv/repos/svn-packages/svn/<PKG>

Moving a package between repos

  ssh repos.archlinux.org "/packages/db-move fromrepo torepo packagename"

i.e.:

  ssh repos.archlinux.org "/packages/db-move testing core openssh"

Alternatively, the move from testing is so common we have helper scripts:

  /packages/testing2x openssh bzip2 coreutils
  /packages/testing2x64 openssh bzip2 coreutils

These scripts only work if the packages on the commandline are either in core or extra. If a package is only in testing, you have to use testing2core, testing2core64, testing2extra or testing2extra64.

"Tagging" releases

Fetch the package dir using archco or communityco or from an svn checkout. Then:

  cd package-name/trunk
  archrelease extra-i686

This makes an svn copy of the trunk entries in a directory named extra-i686 indicating that this package is in the extra repository for the i686 architecture. This will be done automatically when using tools such as extrapkg (see Use devtools to sign, upload and commit).


Miscellaneous Stuff

SVN $Id$ tags

$Id$ tags are a nice helper for PKGBUILDs and should be added to the top of all PKGBUILDs in a comment. However, svn needs an additional push to know that it should modify this line on checkout.

  svn propset svn:keywords "Id" my-package/trunk/PKGBUILD

Cleaning up your checkout

Since you are now maintaining a non-recursive checkout, you may want to get rid of packages that you are no longer tracking:

  svn update package1 package2 --set-depth exclude

Or if you want an empty toplevel again:

  svn update --set-depth empty

Avoid having to enter your password all the time

When working with extrapkg and the other devtools, quite a few ssh connections are established, even when using ssh keys and the ssh agent. You can work around that.

Add this to your ~/.ssh/config:

  ControlPath ~/.ssh/master-%h-%p-%r
  
  Host repos.archlinux.org

Now, before you start working, open a ssh session with

  ssh -M repos.archlinux.org

Enter your password and leave that session open until you are finished. All ssh sessions (including scp and svn+ssh) will now be tunneled through this connection.