Python package guidelines: Difference between revisions
Lahwaacz.bot (talk | contribs) (update interlanguage links) |
Davezerave (talk | contribs) (→Standards based (PEP 517): Add python-flit-core to the list of PEP517 build backends making use of SETUPTOOLS_SCM_PRETEND_VERSION) |
||
(171 intermediate revisions by 54 users not shown) | |||
Line 1: | Line 1: | ||
[[Category: | [[Category:Arch package guidelines]] | ||
[[ja:Python パッケージガイドライン]] | [[ja:Python パッケージガイドライン]] | ||
[[pt:Python package guidelines]] | |||
[[ru:Python package guidelines]] | |||
[[zh-hans:Python package guidelines]] | [[zh-hans:Python package guidelines]] | ||
{{Package guidelines}} | {{Package guidelines}} | ||
Line 8: | Line 9: | ||
== Package naming == | == Package naming == | ||
For Python 3 | For [[Python#Installation|Python 3]] library modules, use {{ic|python-''modulename''}}. Also use the prefix if the package provides a program that is strongly coupled to the Python ecosystem (e.g. ''pip'' or ''tox''). For other applications, use only the program name. | ||
{{Note|The package name should be entirely lowercase.}} | |||
== | == Architecture == | ||
See [[PKGBUILD#arch]]. | |||
A Python package that contains C extensions is architecture-dependent. Otherwise it is most likely architecture-independent. | |||
Packages built using [https://setuptools.pypa.io/ setuptools] define their C extensions using the {{ic|ext_modules}} keyword in {{ic|setup.py}}. | |||
== Source == | |||
{{Note|With [https://rfc.archlinux.page/0020-sources-for-python-packaging/ RFC0020] the default is to use upstream provided source tarballs, instead of PyPI provided sdist tarballs.}} | |||
Download URLs linked from the PyPI website include an unpredictable hash that needs to be fetched from the PyPI website each time a package must be updated. This makes them unsuitable for use in a PKGBUILD. PyPI [https://github.com/pypa/pypi-legacy/issues/438#issuecomment-226940730 provides] an alternative stable scheme: [[PKGBUILD#source]] {{ic|1=source=()}} array should use the following URL templates: | |||
;Source package: | |||
:{{ic|<nowiki>https://files.pythonhosted.org/packages/source/${_name::1}/$_name/$_name-$pkgver.tar.gz</nowiki>}} | |||
;Pure Python wheel package | |||
:{{ic|<nowiki>https://files.pythonhosted.org/packages/py2.py3/${_name::1}/$_name/${_name//-/_}-$pkgver-py2.py3-none-any.whl</nowiki>}} (Bilingual – Python 2 and Python 3 compatible) | |||
:{{ic|<nowiki>https://files.pythonhosted.org/packages/py3/${_name::1}/$_name/${_name//-/_}-$pkgver-py3-none-any.whl</nowiki>}} (Python 3 only) | |||
:Note that the distribution name can contain dashes, while its representation in a wheel filename cannot (they are converted to underscores). | |||
;Architecture specific wheel package | |||
:Additional architecture-specific arrays can be added by appending an underscore and the architecture name, e.g. {{ic|1=source_x86_64=('...')}}. Also {{ic|1=_py=cp310}} can be used to not repeat the Python version: | |||
:{{ic|<nowiki>https://files.pythonhosted.org/packages/$_py/${_name::1}/$_name/${_name//-/_}-$pkgver-$_py-${_py}m-manylinux1_x86_64.whl</nowiki>}} | |||
Note that a custom {{ic|'''_name'''}} variable is used instead of {{ic|pkgname}} since Python packages are generally prefixed with {{ic|python-}}. This variable can generically be defined as follows: | |||
_name=${pkgname#python-} | |||
== Installation methods == | == Installation methods == | ||
Python packages are generally installed using language-specific | Python packages are generally installed using language-specific package manager such as [https://pip.pypa.io/ pip], which fetches packages from an online repository (usually [https://pypi.org/ PyPI], the Python Package Index) and tracks the relevant files. | ||
However, for managing Python packages from within {{ic|PKGBUILD}}s, one needs to "install" the Python package to the temporary location {{ic|''$pkgdir''/usr/lib/python''<Python version>''/site-packages/''$pkgname''}}. | |||
For Python packages using [https://www.python.org/dev/peps/pep-0517/ standard metadata] to specify their build backend in {{ic|pyproject.toml}}, this can most easily achieved using {{pkg|python-build}} and {{pkg|python-installer}}. | |||
Old packages might fail to specify that they use setuptools, and only offer a {{ic|setup.py}} that has to be invoked manually. | |||
{{Note|Dependencies from the package's metadata must be defined in the {{ic|depends}} array otherwise they will not be installed.}} | |||
=== Standards based (PEP 517) === | |||
{{Tip| When building from upstream provided source tarballs and upstream relies on git to derive a version string for the project, it is required to set tooling specific environment variables to {{ic|$pkgver}} before building a wheel: | |||
* {{pkg|python-flit-core}}, {{pkg|python-hatch-vcs}} and {{pkg|python-setuptools-scm}}: {{ic|SETUPTOOLS_SCM_PRETEND_VERSION}} | |||
* {{pkg|python-pbr}}: {{ic|PBR_VERSION}} | |||
* {{pkg|python-pdm-backend}}: {{ic|PDM_BUILD_SCM_VERSION}} | |||
}} | |||
A standards based workflow is straightforward: Build a wheel using {{pkg|python-build}} and install it to {{ic|$pkgdir}} using {{pkg|python-installer}}: | |||
{{bc|1= | |||
makedepends=(python-build python-installer python-wheel) | |||
build() { | |||
cd $_name-$pkgver | |||
python -m build --wheel --no-isolation | |||
} | |||
package() { | |||
cd $_name-$pkgver | |||
python -m installer --destdir="$pkgdir" dist/*.whl | |||
} | |||
}} | |||
where: | |||
* {{ic|1=--wheel}} results in only a wheel file to be built, no source distribution. | |||
* {{ic|1=--no-isolation}} means that the package is built using what is installed on your system (which includes packages you specified in {{ic|depends}}), by default the tool creates an isolated virtual environment and performs the build there. | |||
* {{ic|1=--destdir="$pkgdir"}} prevents trying to directly install in the host system instead of inside the package file, which would result in a permission error | |||
* {{ic|1=--compile-bytecode=...}} or {{ic|1=--no-compile-bytecode}} can be passed to {{ic|installer}}, but the default is sensibly picked, so this should not be necessary. | |||
{{Warning|Skipping {{ic|build}} and putting the {{ic|.whl}} file in your {{ic|source}} array is discouraged in favor of building from source, and should only be used when the latter is not a viable option (for example, packages which '''only''' come with wheel sources, and therefore cannot be built from source).}} | |||
{{Warning|If your package is a [[VCS package guidelines|VCS package]] ({{ic|1=python-…-git}}), include the command {{ic|1=git -C "${srcdir}/${pkgname}" clean -dfx}} in your {{ic|1=prepare}} function. This removes stale wheels along with other build artifacts, and helps prevent issues further down the road. See also upstream issues for [https://github.com/pypa/setuptools/issues/1347 setuptools] and [https://github.com/python-poetry/poetry/issues/1329 Poetry].}} | |||
=== setuptools or distutils === | |||
If no {{ic|pyproject.toml}} can be found or it fails to contain a {{ic|[build-system]}} table, it means the project is using the old legacy format, which uses a ''setup.py'' file which invokes ''setuptools'' or ''distutils''. Note that while ''distutils'' is included in Python's standardlib, having ''setuptools'' installed means that you use a patched version of ''distutils''. | |||
{{bc|1= | |||
makedepends=('python-setuptools') # unless it only requires distutils | |||
build() { | |||
cd $_name-$pkgver | |||
python setup.py build | |||
} | |||
package() { | |||
cd $_name-$pkgver | |||
python setup.py install --root="$pkgdir" --optimize=1 | |||
} | |||
}} | |||
where: | where: | ||
* | * {{ic|1=--root="$pkgdir"}} works like {{ic|1=--destdir}} above | ||
* {{ic|1=-- | * {{ic|1=--optimize=1}} compiles optimized bytecode files (''.opt-1.pyc'') so they can be tracked by [[pacman]] instead of being created on the host system on demand. | ||
* {{ic|1=-- | * Adding {{ic|1=--skip-build}} optimizes away the unnecessary attempt to re-run the build steps already run in the {{ic|build()}} function, if that is the case. | ||
Some packages try to use ''setuptools'' and fall back to ''distutils'' if ''setuptools'' could not be imported. In this case, ''setuptools'' should be added as a {{ic|makedepends}}, so that the resulting Python metadata is better. | |||
If a package needs ''setuptools'' to be built due to including executables (which is not supported by ''distutils''), but only imports ''distutils'', then building will raise a warning, and the resulting package will be broken (it will not contain the executables): | |||
{{bc|1= | |||
/usr/lib/python3.8/distutils/dist.py:274: UserWarning: Unknown distribution option: 'entry_points' | |||
warnings.warn(msg) | |||
}} | |||
An upstream bug should be reported. To work around the problem, an undocumented ''setuptools'' feature can be used: | |||
{{bc|1= | |||
# fails because of distutils | |||
python setup.py build | |||
# works by using a setuptools shim | |||
python -m setuptools.launch setup.py build | |||
}} | |||
If a package uses {{Pkg|python-setuptools-scm}}, the package most likely will not build with an error such as: | |||
{{bc|1= | |||
LookupError: setuptools-scm was unable to detect version for /build/python-jsonschema/src/jsonschema-3.2.0. | |||
Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work. | |||
}} | |||
To get it building {{ic|SETUPTOOLS_SCM_PRETEND_VERSION}} has to be exported as an environment variable with {{ic|$pkgver}} as the value: | |||
export SETUPTOOLS_SCM_PRETEND_VERSION=$pkgver | |||
== | == Check == | ||
{{Warning|Avoid using {{ic|tox}} to run testsuites as it is explicitly designed to test repeatable configurations downloaded from PyPI while {{ic|tox}} is running, and does '''not''' test the version that will be installed by the package. This defeats the purpose of having a ''check'' function at all.}} | |||
Most Python projects providing a testsuite use nosetests or pytest (provided by {{pkg|python-nose}} and {{pkg|python-pytest}}, respectively) to run tests with {{ic|test}} in the name of the file or directory containing the testsuite. In general, simply running {{ic|nosetests}} or {{ic|pytest}} is enough to run the testsuite. | |||
{{bc| | |||
check(){ | |||
cd $_name-$pkgver | |||
# For nosetests | |||
nosetests | |||
# For pytest | |||
pytest | |||
} | |||
}} | |||
If there is a compiled C extension, the tests need to be run using a {{ic|$PYTHONPATH}}, that reflects the current major and minor version of Python in order to find and load it. | |||
{{bc|1= | |||
check(){ | |||
cd $_name-$pkgver | |||
local python_version=$(python -c 'import sys; print("".join(map(str, sys.version_info[:2])))') | |||
# For nosetests | |||
PYTHONPATH="$PWD/build/lib.linux-$CARCH-cpython-$python_version" nosetests | |||
# For pytest | |||
PYTHONPATH="$PWD/build/lib.linux-$CARCH-cpython-$python_version" pytest | |||
} | |||
}} | |||
Some projects provide {{ic|setup.py}} entry points for running the test. This works for both {{ic|pytest}} and {{ic|nosetests}}. | |||
{{bc| | |||
check(){ | |||
cd $_name-$pkgver | |||
# For nosetests | |||
python setup.py nosetests | |||
# For pytest - needs python-pytest-runner | |||
python setup.py pytest | |||
} | |||
}} | |||
== Tips and tricks == | |||
=== Discovering detached PGP signatures on PyPI === | |||
{{Warning|On [https://blog.pypi.org/posts/2023-05-23-removing-pgp/ 2023-05-23] PyPI removed the functionality to provide detached [[OpenPGP]] signatures.}} | |||
If detached PGP signatures for a given Python sdist tarball exist, they should be used to verify the tarball. However, the signature files do not show up directly in the files download section of any given project on pypi.org. To discover the sdist tarballs and their potential signature files, it is possible to use this service to get an overview per project: https://pypi.debian.net/ | |||
For {{Pkg|python-requests}}, this would be https://pypi.debian.net/requests. | |||
=== Using Python version === | |||
Sometimes during preparing, building, testing or installation it is required to refer to the system's major and minor Python version (e.g. {{ic|3.9}} or {{ic|3.10}}). Do not hardcode this and instead use a call to the Python interpreter to retrieve the information and store it in a local variable: | |||
{{bc|1= | |||
check(){ | |||
local python_version=$(python -c 'import sys; print(".".join(map(str, sys.version_info[:2])))') | |||
... | |||
} | |||
}} | |||
=== Using site-packages === | |||
Sometimes during building, testing or installation it is required to refer to the system's {{ic|site-packages}} directory. Do not hardcode this directory and use a call to the Python interpreter instead to retrieve the path and store it in a local variable: | |||
{{bc|1= | |||
check(){ | |||
local site_packages=$(python -c "import site; print(site.getsitepackages()[0])") | |||
... | |||
} | |||
}} | |||
=== Test directory in site-package === | |||
Make sure to not install a directory named just {{ic|tests/}} directly under {{ic|site-packages/}} (i.e. {{ic|/usr/lib/pythonX.Y/site-packages/tests/}}). Doing so could lead to conflicts between packages. Python package projects using ''setuptools'' are sometimes misconfigured to include the directory containing its tests as a top level Python package. If you encounter this, you can help by filing an issue with the package project and ask them to fix this, e.g. [https://github.com/Lightning-AI/lightning/issues/10335 like this]. |
Latest revision as of 09:10, 12 March 2024
32-bit – CLR – CMake – Cross – DKMS – Eclipse – Electron – Font – Free Pascal – GNOME – Go – Haskell – Java – KDE – Kernel – Lisp – Meson – MinGW – Node.js – Nonfree – OCaml – Perl – PHP – Python – R – Ruby – Rust – Shell – VCS – Web – Wine
This document covers standards and guidelines on writing PKGBUILDs for Python software.
Package naming
For Python 3 library modules, use python-modulename
. Also use the prefix if the package provides a program that is strongly coupled to the Python ecosystem (e.g. pip or tox). For other applications, use only the program name.
Architecture
See PKGBUILD#arch.
A Python package that contains C extensions is architecture-dependent. Otherwise it is most likely architecture-independent.
Packages built using setuptools define their C extensions using the ext_modules
keyword in setup.py
.
Source
Download URLs linked from the PyPI website include an unpredictable hash that needs to be fetched from the PyPI website each time a package must be updated. This makes them unsuitable for use in a PKGBUILD. PyPI provides an alternative stable scheme: PKGBUILD#source source=()
array should use the following URL templates:
- Source package
https://files.pythonhosted.org/packages/source/${_name::1}/$_name/$_name-$pkgver.tar.gz
- Pure Python wheel package
https://files.pythonhosted.org/packages/py2.py3/${_name::1}/$_name/${_name//-/_}-$pkgver-py2.py3-none-any.whl
(Bilingual – Python 2 and Python 3 compatible)https://files.pythonhosted.org/packages/py3/${_name::1}/$_name/${_name//-/_}-$pkgver-py3-none-any.whl
(Python 3 only)- Note that the distribution name can contain dashes, while its representation in a wheel filename cannot (they are converted to underscores).
- Architecture specific wheel package
- Additional architecture-specific arrays can be added by appending an underscore and the architecture name, e.g.
source_x86_64=('...')
. Also_py=cp310
can be used to not repeat the Python version: https://files.pythonhosted.org/packages/$_py/${_name::1}/$_name/${_name//-/_}-$pkgver-$_py-${_py}m-manylinux1_x86_64.whl
Note that a custom _name
variable is used instead of pkgname
since Python packages are generally prefixed with python-
. This variable can generically be defined as follows:
_name=${pkgname#python-}
Installation methods
Python packages are generally installed using language-specific package manager such as pip, which fetches packages from an online repository (usually PyPI, the Python Package Index) and tracks the relevant files.
However, for managing Python packages from within PKGBUILD
s, one needs to "install" the Python package to the temporary location $pkgdir/usr/lib/python<Python version>/site-packages/$pkgname
.
For Python packages using standard metadata to specify their build backend in pyproject.toml
, this can most easily achieved using python-build and python-installer.
Old packages might fail to specify that they use setuptools, and only offer a setup.py
that has to be invoked manually.
depends
array otherwise they will not be installed.Standards based (PEP 517)
$pkgver
before building a wheel:
- python-flit-core, python-hatch-vcs and python-setuptools-scm:
SETUPTOOLS_SCM_PRETEND_VERSION
- python-pbr:
PBR_VERSION
- python-pdm-backend:
PDM_BUILD_SCM_VERSION
A standards based workflow is straightforward: Build a wheel using python-build and install it to $pkgdir
using python-installer:
makedepends=(python-build python-installer python-wheel) build() { cd $_name-$pkgver python -m build --wheel --no-isolation } package() { cd $_name-$pkgver python -m installer --destdir="$pkgdir" dist/*.whl }
where:
--wheel
results in only a wheel file to be built, no source distribution.--no-isolation
means that the package is built using what is installed on your system (which includes packages you specified independs
), by default the tool creates an isolated virtual environment and performs the build there.--destdir="$pkgdir"
prevents trying to directly install in the host system instead of inside the package file, which would result in a permission error--compile-bytecode=...
or--no-compile-bytecode
can be passed toinstaller
, but the default is sensibly picked, so this should not be necessary.
build
and putting the .whl
file in your source
array is discouraged in favor of building from source, and should only be used when the latter is not a viable option (for example, packages which only come with wheel sources, and therefore cannot be built from source).python-…-git
), include the command git -C "${srcdir}/${pkgname}" clean -dfx
in your prepare
function. This removes stale wheels along with other build artifacts, and helps prevent issues further down the road. See also upstream issues for setuptools and Poetry.setuptools or distutils
If no pyproject.toml
can be found or it fails to contain a [build-system]
table, it means the project is using the old legacy format, which uses a setup.py file which invokes setuptools or distutils. Note that while distutils is included in Python's standardlib, having setuptools installed means that you use a patched version of distutils.
makedepends=('python-setuptools') # unless it only requires distutils build() { cd $_name-$pkgver python setup.py build } package() { cd $_name-$pkgver python setup.py install --root="$pkgdir" --optimize=1 }
where:
--root="$pkgdir"
works like--destdir
above--optimize=1
compiles optimized bytecode files (.opt-1.pyc) so they can be tracked by pacman instead of being created on the host system on demand.- Adding
--skip-build
optimizes away the unnecessary attempt to re-run the build steps already run in thebuild()
function, if that is the case.
Some packages try to use setuptools and fall back to distutils if setuptools could not be imported. In this case, setuptools should be added as a makedepends
, so that the resulting Python metadata is better.
If a package needs setuptools to be built due to including executables (which is not supported by distutils), but only imports distutils, then building will raise a warning, and the resulting package will be broken (it will not contain the executables):
/usr/lib/python3.8/distutils/dist.py:274: UserWarning: Unknown distribution option: 'entry_points' warnings.warn(msg)
An upstream bug should be reported. To work around the problem, an undocumented setuptools feature can be used:
# fails because of distutils python setup.py build # works by using a setuptools shim python -m setuptools.launch setup.py build
If a package uses python-setuptools-scm, the package most likely will not build with an error such as:
LookupError: setuptools-scm was unable to detect version for /build/python-jsonschema/src/jsonschema-3.2.0. Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.
To get it building SETUPTOOLS_SCM_PRETEND_VERSION
has to be exported as an environment variable with $pkgver
as the value:
export SETUPTOOLS_SCM_PRETEND_VERSION=$pkgver
Check
tox
to run testsuites as it is explicitly designed to test repeatable configurations downloaded from PyPI while tox
is running, and does not test the version that will be installed by the package. This defeats the purpose of having a check function at all.Most Python projects providing a testsuite use nosetests or pytest (provided by python-nose and python-pytest, respectively) to run tests with test
in the name of the file or directory containing the testsuite. In general, simply running nosetests
or pytest
is enough to run the testsuite.
check(){ cd $_name-$pkgver # For nosetests nosetests # For pytest pytest }
If there is a compiled C extension, the tests need to be run using a $PYTHONPATH
, that reflects the current major and minor version of Python in order to find and load it.
check(){ cd $_name-$pkgver local python_version=$(python -c 'import sys; print("".join(map(str, sys.version_info[:2])))') # For nosetests PYTHONPATH="$PWD/build/lib.linux-$CARCH-cpython-$python_version" nosetests # For pytest PYTHONPATH="$PWD/build/lib.linux-$CARCH-cpython-$python_version" pytest }
Some projects provide setup.py
entry points for running the test. This works for both pytest
and nosetests
.
check(){ cd $_name-$pkgver # For nosetests python setup.py nosetests # For pytest - needs python-pytest-runner python setup.py pytest }
Tips and tricks
Discovering detached PGP signatures on PyPI
If detached PGP signatures for a given Python sdist tarball exist, they should be used to verify the tarball. However, the signature files do not show up directly in the files download section of any given project on pypi.org. To discover the sdist tarballs and their potential signature files, it is possible to use this service to get an overview per project: https://pypi.debian.net/
For python-requests, this would be https://pypi.debian.net/requests.
Using Python version
Sometimes during preparing, building, testing or installation it is required to refer to the system's major and minor Python version (e.g. 3.9
or 3.10
). Do not hardcode this and instead use a call to the Python interpreter to retrieve the information and store it in a local variable:
check(){ local python_version=$(python -c 'import sys; print(".".join(map(str, sys.version_info[:2])))') ... }
Using site-packages
Sometimes during building, testing or installation it is required to refer to the system's site-packages
directory. Do not hardcode this directory and use a call to the Python interpreter instead to retrieve the path and store it in a local variable:
check(){ local site_packages=$(python -c "import site; print(site.getsitepackages()[0])") ... }
Test directory in site-package
Make sure to not install a directory named just tests/
directly under site-packages/
(i.e. /usr/lib/pythonX.Y/site-packages/tests/
). Doing so could lead to conflicts between packages. Python package projects using setuptools are sometimes misconfigured to include the directory containing its tests as a top level Python package. If you encounter this, you can help by filing an issue with the package project and ask them to fix this, e.g. like this.