Difference between revisions of "Python package guidelines"

From ArchWiki
Jump to navigation Jump to search
m (sentence case in headings)
(Better URLs for non-wheel-packages)
(Tag: Undo)
 
(83 intermediate revisions by 26 users not shown)
Line 1: Line 1:
[[Category:Package development]]
+
[[Category:Arch package guidelines]]
[[it:Python Package Guidelines]]
+
[[it:Python package guidelines]]
{{Package Guidelines}}
+
[[ja:Python パッケージガイドライン]]
 +
[[pt:Python package guidelines]]
 +
[[zh-hans:Python package guidelines]]
 +
{{Package guidelines}}
 +
This document covers standards and guidelines on writing [[PKGBUILD]]s for [[Python]] software.
  
Writing [[PKGBUILD]]s for software written in [[Python]].
+
== Package naming ==
 +
 
 +
For [[Python#Python 3|Python 3]] library modules, use {{ic|python-''modulename''}}. Also use the prefix if the package provides a program that is strongly coupled to the Python ecosystem (e.g. ''pip'' or ''tox''). For other applications, use only the program name.
 +
 
 +
The same applies to Python 2 except that the prefix (if needed) is {{Ic|python2-}}.
 +
 
 +
{{Note|The package name should be entirely lowercase.}}
 +
 
 +
== Architecture ==
 +
 
 +
See [[PKGBUILD#arch]].
 +
 
 +
A Python package that contains C extensions using the {{ic|ext_modules}} keyword in {{ic|setup.py}}, is architecture-dependent. Otherwise it is most likely architecture-independent.
 +
 
 +
== Source ==
 +
 
 +
Download URLs linked from the PyPI website include an unpredictable hash that needs to be fetched from the PyPI website each time a package must be updated. This makes them unsuitable for use in a PKGBUILD. PyPI [https://github.com/pypa/pypi-legacy/issues/438#issuecomment-226940730 provides] an alternative stable scheme: [[PKGBUILD#source]] {{ic|1=source=()}} array should use the following URL templates:
 +
 
 +
;Source package:
 +
:{{ic|<nowiki>https://files.pythonhosted.org/packages/source/${_name::1}/$_name/$_name-$pkgver.tar.gz</nowiki>}}
 +
;Pure Python wheel package
 +
:{{ic|<nowiki>https://files.pythonhosted.org/packages/py2.py3/${_name::1}/$_name/${_name/-/_}-$pkgver-py2.py3-none-any.whl</nowiki>}} (Bilingual – Python 2 and Python 3 compatible)
 +
:{{ic|<nowiki>https://files.pythonhosted.org/packages/py3/${_name::1}/$_name/${_name/-/_}-$pkgver-py3-none-any.whl</nowiki>}} (Python 3 only)
 +
:Note that the distribution name can contain dashes, while its representation in a wheel filename cannot (they are converted to underscores).
 +
;Arch specific wheel package
 +
:in this example for {{ic|1=source_x86_64=('...')}}. Also {{ic|1=_py=py37}} can be used to not repeat the python version:
 +
:{{ic|<nowiki>https://files.pythonhosted.org/packages/$_py/${_name::1}/$_name/${_name/-/_}-$pkgver-$_py-${_py}m-manylinux1_x86_64.whl</nowiki>}}
 +
 
 +
Note that a custom {{ic|'''_name'''}} variable is used instead of {{ic|pkgname}} since python packages are generally prefixed with {{ic|python-}}. This variable can generically be defined as follows:
 +
_name=${pkgname#python-}
 +
 
 +
== Installation methods ==
 +
 
 +
Python packages are generally installed using language-specific tools, such as [https://pip.pypa.io/ pip] or [https://setuptools.readthedocs.io/en/latest/easy_install.html easy_install], which are comparable to dedicated package managers in that they are designed to fetch the source files from an online repository (usually [https://pypi.org/ PyPI], the Python Package Index) and track the relevant files (for a detailed comparison between the two, see [https://packaging.python.org/pip_easy_install/#pip-vs-easy-install pip vs easy_install]).
 +
 
 +
However, for managing Python packages from within PKGBUILDs, the standard-provided [http://docs.python.org/library/distutils.html distutils] proves to be the most convenient solution since it uses the downloaded source package's {{ic|setup.py}} and easily installs files under {{ic|''$pkgdir''/usr/lib/python''<python version>''/site-packages/''$pkgname''}} directory.
 +
 
 +
=== distutils ===
 +
 
 +
A ''distutils'' PKGBUILD is usually quite simple:
  
== Package naming ==
+
{{bc|1=
For libraries, use {{Ic|python-''modulename''}}. For applications, use the program name. In either case, the package name should be entirely lowercase.
+
build() {
 +
    ''python'' setup.py build
 +
}
 +
 
 +
package() {
 +
    ''python'' setup.py install --root="$pkgdir/" --optimize=1 --skip-build
 +
}
 +
}}
 +
 
 +
where:
 +
 
 +
* ''python'' is replaced with the proper binary, {{ic|python}} or {{ic|python2}}
 +
* {{ic|1=--root="$pkgdir/" }} prevents trying to directly install in the host system instead of inside the package file, which would result in a permission error
 +
* {{ic|1=--optimize=1}} compiles optimized bytecode files ({{ic|.pyo}} for Python 2, {{ic|opt-1.pyc}} for Python 3) so they can be tracked by [[pacman]].
 +
* {{ic|1=--skip-build}} optimizes away the unnecessary attempt to re-run the build steps already run in the {{ic|build()}} function.
 +
 
 +
=== setuptools ===
 +
 
 +
The Python packaging scene has largely migrated from ''distutils'' to ''setuptools'', which is actively developed and functions as a drop-in replacement import in {{ic|setup.py}}. The main difference for packagers is that ''setuptools'' is packaged separately from Python itself, and must be specified as a {{ic|makedepends}}.
 +
 
 +
If the resulting package includes executables which [https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation import the {{ic|pkg_resources}} module], then ''setuptools'' must be additionally specified as a {{ic|depends}} in the split {{ic|package_*()}} functions; alternatively, if the PKGBUILD only installs the Python package for a single version of Python, ''setuptools'' should be moved from {{ic|makedepends}} to {{ic|depends}}.
 +
 
 +
=== pip ===
 +
 
 +
If you need to use ''pip'' (provided by {{Pkg|python-pip}} and {{Pkg|python2-pip}}), ''e.g.'' for installing [https://github.com/pypa/wheel wheel] packages, remember to pass the following flags:
 +
 
 +
PIP_CONFIG_FILE=/dev/null pip install --isolated --root="$pkgdir" --ignore-installed --no-deps *.whl
 +
 
 +
* {{ic|PIP_CONFIG_FILE&#61;/dev/null}} ignores {{ic|{/etc,~/.config}/pip.conf}} that may be appending arbitrary flags to '''pip'''.
 +
* {{ic|--isolated}} ignores environment variables (and again {{ic|{/etc,~/.config}/pip/pip.conf}}) that may otherwise also be appending arbitrary flags to '''pip'''.
 +
* {{ic|--ignore-installed}} is necessary until https://github.com/pypa/pip/issues/3063 is resolved (otherwise '''pip''' skips the install in the presence of an earlier {{ic|--user}} install).
 +
* {{ic|--no-deps}} ensures, that dependencies do not get packaged together with the main package.
 +
 
 +
''pip'' doesn't know how to generate {{ic|.pyo}} files (see https://github.com/pypa/pip/issues/2209). In order to generate them manually after ''pip'' has installed the module, run:
 +
 
 +
python -O -m compileall "${pkgdir}/path/to/module"
 +
 
 +
{{Warning|Use of ''pip'' and/or wheel packages is discouraged in favor of setuptools source packages, and should only be used when the latter is not a viable option (for example, packages which '''only''' come with wheel sources, and therefore cannot be installed using setuptools).}}
 +
 
 +
=== Build-time 2to3 translation ===
 +
 
 +
Most Python projects target either Python 2 or Python 3, or target both using compatibility layers like [https://pythonhosted.org/six/ six]. However, some use the deprecated 2to3 keyword in setuptools to heuristically convert the source code from Python 2 to Python 3 at build time. As a result, the same source directories cannot be used to build both Python 2 and Python 3 split packages.
 +
 
 +
For packages that do this, we need a [[Creating_packages#prepare.28.29|prepare()]] function that copies the source before it is built. Then the Python 2 and Python 3 packages can be converted and built independently without overriding each other.
 +
 
 +
{{bc|1=
 +
makedepends=("python-setuptools" "python2-setuptools")
 +
 
 +
prepare() {
 +
    cp -a foo-$pkgver{,-py2}
 +
}
 +
 
 +
build() {
 +
    cd "$srcdir/foo-$pkgver"
 +
    python setup.py build
 +
 
 +
    cd "$srcdir/foo-$pkgver-py2"
 +
    python2 setup.py build
 +
}
 +
 
 +
package_python-foo() {
 +
    depends=("python")
 +
    cd "$srcdir/foo-$pkgver"
 +
    python setup.py install --root="$pkgdir/" --optimize=1 --skip-build
 +
}
 +
 
 +
package_python2-foo() {
 +
    depends=("python2")
 +
    cd "$srcdir/foo-$pkgver-py2"
 +
    python2 setup.py install --root="$pkgdir/" --optimize=1 --skip-build
 +
}
 +
}}
 +
 
 +
== Check ==
 +
 
 +
{{Warning|Avoid using {{ic|tox}} to run testsuites as it is explicitly designed to test repeatable configurations downloaded from PyPI while {{ic|tox}} is running, and does '''not''' test the version that will be installed by the package. This defeats the purpose of having a ''check'' function at all.}}
 +
 
 +
Most python projects providing a testsuite use nosetests or pytest to run tests with {{ic|test}} in the name of the file or directory containing the testsuite. In general, simply running {{ic|nosetests}} or {{ic|pytest}} is enough to run the testsuite.
 +
 
 +
{{bc|
 +
check(){
 +
    cd "$srcdir/foo-$pkgver"
 +
 
 +
    # For nosetests
 +
    nosetests
 +
 
 +
    # For pytest
 +
    pytest
 +
}
 +
}}
 +
 
 +
If there is a compiled C extension, the tests need to be run using a {{ic|$PYTHONPATH}} hack in order to find and load it.
 +
 
 +
{{bc|1=
 +
check(){
 +
    cd "$srcdir/foo-$pkgver"
 +
 
 +
    # For nosetests
 +
    PYTHONPATH="$PWD/build/lib.linux-$CARCH-3.7" nosetests
 +
 
 +
    # For pytest
 +
    PYTHONPATH="$PWD/build/lib.linux-$CARCH-3.7" pytest
 +
}
 +
}}
  
Python 2 libraries should instead be named {{Ic|python2-''modulename''}}.
+
Some projects provide {{ic|setup.py}} entry points for running the test. This works for both {{ic|pytest}} and {{ic|nosetests}}.
  
== File placement ==
+
{{bc|
Most Python packages are installed with the [http://docs.python.org/library/distutils.html distutils] system using '''setup.py''', which installs files under {{Ic|/usr/lib/python''<python version>''/site-packages/''pkgname''}} directory.
+
check(){
 +
    cd "$srcdir/foo-$pkgver"
  
== Notes ==
+
    # For nosetests
The {{Ic|1=--optimize=1}} parameter compiles {{Ic|.pyo}} files so they can be tracked by [[pacman]].
+
    python setup.py nosetests
  
In most cases, you should put {{Ic|any}} in the {{Ic|arch}} array since most Python packages are architecture independent.
+
    # For pytest - needs python-pytest-runner
 +
    python setup.py pytest
 +
}
 +
}}
  
Please do not install a directory named just {{Ic|tests}}, as it easily conflicts with other Python packages (for example, {{Ic|/usr/lib/python2.7/site-packages/tests/}}).
+
== Notes ==
  
== Example ==
+
Please do not install a directory named just {{ic|tests}}, as it easily conflicts with other Python packages (for example, {{ic|/usr/lib/python2.7/site-packages/tests/}}).
An example PKGBUILD can be found at {{Ic|/usr/share/pacman/PKGBUILD-python.proto}}, which is in the {{Pkg|abs}} package.
 

Latest revision as of 11:42, 11 March 2019

Package creation guidelines

CLRCrossEclipseElectronFree PascalGNOMEGoHaskellJavaKDEKernelLispMinGWNode.jsNonfreeOCamlPerlPHPPythonRRubyRustVCSWebWine

This document covers standards and guidelines on writing PKGBUILDs for Python software.

Package naming

For Python 3 library modules, use python-modulename. Also use the prefix if the package provides a program that is strongly coupled to the Python ecosystem (e.g. pip or tox). For other applications, use only the program name.

The same applies to Python 2 except that the prefix (if needed) is python2-.

Note: The package name should be entirely lowercase.

Architecture

See PKGBUILD#arch.

A Python package that contains C extensions using the ext_modules keyword in setup.py, is architecture-dependent. Otherwise it is most likely architecture-independent.

Source

Download URLs linked from the PyPI website include an unpredictable hash that needs to be fetched from the PyPI website each time a package must be updated. This makes them unsuitable for use in a PKGBUILD. PyPI provides an alternative stable scheme: PKGBUILD#source source=() array should use the following URL templates:

Source package
https://files.pythonhosted.org/packages/source/${_name::1}/$_name/$_name-$pkgver.tar.gz
Pure Python wheel package
https://files.pythonhosted.org/packages/py2.py3/${_name::1}/$_name/${_name/-/_}-$pkgver-py2.py3-none-any.whl (Bilingual – Python 2 and Python 3 compatible)
https://files.pythonhosted.org/packages/py3/${_name::1}/$_name/${_name/-/_}-$pkgver-py3-none-any.whl (Python 3 only)
Note that the distribution name can contain dashes, while its representation in a wheel filename cannot (they are converted to underscores).
Arch specific wheel package
in this example for source_x86_64=('...'). Also _py=py37 can be used to not repeat the python version:
https://files.pythonhosted.org/packages/$_py/${_name::1}/$_name/${_name/-/_}-$pkgver-$_py-${_py}m-manylinux1_x86_64.whl

Note that a custom _name variable is used instead of pkgname since python packages are generally prefixed with python-. This variable can generically be defined as follows:

_name=${pkgname#python-}

Installation methods

Python packages are generally installed using language-specific tools, such as pip or easy_install, which are comparable to dedicated package managers in that they are designed to fetch the source files from an online repository (usually PyPI, the Python Package Index) and track the relevant files (for a detailed comparison between the two, see pip vs easy_install).

However, for managing Python packages from within PKGBUILDs, the standard-provided distutils proves to be the most convenient solution since it uses the downloaded source package's setup.py and easily installs files under $pkgdir/usr/lib/python<python version>/site-packages/$pkgname directory.

distutils

A distutils PKGBUILD is usually quite simple:

build() {
    python setup.py build
}

package() {
    python setup.py install --root="$pkgdir/" --optimize=1 --skip-build
}

where:

  • python is replaced with the proper binary, python or python2
  • --root="$pkgdir/" prevents trying to directly install in the host system instead of inside the package file, which would result in a permission error
  • --optimize=1 compiles optimized bytecode files (.pyo for Python 2, opt-1.pyc for Python 3) so they can be tracked by pacman.
  • --skip-build optimizes away the unnecessary attempt to re-run the build steps already run in the build() function.

setuptools

The Python packaging scene has largely migrated from distutils to setuptools, which is actively developed and functions as a drop-in replacement import in setup.py. The main difference for packagers is that setuptools is packaged separately from Python itself, and must be specified as a makedepends.

If the resulting package includes executables which import the pkg_resources module, then setuptools must be additionally specified as a depends in the split package_*() functions; alternatively, if the PKGBUILD only installs the Python package for a single version of Python, setuptools should be moved from makedepends to depends.

pip

If you need to use pip (provided by python-pip and python2-pip), e.g. for installing wheel packages, remember to pass the following flags:

PIP_CONFIG_FILE=/dev/null pip install --isolated --root="$pkgdir" --ignore-installed --no-deps *.whl
  • PIP_CONFIG_FILE=/dev/null ignores {/etc,~/.config}/pip.conf that may be appending arbitrary flags to pip.
  • --isolated ignores environment variables (and again {/etc,~/.config}/pip/pip.conf) that may otherwise also be appending arbitrary flags to pip.
  • --ignore-installed is necessary until https://github.com/pypa/pip/issues/3063 is resolved (otherwise pip skips the install in the presence of an earlier --user install).
  • --no-deps ensures, that dependencies do not get packaged together with the main package.

pip doesn't know how to generate .pyo files (see https://github.com/pypa/pip/issues/2209). In order to generate them manually after pip has installed the module, run:

python -O -m compileall "${pkgdir}/path/to/module"
Warning: Use of pip and/or wheel packages is discouraged in favor of setuptools source packages, and should only be used when the latter is not a viable option (for example, packages which only come with wheel sources, and therefore cannot be installed using setuptools).

Build-time 2to3 translation

Most Python projects target either Python 2 or Python 3, or target both using compatibility layers like six. However, some use the deprecated 2to3 keyword in setuptools to heuristically convert the source code from Python 2 to Python 3 at build time. As a result, the same source directories cannot be used to build both Python 2 and Python 3 split packages.

For packages that do this, we need a prepare() function that copies the source before it is built. Then the Python 2 and Python 3 packages can be converted and built independently without overriding each other.

makedepends=("python-setuptools" "python2-setuptools")

prepare() {
    cp -a foo-$pkgver{,-py2}
}

build() {
    cd "$srcdir/foo-$pkgver"
    python setup.py build

    cd "$srcdir/foo-$pkgver-py2"
    python2 setup.py build
}

package_python-foo() {
    depends=("python")
    cd "$srcdir/foo-$pkgver"
    python setup.py install --root="$pkgdir/" --optimize=1 --skip-build
}

package_python2-foo() {
    depends=("python2")
    cd "$srcdir/foo-$pkgver-py2"
    python2 setup.py install --root="$pkgdir/" --optimize=1 --skip-build
}

Check

Warning: Avoid using tox to run testsuites as it is explicitly designed to test repeatable configurations downloaded from PyPI while tox is running, and does not test the version that will be installed by the package. This defeats the purpose of having a check function at all.

Most python projects providing a testsuite use nosetests or pytest to run tests with test in the name of the file or directory containing the testsuite. In general, simply running nosetests or pytest is enough to run the testsuite.

check(){
    cd "$srcdir/foo-$pkgver"

    # For nosetests
    nosetests

    # For pytest
    pytest
}

If there is a compiled C extension, the tests need to be run using a $PYTHONPATH hack in order to find and load it.

check(){
    cd "$srcdir/foo-$pkgver"

    # For nosetests
    PYTHONPATH="$PWD/build/lib.linux-$CARCH-3.7" nosetests

    # For pytest
    PYTHONPATH="$PWD/build/lib.linux-$CARCH-3.7" pytest
}

Some projects provide setup.py entry points for running the test. This works for both pytest and nosetests.

check(){
    cd "$srcdir/foo-$pkgver"

    # For nosetests
    python setup.py nosetests

    # For pytest - needs python-pytest-runner
    python setup.py pytest
}

Notes

Please do not install a directory named just tests, as it easily conflicts with other Python packages (for example, /usr/lib/python2.7/site-packages/tests/).