Difference between revisions of "Python package guidelines"

From ArchWiki
Jump to: navigation, search
(pip: warn against its use except when needed)
(Source: new section, move PyPI stuff out of the footnotes as it's quite central to python packaging)
(One intermediate revision by the same user not shown)
Line 15: Line 15:
 
{{Note|The package name should be entirely lowercase.}}
 
{{Note|The package name should be entirely lowercase.}}
  
;Versioned packages
+
== Source ==
If you need to add a versioned package then use {{Ic|python-''modulename''-''version''}}, e.g. {{Ic|python-colorama-0.2.5}}. So python dependency {{Ic|colorama==0.2.5}} will turn into {{Ic|python-colorama-0.2.5}} Arch package.
+
 
 +
PyPI URLs of the form {{ic|<nowiki>https://pypi.python.org/packages/source/${_name:0:1}/${_name}/${_name}-${pkgver}.tar.gz</nowiki>}} were silently abandoned for new package versions in the course of 2016, replaced by a scheme using an unpredictable hash that needs to be fetched from the PyPI website each time a package must be updated[https://github.com/pypa/pypi-legacy/issues/438#issuecomment-226940764].
 +
As downstream packagers voiced their concerns to PyPI maintainers[https://github.com/pypa/pypi-legacy/issues/438], a new stable scheme was provided[https://github.com/pypa/pypi-legacy/issues/438#issuecomment-226940730]: [[PKGBUILD#source]] {{ic|1=source=()}} array should now use the following URL templates:
 +
 
 +
;Source package:
 +
:{{ic|<nowiki>https://files.pythonhosted.org/packages/source/${_name::1}/${_name}/${_name}-${pkgver}.tar.gz</nowiki>}}
 +
;Bilingual wheel package (Python 2 and Python 3 compatible)
 +
:{{ic|<nowiki>https://files.pythonhosted.org/packages/py2.py3/${_name::1}/$_name/$_name-$pkgver-py2.py3-none-any.whl</nowiki>}}
 +
;Arch specific wheel package
 +
:in this example for {{ic|1=source_x86_64=('...')}}. Also {{ic|1=_py=py36}} can be used to not repeat the python version:
 +
:{{ic|<nowiki>https://files.pythonhosted.org/packages/$_py/${_name::1}/$_name/$_name-$pkgver-$_py-${_py}m-manylinux1_x86_64.whl</nowiki>}}
 +
 
 +
Note that a custom {{ic|'''_name'''}} variable is used instead of {{ic|pkgname}} since python packages are generally prefixed with {{ic|python-}}. This variable can generically be defined as follows:
 +
_name=${pkgname#python-}
  
 
== Installation methods ==
 
== Installation methods ==
Line 71: Line 84:
  
 
Please do not install a directory named just {{Ic|tests}}, as it easily conflicts with other Python packages (for example, {{Ic|/usr/lib/python2.7/site-packages/tests/}}).
 
Please do not install a directory named just {{Ic|tests}}, as it easily conflicts with other Python packages (for example, {{Ic|/usr/lib/python2.7/site-packages/tests/}}).
 
=== PyPI download URLs ===
 
 
PyPI URLs of the form {{ic|<nowiki>https://pypi.python.org/packages/source/${_name:0:1}/${_name}/${_name}-${pkgver}.tar.gz</nowiki>}} were silently abandoned for new package versions in the course of 2016, replaced by a scheme using an unpredictable hash that needs to be fetched from the PyPI website each time a package must be updated[https://github.com/pypa/pypi-legacy/issues/438#issuecomment-226940764].
 
As downstream packagers voiced their concerns to PyPI maintainers[https://github.com/pypa/pypi-legacy/issues/438], a new stable scheme was provided[https://github.com/pypa/pypi-legacy/issues/438#issuecomment-226940730]: [[PKGBUILD#source]] {{ic|1=source=()}} array should now use the following URL templates:
 
 
;Source package:
 
:{{ic|<nowiki>https://files.pythonhosted.org/packages/source/${_name::1}/${_name}/${_name}-${pkgver}.tar.gz</nowiki>}}
 
;Bilingual wheel package (Python 2 and Python 3 compatible)
 
:{{ic|<nowiki>https://files.pythonhosted.org/packages/py2.py3/${_name::1}/$_name/$_name-$pkgver-py2.py3-none-any.whl</nowiki>}}
 
;Arch specific wheel package
 
:in this example for {{ic|1=source_x86_64=('...')}}. Also {{ic|1=_py=py36}} can be used to not repeat the python version:
 
:{{ic|<nowiki>https://files.pythonhosted.org/packages/$_py/${_name::1}/$_name/$_name-$pkgver-$_py-${_py}m-manylinux1_x86_64.whl</nowiki>}}
 
 
Note that a custom {{ic|'''_name'''}} variable is used instead of {{ic|pkgname}} since python packages are generally prefixed with {{ic|python-}}. This variable can generically be defined as follows:
 
_name=${pkgname#python-}
 

Revision as of 22:41, 10 June 2018

Package creation guidelines

CLRCrossEclipseFree PascalGNOMEGoHaskellJavaKDEKernelLispMinGWNode.jsNonfreeOCamlPerlPHPPythonRRubyVCSWebWine

This document covers standards and guidelines on writing PKGBUILDs for Python software.

Package naming

For Python 3 library modules, use python-modulename. Also use the prefix if the package provides a program that is strongly coupled to the Python ecosystem (e.g. pip or tox). For other applications, use only the program name.

The same applies to Python 2 except that the prefix (if needed) is python2-.

Note: The package name should be entirely lowercase.

Source

PyPI URLs of the form https://pypi.python.org/packages/source/${_name:0:1}/${_name}/${_name}-${pkgver}.tar.gz were silently abandoned for new package versions in the course of 2016, replaced by a scheme using an unpredictable hash that needs to be fetched from the PyPI website each time a package must be updated[1]. As downstream packagers voiced their concerns to PyPI maintainers[2], a new stable scheme was provided[3]: PKGBUILD#source source=() array should now use the following URL templates:

Source package
https://files.pythonhosted.org/packages/source/${_name::1}/${_name}/${_name}-${pkgver}.tar.gz
Bilingual wheel package (Python 2 and Python 3 compatible)
https://files.pythonhosted.org/packages/py2.py3/${_name::1}/$_name/$_name-$pkgver-py2.py3-none-any.whl
Arch specific wheel package
in this example for source_x86_64=('...'). Also _py=py36 can be used to not repeat the python version:
https://files.pythonhosted.org/packages/$_py/${_name::1}/$_name/$_name-$pkgver-$_py-${_py}m-manylinux1_x86_64.whl

Note that a custom _name variable is used instead of pkgname since python packages are generally prefixed with python-. This variable can generically be defined as follows:

_name=${pkgname#python-}

Installation methods

Python packages are generally installed using language-specific tools, such as pip or easy_install, which are comparable to dedicated package managers in that they are designed to fetch the source files from an online repository (usually PyPI, the Python Package Index) and track the relevant files (for a detailed comparison between the two, see pip vs easy_install).

However, for managing Python packages from within PKGBUILDs, the standard-provided distutils proves to be the most convenient solution since it uses the downloaded source package's setup.py and easily installs files under $pkgdir/usr/lib/python<python version>/site-packages/$pkgname directory.

distutils

A distutils PKGBUILD is usually quite simple:

build() {
    python setup.py build
}

package() {
    python setup.py install --root="$pkgdir/" --optimize=1 --skip-build
}

where:

  • python is replaced with the proper binary, python or python2
  • --root="$pkgdir/" prevents trying to directly install in the host system instead of inside the package file, which would result in a permission error
  • --optimize=1 compiles optimized bytecode files (.pyo for Python 2, opt-1.pyc for Python 3) so they can be tracked by pacman.
  • --skip-build optimizes away the unnecessary attempt to re-run the build steps already run in the build() function.

setuptools

The Python packaging scene has largely migrated from distutils to setuptools, which is actively developed and functions as a drop-in replacement import in setup.py. The main difference for packagers is that setuptools is packaged separately from Python itself, and must be specified as a makedepends.

If the resulting package includes executables which import the pkg_resources module, then setuptools must be additionally specified as a depends in the split package_*() functions; alternatively, if the PKGBUILD only installs the Python package for a single version of Python, setuptools should be moved from makedepends to depends.

pip

If you need to use pip (provided by python-pip and python2-pip), e.g. for installing wheel packages, remember to pass the following flags:

PIP_CONFIG_FILE=/dev/null pip install --isolated --root="$pkgdir" --ignore-installed --no-deps *.whl
  • PIP_CONFIG_FILE=/dev/null ignores {/etc,~/.config}/pip.conf that may be appending arbitrary flags to pip.
  • --isolated ignores environment variables (and again {/etc,~/.config}/pip/pip.conf) that may otherwise also be appending arbitrary flags to pip.
  • --ignore-installed is necessary until https://github.com/pypa/pip/issues/3063 is resolved (otherwise pip skips the install in the presence of an earlier --user install).
  • --no-deps ensures, that dependencies do not get packaged together with the main package.

pip doesn't know how to generate .pyo files (see https://github.com/pypa/pip/issues/2209). In order to generate them manually after pip has installed the module, run:

python -O -m compileall "${pkgdir}/path/to/module"
Warning: Use of pip and/or wheel packages is discouraged in favor of setuptools source packages, and should only be used when the latter is not a viable option (for example, packages which only come with wheel sources, and therefore cannot be installed using setuptools).

Notes

In most cases, you should put any in the arch array since most Python packages are architecture independent.

Please do not install a directory named just tests, as it easily conflicts with other Python packages (for example, /usr/lib/python2.7/site-packages/tests/).