Talk:Python package guidelines

From ArchWiki
Jump to navigation Jump to search

There is going to be an official policy drafted to deal with .pyc/.pyo files. See: https://wiki.archlinux.org/index.php/User:Allan/Python_Packaging_Policy

tests directory

The article states that one shouldn't install a "tests" directory. My package however, after running through setup.py, contains such a directory. What is the recommended way to deal with this? Can this directory be deleted? Couldn't find a clue in other Python packages I've looked at, so can someone maybe point me in the right direction?

You should probably delete it from "${pkgdir}". I don't know of any reason to have such a directory, unless it contains unittests for the module. Which probably means upstream should be advised to fix their setup.py because there is no reason to run unittests on an installed module, and the module itself certainly shouldn't rely on methods hidden in the unittests...
I don't actually know of any misbehaving modules to point to which violate that rule though.
Eschwartz (talk) 18:52, 19 December 2016 (UTC)
To the best of my knowledge, the recommendation is designed to prevent namespace conflicts. If I install your package and execute python -c 'import tests', and your code is imported, that's bad; If I install your package and execute python -c 'from your_package import tests', and your code is imported, that's fine. Getting this right is a little tricky. For an example of Python code that sidesteps these issues, see this package. The top-level "tests" directory is not — as of this PR — installed by setup.py, and therefore produces no namespace conflicts. The "pulp_smash/tests" directory is installed by setup.py and available as "pulp_smash.tests", and therefore produces no namespace conflicts. Ichimonji10 (talk) 21:56, 19 December 2016 (UTC)
I'm experiencing these issues with python-pendulum and python-pytzdata, both of which seem to install a global tests directory. Not too familiar with Python packaging, so I'm not sure if it is ok to simply delete those directories? --Johnpatcher (talk) 06:54, 20 December 2016 (UTC)

Python version directories

For example debian packaging guidelines dictate that python files go into /usr/lib/pythonX/ and OpenSUSE wants files to go into /usr/lib/pythonX.Y/.

Arch should have a clear guideline where packaged files go into. Most packages have their python files in /usr/lib/pythonX.Y/ but the package openimageio has python files in /usr/lib/python3/.

Z3ntu (talk) 12:35, 24 January 2017 (UTC)

The /usr/lib/python3/ path does not work:
>>> import sys
>>> sys.path
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/lahwaacz/.local/lib/python3.6/site-packages', '/usr/lib/python3.6/site-packages']
Hence it is a bug and should be reported on the bug tracker. -- Lahwaacz (talk) 12:46, 24 January 2017 (UTC)

Remove pip section in favor of wheels section?

I can’t come up with any reason to use pip other than installing wheels, and wheels are just as easily installed by

  1. Unzipping them into the $pkgdir
  2. Creating a little wrapper scripts for all console_scripts it provides (Which can be done manually or using install-wheel-scriptsAUR)

Do you agree? – flying sheep 18:08, 12 March 2019 (UTC)

The relevant specification for installing wheels is PEP 427 which includes a helpful example on how to extract a wheel. There are a few steps missing in install-wheel-scriptsAUR. Pyfisch (talk) 20:30, 24 February 2021 (UTC)
As far as I can tell, I need pip for installing sdists (in my case built for pep517 packages using python-build). That said, pip definitely has its quirks, and a simpler solution would definitely be appreciated.Rmsc (talk) 12:49, 13 April 2021 (UTC)

pyproject2setuppy as dephell alternative for flit/poetry

When packaging first flit/poetry packages for Gentoo, I've written pyproject2setuppy to avoid the dependency hell involved in other existing solutions (read: I really didn't want to have to package dephell). Perhaps you'd be interested in using it in place of dephell. It's ~200 lines of Python with the only dependency being toml, so it's much lighter on the packager. When the new build systems become more popular, it may also help you avoid circular dependencies. MGorny (talk) 05:41, 4 May 2020 (UTC)

Is it time to deprecate Python2?

Python2 is now officially not supported by upstream anymore, and Arch will move on as well. I suggest we make clear in this document, that uploading Python2-modules just for the sake of having them is not ok anymore. Necessary dependencies of un-migrated programs should stay around for the time being. But we have moved on with 32bit, and will do so with Python2 at some point as well. I feel it is time to take first small steps in that direction. Fordprefect (talk) 13:48, 20 August 2020 (UTC)

Future of Python packaging in Arch Linux?

Since PEP 517 and PEP 518 are slowly gaining traction and setuptools wants to deprecate and remove setup.py install the current guidelines will need to be updated. Right now there are multiple workarounds including python-dephell. Is there already a plan to migrate to a standards based build process how does it look like? As far as I know both a package builder and a package installer are needed for this if Arch wants to avoid pip. pypa/build and pradyunsg/installer appear to be packages for this task. FFY00 contributes to both these packages and maintains lots of Arch Python packages, so I assume they are interested in using them here? --Pyfisch (talk) 21:06, 24 February 2021 (UTC)

There's already packages in community that use Poetry. python-rich 10.0.1-1
Grawlinson (talk) 00:22, 1 April 2021 (UTC)
Someone added `python-build` instructions to the wiki but I'm interested in us actually discussing the needed changes in an official capacity before randomly adding them to the wiki.
Foxboron (talk) 12:02, 13 April 2021 (UTC)
That was me, sorry. The current guidelines simply don't work for pep517 packages that install scripts. After spending hours trying to get this to work for my PKGBUILDS, I assumed these would be useful right now for this particular niche. So while perhaps a bit rude (sorry..), it wasn't exactly random.Rmsc (talk) 12:37, 13 April 2021 (UTC)
Yes, calling it "random" wasn't nice :) Sorry. It was more a reference to adding it without any discussion then the change itself. I have asked FFY00 if we should draft something or wait. Hopefully something is going to crop up the next week or something.
Foxboron (talk) 12:56, 13 April 2021 (UTC)
python-build is mature, the usage of pip is the issue here, as it's not very nice. I really don't like dephell, as IMO it is essentially a hack and can fail in obscure ways, but other people think it is fine. I would mention the two approaches, python-dephell and python-build/python-pip, for now. Once we have a proper standalone wheel installer that can replace pip, I will submit a RFC proposing to change all packages to python-build/python-installer. Btw, there are some issues with the change. --skip-dependencies does not prevent pip from being used, --no-isolation does that, --skip-dependencies will simply skip the check that looks if the dependencies are met, I don't think it should be used here. We should also be building a wheel, to install with pip, not a sdist. The correct command would be `python -m build --wheel --no-isolation` or `python -m build -wn`. FFY00 (talk) 13:17, 13 April 2021 (UTC)
No problem, my bad :) I've tested the proposed fixes (no --skip-dependencies and --wheel) and it works even better (pip would always build a wheel no matter what). I also dislike having to use pip install, and would rather replace it with a stand-alone installer. If you feel it's appropriate as a temporary thing, I can fix and reintroduce the python-build/python-pip changes.
In the meantime I was also preparing a few changes to the pip section. The issue in https://github.com/pypa/pip/issues/2209 is no longer relevant (was closed in 2017), but the pyo and pyc files generated by pip are still broken (they contain references to $pkgdir). There are a few community packages affected by the problem (tensorflow stuff, for instance). I would like to suggest adding --no-compile to the recommended pip flags, and using 'compileall' instead. If that's ok, I have the change ready to submit.Rmsc (talk) 15:50, 13 April 2021 (UTC)
pip builds wheels in isolation (isolated environment) by default and IIRC it had some issues building without isolation. I agree that passing --no-compile to pip and running compileall afterwards is a correct solution. FFY00 (talk) 17:24, 13 April 2021 (UTC)

Pytest downloads dependencies if unmet

I'm currently looking at a bug in python-keyring caused by the Arch packaging shipping a) without specifying a required dependency and b) without the required version of the dependency being in the Arch repos yet. And yet, python-keyring has a check() function and the tests pass. The reason for this is that pytest actually downloads unmet dependencies from PyPI solely for the purposes of running the tests. This kind of defeats some of the purpose of the check() function, as this ought to be an error. I propose changing the recommended 'setup.py pytest' line to:

python setup.py pytest --allow-hosts ','

Which blocks it from downloading anything, turning unmet dependencies into errors. There are probably similar changes than can be made to other test runners. Chrisjbillington (talk) 03:23, 13 March 2021 (UTC)

python3- upstream name

Are there any guidelines for when package upstream has name which begins with python3-?

Example is this one https://aur.archlinux.org/packages/python3-saml where the upstream repo is also named python3-saml. Should it be python3- or be renamed to python-?

Cheers. Micwoj92 (talk) 00:04, 3 April 2021 (UTC)