Talk:Python package guidelines

tests directory

The article states that one shouldn't install a "tests" directory. My package however, after running through setup.py, contains such a directory. What is the recommended way to deal with this? Can this directory be deleted? Couldn't find a clue in other Python packages I've looked at, so can someone maybe point me in the right direction? —This unsigned comment is by Johnpatcher (talk) 13h30, 19 December 2016 (UTC). Please sign your posts with ~~~~!

You should probably delete it from "${pkgdir}". I don't know of any reason to have such a directory, unless it contains unittests for the module. Which probably means upstream should be advised to fix their setup.py because there is no reason to run unittests on an installed module, and the module itself certainly shouldn't rely on methods hidden in the unittests...

I don't actually know of any misbehaving modules to point to which violate that rule though.

Eschwartz (talk) 18:52, 19 December 2016 (UTC)

To the best of my knowledge, the recommendation is designed to prevent namespace conflicts. If I install your package and execute python -c 'import tests', and your code is imported, that's bad; If I install your package and execute python -c 'from your_package import tests', and your code is imported, that's fine. Getting this right is a little tricky. For an example of Python code that sidesteps these issues, see this package. The top-level "tests" directory is not — as of this PR — installed by setup.py, and therefore produces no namespace conflicts. The "pulp_smash/tests" directory is installed by setup.py and available as "pulp_smash.tests", and therefore produces no namespace conflicts. Ichimonji10 (talk) 21:56, 19 December 2016 (UTC)

I'm experiencing these issues with python-pendulum and python-pytzdata, both of which seem to install a global tests directory. Not too familiar with Python packaging, so I'm not sure if it is ok to simply delete those directories? --Johnpatcher (talk) 06:54, 20 December 2016 (UTC)

The article states that one shouldn't install a "tests" directory. - unfortunately, this is no longer the case. I think this statement must be added to the article, or at least a discussion on that. I thought for a while about that, and I am really sure that users don't need program tests: they should be done only by developers. -- Ynikitenko (talk) 14:27, 23 June 2022 (UTC)

It is stated in Python package guidelines#Test directory in site-package and the wording did not change substantially since 2016. — Lahwaacz (talk) 15:59, 23 June 2022 (UTC)

No, in that section it is stated: not install a directory named just tests into site-packages. I understand this phrase as "you can install tests into site-packages/mypackage-tests". I think there should be a phrase about any tests (that they need to be installed in very rare cases). -- Ynikitenko (talk) 16:12, 23 June 2022 (UTC)

The wording is basically the same since at least December 2016 which is when this discussion started. As for your wording: there can't be such rule because many official packages would violate it, look at e.g. python-pandas or python-sympy. — Lahwaacz (talk) 17:00, 23 June 2022 (UTC)

Well, I didn't see that revision (I quoted only this talk here). Thanks for the links! I think, since this is an educational article for package creators, it would be an improvement if we could formulate in what cases tests should be present, and it what they should not? For example, as a hypothetical user of python-sympy I could wonder: do I need to have its tests installed? Why is the repository 10Mb large? (is it because of tests or not...) -- Ynikitenko (talk) 17:09, 23 June 2022 (UTC)

The phrase is supposed to be about buggy packages which misconfigure setuptools. Because setuptools configuration is ancient and basically a cargo cult, people copy configurations that result in not only the intended <repository>/<package-name>/ being packaged, but also <repository>/tests/. Many packages have this problem, see e.g. here. I added some clarifying text in revision 736240 – flying sheep 15:33, 3 July 2022 (UTC)

Remove pip section in favor of wheels section?

I can’t come up with any reason to use pip other than installing wheels, and wheels are just as easily installed by

1. Unzipping them into the $pkgdir
2. Creating a little wrapper scripts for all console_scripts it provides (Which can be done manually or using install-wheel-scripts^AUR)

Do you agree? – flying sheep 18:08, 12 March 2019 (UTC)

The relevant specification for installing wheels is PEP 427 which includes a helpful example on how to extract a wheel. There are a few steps missing in install-wheel-scripts^AUR. Pyfisch (talk) 20:30, 24 February 2021 (UTC)

As far as I can tell, I need pip for installing sdists (in my case built for pep517 packages using python-build). That said, pip definitely has its quirks, and a simpler solution would definitely be appreciated.Rmsc (talk) 12:49, 13 April 2021 (UTC)

python-installer now supports our use case with its new CLI, and takes care of more parts than the scripts (e.g. data) – flying sheep 15:52, 19 February 2022 (UTC)

pyproject2setuppy as dephell alternative for flit/poetry

When packaging first flit/poetry packages for Gentoo, I've written pyproject2setuppy to avoid the dependency hell involved in other existing solutions (read: I really didn't want to have to package dephell). Perhaps you'd be interested in using it in place of dephell. It's ~200 lines of Python with the only dependency being toml, so it's much lighter on the packager. When the new build systems become more popular, it may also help you avoid circular dependencies. MGorny (talk) 05:41, 4 May 2020 (UTC)

Is it time to deprecate Python2?

Python2 is now officially not supported by upstream anymore, and Arch will move on as well. I suggest we make clear in this document, that uploading Python2-modules just for the sake of having them is not ok anymore. Necessary dependencies of un-migrated programs should stay around for the time being. But we have moved on with 32bit, and will do so with Python2 at some point as well. I feel it is time to take first small steps in that direction. Fordprefect (talk) 13:48, 20 August 2020 (UTC)

This page no longer mentions packaging Python 2 (used to mention, though), and python2 and the packages that depend on it were moved to AUR. Some AUR packages still depend on Python 2, so I don't think that uploading Python-2 modules to AUR should be prohibited (I assume the OP in 2020 meant uploading to Arch official repositories but those are long gone now, so I mentioned AUR instead). – RafaelFF (talk) 16:51, 28 February 2024 (UTC)

Future of Python packaging in Arch Linux?

Since PEP 517 and PEP 518 are slowly gaining traction and setuptools wants to deprecate and remove setup.py install the current guidelines will need to be updated. Right now there are multiple workarounds including python-dephell. Is there already a plan to migrate to a standards based build process how does it look like? As far as I know both a package builder and a package installer are needed for this if Arch wants to avoid pip. pypa/build and pradyunsg/installer appear to be packages for this task. FFY00 contributes to both these packages and maintains lots of Arch Python packages, so I assume they are interested in using them here? --Pyfisch (talk) 21:06, 24 February 2021 (UTC)

There's already packages in community that use Poetry. python-rich 10.0.1-1

Grawlinson (talk) 00:22, 1 April 2021 (UTC)

Someone added `python-build` instructions to the wiki but I'm interested in us actually discussing the needed changes in an official capacity before randomly adding them to the wiki.

Foxboron (talk) 12:02, 13 April 2021 (UTC)

That was me, sorry. The current guidelines simply don't work for pep517 packages that install scripts. After spending hours trying to get this to work for my PKGBUILDS, I assumed these would be useful right now for this particular niche. So while perhaps a bit rude (sorry..), it wasn't exactly random.Rmsc (talk) 12:37, 13 April 2021 (UTC)

Yes, calling it "random" wasn't nice :) Sorry. It was more a reference to adding it without any discussion then the change itself. I have asked FFY00 if we should draft something or wait. Hopefully something is going to crop up the next week or something.

Foxboron (talk) 12:56, 13 April 2021 (UTC)

python-build is mature, the usage of pip is the issue here, as it's not very nice. I really don't like dephell, as IMO it is essentially a hack and can fail in obscure ways, but other people think it is fine. I would mention the two approaches, python-dephell and python-build/python-pip, for now. Once we have a proper standalone wheel installer that can replace pip, I will submit a RFC proposing to change all packages to python-build/python-installer. Btw, there are some issues with the change. --skip-dependencies does not prevent pip from being used, --no-isolation does that, --skip-dependencies will simply skip the check that looks if the dependencies are met, I don't think it should be used here. We should also be building a wheel, to install with pip, not a sdist. The correct command would be `python -m build --wheel --no-isolation` or `python -m build -wn`. FFY00 (talk) 13:17, 13 April 2021 (UTC)

No problem, my bad :) I've tested the proposed fixes (no --skip-dependencies and --wheel) and it works even better (pip would always build a wheel no matter what). I also dislike having to use pip install, and would rather replace it with a stand-alone installer. If you feel it's appropriate as a temporary thing, I can fix and reintroduce the python-build/python-pip changes.

In the meantime I was also preparing a few changes to the pip section. The issue in https://github.com/pypa/pip/issues/2209 is no longer relevant (was closed in 2017), but the pyo and pyc files generated by pip are still broken (they contain references to $pkgdir). There are a few community packages affected by the problem (tensorflow stuff, for instance). I would like to suggest adding --no-compile to the recommended pip flags, and using 'compileall' instead. If that's ok, I have the change ready to submit.Rmsc (talk) 15:50, 13 April 2021 (UTC)

pip builds wheels in isolation (isolated environment) by default and IIRC it had some issues building without isolation. I agree that passing --no-compile to pip and running compileall afterwards is a correct solution. FFY00 (talk) 17:24, 13 April 2021 (UTC)

As a side note for compileall - since Python 3.9 (Issue 38112: Compileall improvements - Python tracker), compileall supports -s and -p for stripping and prepending paths, so that there will not be $pkgdir in pyc files. The overall affect is similar to what -d does, while more intuitive. Here is an example usage:

python -m compileall -s "$pkgdir" -p / "$pkgdir"/usr/lib

Yan12125 (talk) 07:51, 26 November 2021 (UTC)

I tried to have a quick summary for possible solutions in the below table. Feel free to update it if some information is incorrect/missing/out-dated. --Yan12125 (talk) 08:09, 26 November 2021 (UTC)

Python packaging tools targetting PEP 517/PEP 518

Tools	Pros	Cons
dephell + setuptools		No longer maintained (GitHub project archived) Not actually building packages following PEP 517/PEP 518
pyproject2setuppy + setuptools	Actively maintained	Similar to dephell, not actually building packages following PEP 517/PEP 518
python-build + python-install	python-build is a PyPA project python-install is maintained by an Arch TU	Requires further development; no stable release yet
python-build + python-pip	pip is actively maintained and widely used	many dependencies Does not support stripping $pkgdir from pycs out of box
python-build + python-installer	python-installer is maintained by a PyPA member	No CLI yet https://github.com/pradyunsg/installer/pull/66

So since python-installer now has a CLI, we can use it. I updated the page accordingly, as the last table entry now reads as follows.

This approach will probably be the future. I think it allows for bootstrapping our whole Python toolchain. – flying sheep 15:49, 19 February 2022 (UTC)

According to the packaging guidelines, invocation of `setup.py` directly is deprecated. [1]

It suggests just building a wheel using python -m build

From my testing this works fine. Should a distinction between a project with a `pyproject.toml` and `setup.py` even be made?

[1]: https://packaging.python.org/en/latest/discussions/setup-py-deprecated/#what-commands-should-be-used-instead Levitating (talk) 04:03, 29 January 2024 (UTC)

Python packaging tools targetting PEP 517/PEP 518

Tools	Pros	Cons
python-build + python-installer	Are PyPA projects A lot of design consideration went into both to enable explicitly our use case

Rereading PEP517, explicit invocation of setup.py is unnecessary and deprecated -- it explicitly states: "If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py (either directly, or by implicitly invoking the setuptools.build_meta:__legacy__ backend)."

In other words, it should be safe to recommend using python-build and python-installer everywhere, with the possible caveat that we should recommend falling back to looking in setup.py for dependencies if pyproject.toml doesn't exist.

Indeed, checking eg with papis-git^AUR, despite the project not having been rewritten for PEP517, a PEP517-style build works fine for it.

Gesh (talk) 10:53, 18 February 2024 (UTC)

Pytest downloads dependencies if unmet

I'm currently looking at a bug in python-keyring caused by the Arch packaging shipping a) without specifying a required dependency and b) without the required version of the dependency being in the Arch repos yet. And yet, python-keyring has a check() function and the tests pass. The reason for this is that pytest actually downloads unmet dependencies from PyPI solely for the purposes of running the tests. This kind of defeats some of the purpose of the check() function, as this ought to be an error. I propose changing the recommended 'setup.py pytest' line to:

python setup.py pytest --allow-hosts ','

Which blocks it from downloading anything, turning unmet dependencies into errors. There are probably similar changes than can be made to other test runners. Chrisjbillington (talk) 03:23, 13 March 2021 (UTC)

Thoughts on PEP-610's direct_url.json

PEP-610 has been supported in poetry for a while now.

It creates a direct_url.json file that serves no purpose except to take up space.

Grawlinson (talk) 19:43, 25 August 2021 (UTC)

That is not true. https://www.python.org/dev/peps/pep-0610/#freezing-an-environment particularly stands out. FFY00 (talk)

Add Troubleshooting section?

There are two "If"s from setuptools or distutils section that seem to me as Troublesooting-related contents, so I think the should be moved there.

First "If" is:

______

If a package needs setuptools to be built due to including executables (which is not supported by distutils), but only imports distutils, then building will raise a warning, and the resulting package will be broken (it will not contain the executables):

/usr/lib/python3.8/distutils/dist.py:274: UserWarning: Unknown distribution option: 'entry_points'
  warnings.warn(msg)

An upstream bug should be reported. To work around the problem, an undocumented setuptools feature can be used:

# fails because of distutils
python setup.py build

# works by using a setuptools shim
python -m setuptools.launch setup.py build

______

And the second one is:

______

If a package uses python-setuptools-scm, the package most likely will not build with an error such as:

LookupError: setuptools-scm was unable to detect version for /build/python-jsonschema/src/jsonschema-3.2.0.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

To get it building SETUPTOOLS_SCM_PRETEND_VERSION has to be exported as an environment variable with $pkgver as the value:

export SETUPTOOLS_SCM_PRETEND_VERSION=$pkgver

______

Opinions? -- RafaelFF (talk) 16:33, 28 February 2024 (UTC)

Prefer VCS source for setuptools-scm and friends

I'm proposing to change the notice about python-setuptools-scm, python-hatch-vcs and friends to prefer VCS source instead. These tools actually rely on VCS metadata to determine which files to include & to install.

Passing only the version workarounds the build, but could still silently generates an invalid installation that misses some data files. Even if it's not the case now, it could be anytime in the future.

So here I'm proposing to prefer using VCS sources directly to keep the information and ensure we are packaging correctly. Felixonmars (talk) 17:35, 2 March 2024 (UTC)