Talk:AUR helpers

From ArchWiki
Revision as of 14:02, 7 November 2018 by Spyhawk (talk | contribs) (→‎remove Batch interaction 2: added clarity)
Jump to navigation Jump to search

"Reference" implementation

This is an alternative to #Reliable Updater. Instead of an arbitrary set of test packages, we could write up a "specification" on what a reliable AUR helper should do. This should also be more helpful for potential AUR helper writers who otherwise have to wade through complex, fully-featured AUR helpers.

I propose a minimal reference implementation with the following points:

  • No client-side workarounds for upstream limitations. In particular, a reference implementation does not need to score full points on split packages, as makepkg --pkg was removed with pacman 5.
  • Minimal language constructs in e.g. a scripting language like dash.
  • Prefer simplicity of implementation over being fully featured. In particular, an implementation may only support git clone and not git diff.

My initial plan was to keep such an implementation in a man page aurhelper(7) (hosted as part of aurutils), but we can consider including on a sub-page of this article. It could be then linked from the comparison table. Thoughts? -- Alad (talk) 13:28, 8 March 2018 (UTC)

Generally agree with the idea, but I don't think there is a way around a set of PKGBUILDs that could be used to test helpers in a local AUR instance. F.e., I wouldn't define a "reliable" helper that doesn't handle split packages well. Since helpers are tolerated rather than supported, upstream limitations of the AUR might be temporary or permanent, meaning the limitation would actually be in the helper itself (f.e. like regex support). Also, I'd use pseudo code for such a reference as the actual implementation itself doesn't matter, unless you'd like to write a new minimalist helper. Spyhawk (talk) 15:26, 8 March 2018 (UTC)
Apart from FS#56602, I can't think of a case where upstream opposed removing limitations, even if helpers directly benefited. cf. the regex support discussed in [1] or the exit codes finally introduced in makepkg 5.1 which made automatic building significantly easier imo. To me it seems that the main reason we have these AUR limations is due to the minimal interest of helper writers in contributing upstream, and upstream itself having different priorities. Not sure why former is the case, the PHP codebase may play part in it - at least it does for me.
You can keep dash close enough to pseudo-code, I guess less so if you want a complete example rather than exemplary code blocks. For the PKGBUILD set, I use this: [2] -- Alad (talk) 18:34, 8 March 2018 (UTC)
My understanding is that changes that aren't invasive will be accepted upstream, but otherwise might be rejected (see [3]). One prominent example that comes to mind is FS#48796. It's not really relevant anymore since x86 has been officially dropped, but the solution would involve duplicating DB tables on the server, which isn't trivial to implement/migrate. Many of the feature requests involve non-trivial code change, which is the main reason nobody pushed patches; I dislike PHP but the language itself isn't too hard either. For regex, see the bottom of [4], which is the follow-up of your link above.
Your testsuite seems interesting (thanks for the link), but one advantage of having a fixed set of packages is that these packages might be updated and change, making these edge cases difficult to test. This happened quite a few times with my own list of test packages in the past and this was rather annoying. Spyhawk (talk) 20:20, 8 March 2018 (UTC)

Expand Secure criteria to include other (non-PKGBUILD) bundled files

[5], in particular [6]

The new criteria would be as follows:

  • PKGBUILD, no other files -> Partial
  • Other subset of files that includes the PKGBUILD -> Partial
  • No PKGBUILD -> No
  • All files in the git repo or tar archive -> Yes

Similar to the Diff view column. -- Alad (talk) 16:32, 4 July 2018 (UTC)

good idea, you also mentioned this for aurman a few months ago, see: https://github.com/polygamma/aurman/issues/25#issuecomment-371971155 really a good idea to implement it in a way, so that changes of all known files are being shown Polygamma (talk) 17:07, 4 July 2018 (UTC)
"All files in the git repo or tar archive -> Yes" What exactly do you mean by all files? Build files often contain non text files such as images. Git diff is smart enough to hide these but then you could consider that partial because not all files are covered.
In my opinion all a helper has to do to be secure it pause and allow the user to read the build files. The helper does not even need to offer to open them for you that's the user's responsibility. Anything more than that is nice to have but not strictly needed. Morganamilo (talk) 20:25, 4 July 2018 (UTC)
If this qualifies as "nice to have", there has to be an explicit warning that a green entry in the "Secure" column does not cover other files, files which may cause more harm than the PKGBUILD itself (such as .install files or exectuables called from the PKGBUILD). In either case it's misleading, since you either give the impression that viewing PKGBUILDs alone is sufficient (with the current criteria), or include a warning that diminguishes the value of the criteria in the first place.
Latter is similar to "Native pacman", in that you have a warning at the article top warning against any sort of pacman wrapping, and criteria in the table that ignore this warning, or even reward behavior which goes against it. -- Alad (talk) 17:07, 8 July 2018 (UTC)
That's a fair point, what about changing the name to "show files before sourcing" or something? Seems more accurate. Then it would make sense that not showing .install files to be partial. The only problem I see that it's not as hard hitting as "secure". Morganamilo (talk) 20:11, 8 July 2018 (UTC)
It cuts both ways: it's an effective deterrent against broken helpers, but it also gives the impression that using a "Secure" helper makes usage of the AUR safe, which it definitely doesn't. I'm not sure on what different name to use, though. -- Alad (talk) 17:25, 14 July 2018 (UTC)
I guess "File view" could work. -- Alad (talk) 17:44, 14 July 2018 (UTC)
The column name was updated to "File review". Are there remaining helpers that only display the PKGBUILD? (trizenAUR springs to mind) -- Alad (talk) 15:30, 23 August 2018 (UTC)

Pacman wrappers Specificity

I wanted to introduce the manual AUR package selection specificity for pacman wrappers. For example yay and pakku only allow -Syu where they update system packages and AUR packages at the same time. pikaur when doing -Syu also updates system packages and AUR packages at the same time but prompt user for manual selection for AUR packages, because for example I want to update to the new version of X program but not the Y lib (AUR) that is used only by Z program (AUR) so until Z is not updated I don't want to update Y. And yaourt when doing -Syu updates only system packages, -Syu --aur is needed for updating AUR packages as well, and a manual selection for AUR packages is available. That's why I wanted to introduce the manual AUR package selection specificity. -- Noraj (talk) 11:16, 9 September 2018 (UTC)

Why just not specify --ignore ylib? It is problem with zprogram if it does not use versioned deps and allows library update that breaks it. Helper should just fail to satisfy AUR deps in correct case, bail out and user should specify --ignore manually explicitly IMO.
Additionally yay asks you what AUR packages to not upgrade that also provides solution to this XY problem.
All-in-all if you can no longer build some package from AUR correctly because of dependency update that package is broken and should be fixed or be removed instead. And helpers should ideally not provide workarounds for problems that broken package introduce.
-- Svito (talk) 13:06, 9 September 2018 (UTC)
It's not really about broken programs. It could just as easily apply to some package that you're sure will compile fine, but you also remember takes 45 minutes to build, which you don't want to spend right now.
Also I really don't consider using versioned deps, to be the correct case. -- Eschwartz (talk) 14:45, 9 September 2018 (UTC)
That's exactly my use case. Whether to allow it as a specificity or not? I'd probably say sure. But I also think there should be a cap on how many specificities an entry may have. I'd say max 4 or 5, otherwise the column may as well be called feature list. Morganamilo (talk) 14:57, 9 September 2018 (UTC)
Text "manual package selection" was only added to pikaur and yaourt so I reverted it, because it is unclear what exactly that label means, why only those 2 support it and was not explained in edit summary. I still don't understand about what original proposal was since it is implied yay and pakku don't support some way of doing it that would qualify. If it is about "skip build selection" or somebody understand what is this about please reopen. -- Svito (talk) 21:11, 6 November 2018 (UTC)

New test cases for dependency resolution

ros-foo-desktop-meta have always been difficult to build, even more so with KDE4 libs moved to AUR (see arch-dev-public). Besides the sheer number of dependencies, they otherwise have little interesting properties either.

I propose to instead use various cross-compilation packages as test cases, e.g. mingw-w64-zlibAUR and powerpc-linux-gnu-gccAUR. These appear very efficient at exposing problems with complex dependency algorithms (see for example [7] or aurman's issues with nsisAUR) and don't take 2 years to build either. -- Alad (talk) 11:34, 14 September 2018 (UTC)

We could also add some simpler cases, like fortune-mod-all-enAUR, and add details similar to the Split packages description. That way, all existing entries with "Yes" in Reliable solver would at minimum have "Partial". -- Alad (talk) 12:01, 14 September 2018 (UTC)
I believe the mingw stuff has a bunch of circular dependencies and a bootstrapping process. Do you think AUR helper's should be expected to handle this? Morganamilo (talk) 15:25, 14 September 2018 (UTC)
No, I don't think so. There will always be cases that can only be dealt reasonably manually, simply because the involved complexity in implementation isn't worth it. See also related sicussion Talk:AUR_helpers#.22Reference.22_implementation above, where a set of fixed reference packages would be better than live packages. No idea how to dealt with than without a local AUR instance though. -Spyhawk (talk) 10:42, 15 September 2018 (UTC)
The mingw packages haven't had cycles in (global) depends for a while. As such, makepkg -r works fine for these packages without manual intervention or "bootstrapping". If helpers fail, it may be because of handling split depends contrary to PKGBUILD(5), or other flaws in their dependency algorithms. -- Alad (talk) 10:05, 18 September 2018 (UTC)

add RPC description (proposal)

The page mentions an "RPC" interface once in the page, inside description of a "reliable parser". I think it would make sense to explain what it is by changing "RPC" text to a link with the following destination: https://aur.archlinux.org/rpc.php Thoughts? Vasya (talk) 08:47, 4 November 2018 (UTC)

please add "aur.rs" to "Libraries" section

aur.rs -- Rust library for accessing aur RPC API

https://github.com/zeyla/aur.rs Vasya (talk) 11:47, 4 November 2018 (UTC)

Possible improvement: merge "libraries" with RPC wiki page

This section https://wiki.archlinux.org/index.php/AUR_helpers#Libraries currently seems to duplicate this one: https://wiki.archlinux.org/index.php/Aurweb_RPC_interface#Associated_code Any ideas how to fix that?

One alternative is to change the AUR_helpers#Libraries section to only contain one link, pointing to RPC libraries, which in turn will have all the current data. Thouhgts?

UPD: I've updated the RPC#Libraries section. If everything's is all right, we only have to clean up the AUR_helpers part now. Vasya (talk) 09:13, 5 November 2018 (UTC)

I guess Aurweb RPC interface is more fitting place for libraries. Instead of having one link in #Libraries section I propose adding short descriptions to first 3 sections explaining what they do or expand upon:
Search and download
These helpers search AUR via Aurweb RPC interface and can download AUR packages as tarballs or via git.
Download and build
In addition to #Search and download these helpers can set up build environment and build packages via makepkg.
Pacman wrappers
In addition to #Download and build these helpers can manage packages from official repositories via pacman.
I would like more opinions from the team.
-- Svito (talk) 14:31, 5 November 2018 (UTC)
I don't know about the categorization, but I've been wondering what a "pacman wrapper" is myself, so at least for me, some small description would definitely be useful Vasya (talk) 15:23, 5 November 2018 (UTC)
While at it I would move package-queryAUR there too, and rename section to Libraries and utilities. Worth noting that python3-aurAUR comes with aurquery and few other utilities. -- Svito (talk) 00:26, 6 November 2018 (UTC)
I'm not fond of these proposals. 1. the line between AUR "libraries" and AUR "helpers" is not clear, as you may see shell utilities as "libraries" (in the case of package-query or aurutils) and "libraries" as shell utilities (in the case of python3-aur). 2. The "addition" in pacman wrappers is completely superficial: it is a matter of syntax and passing commands to pacman. The only emphasis required for this is the existing warning on broken systems. -- Alad (talk) 12:22, 6 November 2018 (UTC)
I draw the line for tools to be generally useful and not just building block for other tool or example how one can implement both Archweb and Aurweb RPC interfaces in one tool and it not being a shared library. To be clear I do not consider package-query a library, what I am implying that it provides an piece of code that does half of what you might expect of AUR helper. Technically it is complete AUR helper when you consider it can download PKGBUILD to immediately discard it. [8]
aurutils is not even comparable because despite being modular each part is not packaged separately or requires you to run it from specific language interpreter or write any code to make its job.
About pacman wrappers I agree with your statement. Warning in that section is not going anywhere, I was suggesting to have a description for each section in addition to that warning. E.g. as noted by Vasya it is not clear what meaning of "pacman wrapper" is to the new reader, which I'm assuming he is one of.
-- Svito (talk) 15:27, 6 November 2018 (UTC)
I have no idea what your second paragraph is supposed to say. The executables in aurutilsAUR are free standing like those of package-queryAUR, and if you want to use package-query to write a full AUR helper you'll need more than a few lines of code. (And if you're satisfied with the specific task the executable was designed for, you don't.) That package-query is written in C and aurutils in bash is besides the point.
In actual AUR helper practice, this whole discussion is moot because the only thing close to a "library" that was ever reused by more than 1 project is cowerAUR, and python3-aurAUR for some niche projects like aur-comment-fetcherAUR. (Ironically, cower is not even modular, it just has limited scope.) Otherwise, it's exactly 1 library for 1 project: python3-aurAUR only in bauerbillAUR, pkgbuilderAUR and aurutilsAUR as projects with modular code, package-queryAUR only in yaourtAUR, aur.rsAUR only in ruaAUR, haskell-archlinuxAUR only in auraAUR, go-alpm only in yayAUR, and so on.
For pacman wrappers, it should suffice to clarify what a "wrapper" in general is. E.g. w:Wrapper function. Making it stand out like some notable feature makes no sense. -- Alad (talk) 17:03, 6 November 2018 (UTC)
Alright, I may be getting you. This page for some time has been listing only haskell-archlinuxAUR (deprecated) and python3-aurAUR which I guess is represented through bauerbillAUR and pbgetAUR. Aurweb RPC interface was listing only python3-aurAUR and python-aurAUR.
Maybe just remove Libraries sections from both articles because every single helper is using Aurweb RPC via library or not.
About package-queryAUR I'm still not convinced that tool is either useful enough or notable enough or malitious enough to list it in the table, aside from being pulled and used by yaourtAUR. If its dev asked to put it on the wiki I guess but I doubt they did.
-- Svito (talk) 18:02, 6 November 2018 (UTC)
Removed package-query from the #Search and download table: [9] As an alternative I propose renaming "Libraries" to "Other" (no weighed term) and re-add package-query there as a simple Template:App item: [10] -- Alad (talk) 19:11, 6 November 2018 (UTC)
Looks good to me. -- Svito (talk) 20:43, 6 November 2018 (UTC)

remove Batch interaction 2

Some previous discussion: [11]

On IRC there was some confusion on what "Summary of package upgrades" is supposed to mean. Literally, it means that any (AUR) upgrades or installations a helper will perform are printed to screen, similar to pacman's VerbosePkgLists. This is simple to implement - by definition a AUR helper must know about package names and their versions - and does not warrant a separate mention.

Historically, it's about pacman wrappers running -Sy so they can 1. save a single keypress 2. color the output. Former is a questionable argument when all pacman wrappers have a single prompt per package (so potentially hundreds of keypresses), and latter is already available from pacman itself. (with the Color option) As such, I propose to remove batch interaction 2 from the criteria. -- Alad (talk) 12:52, 7 November 2018 (UTC)

If I might jump in, I don't think "one less key press" is a valid argument, nor does the color which is orthogonal to the feature. Batch interaction is not strictly about reducing the number of keys one has to enter, but it is about reducing the time required by... well, batching every step at the beginning. Batch 2 is about grouping repo *and* AUR packages summaries and initial validation together, rather than doing the repo packages update, then displaying the AUR summary before the AUR packages update. I'd suggest adjusting the current loose definition here. --Spyhawk (talk) 14:00, 7 November 2018 (UTC)