Talk:AUR helpers

From ArchWiki
(Redirected from Talk:Yaourt)
Jump to: navigation, search
Note: Moderation — If your AUR helper does partial upgrades without explicit user intervention (i.e, specifying -Sy on the command line), it has no place on this page or anywhere else on ArchWiki. No exceptions. -- Alad (talk) 09:37, 20 September 2015 (UTC)

Comparison table - build directory

Considering /tmp is mounted as tmpfs on Arch, and the potential downsides from building in RAM (running out of space), I think a column with the default build location for various helpers would be helpful.

The default values I've garnered so far, assuming TMPDIR is not set:

  • aurutils: $XDG_CACHE_HOME
  • pacaur: $XDG_CACHE_HOME (changed from /tmp, see [1])
  • bauerbill: $PWD/build
  • pkgbuilder: $PWD, /tmp when specified with -S
  • packer: /tmp (TMPDIR)
  • yaourt: /tmp (yaourtrc)

-- Alad (talk) 18:16, 1 April 2016 (UTC)

Yes, this could be useful. Although you'd want not to use color here, since users that know what they're doing would prefer to use /tmp (or setting up BUILDDIR to /tmp). --Spyhawk (talk) 11:15, 3 April 2016 (UTC)
+1. see also #Multi-thread support. --Indigo (talk) 11:33, 3 April 2016 (UTC)
Well, while it does have benefits for some users, it's still a bad default. As you say though, this is easy enough to change either way, unlike any of the behaviour described in the other columns.
We could leave out the colors, but mention the drawbacks/benefits in the "meanings" paragraph. -- Alad (talk) 13:35, 4 April 2016 (UTC)
It is bad default because some users have no idea about what they are doing, but this is strictly related to user preferences. Adding the meaning instead of colors sounds like the perfect solution to me. --Spyhawk (talk) 14:35, 4 April 2016 (UTC).

Reliable Updater

Interested in feedback on possibly adding Reliable Updater as a category to Comparison table.

ie: Does it handle accurate update status on VCS packages? Does it handle accurate update status when developer fails to update .SCRINFO?

And any other unmentioned situations. Cody Learner (talk) 18:49, 22 February 2018 (UTC)

The second is an issue only pacaur has, by design to "improve metadata on the AUR". It has nothing to do with what an AUR helper should do. The first is at best a specificity, since the AUR has no perception of what a VCS package is. See FS#56602. -- Alad (talk) 20:02, 22 February 2018 (UTC)
I think the most important here to provide reliable testcase to prove the reliability of updater :-) I would suggest mb creating a repo a with some stub PKGBUILDs which could be used as testcases for criterias in the table. Actionless (talk) 20:19, 22 February 2018 (UTC)
Not sure what a testcase of such would look like, since scoring on the other criteria should guarantee reliable updates apart from some pecularities outlined above.
About the second case, it has been suggested before to create some centralized place for testing helpers instead of a few arbitrarily chosen AUR packages. However, since AUR helpers are (by definition) for the AUR, I wonder how you'd go about testing these helpers with an external repository. PKGBUILDs specifically made for testing helpers would not be accepted on AUR anyways as too specific. -- Alad (talk) 22:30, 22 February 2018 (UTC)
And what about adding packages to AUR but with some special prefix in package name (`_stub-package-test-reliable-solver`) and very explicit description ("DON'T INSTALL ME. Stub package intended for testing AUR helpers for 'reliable solver' criteria.") and so on? Actionless (talk) 23:53, 22 February 2018 (UTC)
Considering AUR helpers are something that's tolerated instead of supported, I doubt such packages explicitely targeting them with no use otherwise would have a long lifetime. -- Alad (talk) 12:00, 23 February 2018 (UTC)
See #"Reference" implementation for an alternative. -- Alad (talk) 13:29, 8 March 2018 (UTC)
I would argue this is covered by the new "Pacman wrap" column. That said there's some strange cases (e.g. rakudoAUR or nvidia-betaAUR) which some helpers can install successfully but fail to update afterward. Usually this involves version requirements (though note FS#54906). -- Alad (talk) 22:31, 18 March 2018 (UTC)

"AUR repo diff"

[2] no idea what that's supposed to mean. If it's git diffs, half of the table supports those. -- Alad (talk) 07:59, 8 March 2018 (UTC)

yup, that means exactly that. i wasn't sure if it worth mentioning or not Actionless (talk) 12:24, 8 March 2018 (UTC)
when a helper supports git clone but not git diff, we could consider using Template:Y instead of Template:Yes for the git column. -- Alad (talk) 12:58, 8 March 2018 (UTC)
good idea Actionless (talk) 12:59, 8 March 2018 (UTC)

"Reference" implementation

This is an alternative to #Reliable_Updater. Instead of an arbitrary set of test packages, we could write up a "specification" on what a reliable AUR helper should do. This should also be more helpful for potential AUR helper writers who otherwise have to wade through complex, fully-featured AUR helpers.

I propose a minimal reference implementation with the following points:

  • No client-side workarounds for upstream limitations. In particular, a reference implementation does not need to score full points on split packages, as makepkg --pkg was removed with pacman 5.
  • Minimal language constructs in e.g. a scripting language like dash.
  • Prefer simplicity of implementation over being fully featured. In particular, an implementation may only support git clone and not git diff.

My initial plan was to keep such an implementation in a man page aurhelper(7) (hosted as part of aurutils), but we can consider including on a sub-page of this article. It could be then linked from the comparison table. Thoughts? -- Alad (talk) 13:28, 8 March 2018 (UTC)

Generally agree with the idea, but I don't think there is a way around a set of PKGBUILDs that could be used to test helpers in a local AUR instance. F.e., I wouldn't define a "reliable" helper that doesn't handle split packages well. Since helpers are tolerated rather than supported, upstream limitations of the AUR might be temporary or permanent, meaning the limitation would actually be in the helper itself (f.e. like regex support). Also, I'd use pseudo code for such a reference as the actual implementation itself doesn't matter, unless you'd like to write a new minimalist helper. Spyhawk (talk) 15:26, 8 March 2018 (UTC)
Apart from FS#56602, I can't think of a case where upstream opposed removing limitations, even if helpers directly benefited. cf. the regex support discussed in [3] or the exit codes finally introduced in makepkg 5.1 which made automatic building significantly easier imo. To me it seems that the main reason we have these AUR limations is due to the minimal interest of helper writers in contributing upstream, and upstream itself having different priorities. Not sure why former is the case, the PHP codebase may play part in it - at least it does for me.
You can keep dash close enough to pseudo-code, I guess less so if you want a complete example rather than exemplary code blocks. For the PKGBUILD set, I use this: [4] -- Alad (talk) 18:34, 8 March 2018 (UTC)
My understanding is that changes that aren't invasive will be accepted upstream, but otherwise might be rejected (see [5]). One prominent example that comes to mind is FS#48796. It's not really relevant anymore since x86 has been officially dropped, but the solution would involve duplicating DB tables on the server, which isn't trivial to implement/migrate. Many of the feature requests involve non-trivial code change, which is the main reason nobody pushed patches; I dislike PHP but the language itself isn't too hard either. For regex, see the bottom of [6], which is the follow-up of your link above.
Your testsuite seems interesting (thanks for the link), but one advantage of having a fixed set of packages is that these packages might be updated and change, making these edge cases difficult to test. This happened quite a few times with my own list of test packages in the past and this was rather annoying. Spyhawk (talk) 20:20, 8 March 2018 (UTC)

Add "pacman wrap" column

Since we have a Clean build column, I believe we should also have a "Pacman wrap" column that details how helpers wrap pacman (if applicable). A green entry should be reserved to helpers that only use pacman directly, while avoiding potentially harmful commands such as pacman -Ud or pacman -Rdd. A yellow entry could be for direct pacman use but including use of latter commands. A red entry would be for reimplementations through libalpm or other means. An N/A (grey) entry is used if no pacman wrapping is done in the first place.

The reasoning for this proposal is that while the table gives a good overview on helper features, it gives no indication whatsoever on potential harmful effects due to reimplementations of pacman. This is already having an effect on the support channels, where at best it makes debugging more difficult (see for example [7]), and having such a column might encourage future helper writers to avoid making this same mistake. -- Alad (talk) 02:59, 11 March 2018 (UTC)

For example in pikaur I do use -Rdd in one place: [8]
However I don't know any other possible way to handle the removal of conflicting package before installing its replacement with --noconfirm option.
But to avoid the problem on the further steps it's guarded with transaction: [9] so if something will fail that operation will be canceled.
Also it's not clear to me if there is any difference between pacman -Su and pacman -S $(pacman -Quq). Actionless (talk) 03:56, 11 March 2018 (UTC)
If pacman would have an option to not confirm about installing packages (:: Proceed with installation? [Y/n]) but confirm everything else (like conflicts and so on) that would actually solve the issue at all. Actionless (talk) 04:31, 11 March 2018 (UTC)
You can use the undocumented --ask flag (see pacman git log) or better, pacinstall --remove from pacutils which has the ability to install and remove packages concurrently. -- Alad (talk) 18:20, 12 March 2018 (UTC)
Thanks! Is there a similar way to avoid extra question in 'makepkg --syncdeps --rmdeps'?
UPD: i've also updated pikaur to avoid using --nodeps and --noconfirm. Actionless (talk) 03:33, 13 March 2018 (UTC)
I don't know, you can try passing --ask to makepkg I guess. -- Alad (talk) 20:58, 18 March 2018 (UTC)
That was the first i've tried, but makepkg was complaining what it's unknown argument. Actionless (talk) 05:34, 20 March 2018 (UTC)
Yep, it looks like the arguments are hardcoded: [10] -- Alad (talk) 06:44, 20 March 2018 (UTC)
I guess that's actually a good idea, I'd just like to include one more thing. That column should yield details about the behavior of that aur helper when using the helper without non native pacman flags. Surely I have something in mind why I say that: aurman flag --do_everything, which is a non native pacman flag, which is also being described in the readme as not recommended, but which may be used. it changes the behavior of aurman -Syu in the first place, so that would be a candidate for the "non-green" entry, but I bet there are other aur helpers out there, which by default only do sane things, but may be used in another way, when specifying any config options or command line flags. one could go one step further maybe: only if the aur helper specifies those side effects in the readme, man page or whatever, the aur helper gets a "green" entry. hence those side effects may only occur if the user uses a flag, whereas those side effects have been clarified enough Polygamma (talk) 15:39, 11 March 2018 (UTC)
Updated the table (nearly lost all my changes due to edit conflicts...): [11] Some notes:
  • The source code of naaman and yaourt was incomprehensible to me, but yaourt at least parses sync dbs by hand (yaourt -Q) and separates pacman -Syu to pacman -Sy && <stuff> && pacman -Su so it got a red entry.
  • yay got a red entry, see below.
  • pikaur got a red entry due to issues like [12], which while resolved, give no indication that pikaur started using pacman rather than a manual (py)alpm implementation. aurman got a green entry due to it only replacing pacman when the --do_everything flag is passed. Correct me if I'm wrong on either.
  • I was gratuitous with the N/A flag; e.g. cower uses libalpm(3) but it's not used as a pacman wrapper.
-- Alad (talk) 21:04, 18 March 2018 (UTC)
Looking very good! The page is much clearer now. Thanks Alad for the rework. Spyhawk (talk) 14:10, 19 March 2018 (UTC)
Also maybe the column should be renamed to something that better reflects Yes/No entries, like "Native pacman" or such. That or change the Yes/No entries to specific ones such as {{R|libalpm}}. Former should be simpler, since details can be explained in added links. -- Alad (talk) 21:14, 18 March 2018 (UTC)
I think it's important to keep in mind the table is supposed to be a quick reference to choose a helper. Looking at the average AUR user, I'm not sure adding such technical details is relevant in that context. Spyhawk (talk) 14:14, 19 March 2018 (UTC)
Also note that PKGBUILDerAUR has "Yes" in the pacman wrap column, although it uses -F for something else than pacman. -- Lahwaacz (talk) 16:46, 19 March 2018 (UTC)
This seems like an oversight (pkgbuilder -F was present before pacman v5), so I've opened a ticket here: [13] -- Alad (talk) 20:19, 19 March 2018 (UTC)
If yay -Syu is in fact -Syyu, that sounds more like a "bug" that should be fixed. Has this been reported, or acknowledged and dismissed? Spyhawk (talk) 13:51, 13 March 2018 (UTC)
I'm not sure on the -Syy part, but in general, acknowledged and dismissed: [14] -- Alad (talk) 20:58, 18 March 2018 (UTC)
Fair enough, I did miss that ticket. Spyhawk (talk) 14:10, 19 March 2018 (UTC)
I want some clarifications on the difference between:
1) pacman -Su and pacman -S $(pacman -Quq)
2) pacman -Syu and pacman -Sy ; pacman -Su
Actionless (talk) 03:38, 20 March 2018 (UTC)
Some helpers like yayAUR separate a -Syu not only into -Sy + -Su, but into -Sy and then appending the updated packages to pacman -S. As linked above, this makes it very hard to distinguish if a partial upgrade was performed or not.
A separation to -Sy + -Su is less bad, or at least not much worse than a user interrupting an -Syu after the databases were synced. It's still not standard behavior though. I guess you'd do it to format output? -- Alad (talk) 03:53, 20 March 2018 (UTC)
I think we need to reconsider criteria for that section. I think the best way will be to split the helpers into 3 categories:
1) precise pacman wrapping
2) some minor things like example above, but without potential db breakage
3) like --nodeps or so
Actionless (talk) 05:28, 20 March 2018 (UTC)
In pikaur i do that separation not only to format output but also in order to compute the conflicts (to ask about them in advance, however conflicts themselves are resolved via ask, not manually). Also to be able to manually edit list of packages. I hope i didn't forgot anything else. Actionless (talk) 05:32, 20 March 2018 (UTC)
AUR helpers that require full pacman reimplementations with libalpm and/or strange command lines to achieve the same thing as helpers that do none of these don't need the same recognition on the table. 3) is basically a Yellow entry, 1) green.
If you want 2) to be green you'd need all kinds of exceptions which I doubt is very helpful (cf. Spyhawk's comment on "quick reference"). An alternative would be to keep onorthodox things beyond an optional flag. Like aurman has the --do-everything flag, which if it were default, would warrant it a red table entry. -- Alad (talk) 05:55, 20 March 2018 (UTC)
I was thinking 1) green; 2) yellow; 3) red. Actionless (talk) 06:00, 20 March 2018 (UTC)
Well, I'm not sure I'd give a helper a red entry solely for it using e.g. pacman -Ud someplace. On the other hand, for pikaur's case another report sprang up which I assume is again due to its pacman reimplementation: [15]
My issue is to give designs based on reimplementing pacman anything more than red, for the same reason as what's in the warning at the top of the article. The authors can claim that their implementation is correct and has no breakage when using it over pacman(8), but in the end it's fully unsupported by anyone on the arch or pacman team.
Giving users an ability to easily check whether they use some custom/unsupported implementation was the main motivation behind this new column, rather than the overly broad warning "don't use pacman wrappers at all". This doesn't mean you can't use libalpm or other manual implementations at all - most of the entries on the table do in some way - it's about the scope of things and whether using $HELPER <some operation usually done with pacman> will actually have the same effect as pacman <some operation usually done with pacman>. -- Alad (talk) 06:10, 20 March 2018 (UTC)
I am not saying what you need to lower down an alarm level for pikaur. I am saying what you need to increase the alarm level (mb with additional marks) for those helpers which are using things like --nodeps or other potentially damageable for the package db consistency. Actionless (talk) 06:17, 20 March 2018 (UTC)
Oh, right. Well that sounds fair to me. It's not like we have these distinctions in the "inactive" section or the "secure" column either. -- Alad (talk) 06:20, 20 March 2018 (UTC)
Done [16]. I guess if we have a helper that really only does minor things like pacman -Sy && <do some parsing> && pacman -Su that a yellow entry might fit. Or as said, do -Syu by default and add a --fancyparse flag or such. -- Alad (talk) 06:26, 20 March 2018 (UTC)
Note that in most cases a link should suffice, e.g. a github post where it is explained why pacman -Syu is done but split to -Sy and -Su. -- Alad (talk) 06:39, 20 March 2018 (UTC)
Let me try to explain, why splitting -Syu to -Sy + -Su is the only way to do things right and not harmful at all.
1. -y option is a special option which can be used not only for installing packages: -Syi, -Sys, -Syg, etc. Refresh always happens before any of these operations start. They are not atomic, as well as -Sy package or -Syu. There are also delays between refreshing repos, there is delay for some calculations which happen after refresh. Splitting -Syu to -Sy and -Su only makes this delay longer and nothing more.
2. -Su package operation is atomic. Reimplementing this operation as -Su + -S package is wrong. At least, user would be asked twice.
3. Let's look at this specific case: -Syu package, when package package was moved from AUR to Arch repos. Before refresh, we can't know our package present in repos, so we can't check AUR db before -y. On the other hand, we shouldn't split -Su package to -Su + -S package (see above), which means we should know whether package in repos or not, so we could call pacman -Su package or just pacman -Su depending on package presence. This means that the only right place to check AUR db is after -Sy but before -Su package. Kitsunyan (talk) 10:56, 20 March 2018 (UTC)
It's not "the only right way", it's "the way of trying to do too much". @Polygamma maybe you could expand on this since aurman doesn't separate -Syu, does it? -- Alad (talk) 14:22, 20 March 2018 (UTC)
If a package is moved from AUR to the Arch repos, pacman -Syu package would do the right thing so you wouldn't need to check the AUR at all, since even if the package was still in AUR, the official package should take precedence. If the package is moved from the Arch repos to the AUR, pacman -Syu package would fail and I think it's important to show that to the user to decide if he still wants the package. In other cases, you can check the AUR whenever you want. -- Lahwaacz (talk) 11:55, 20 March 2018 (UTC)
In AUR → Arch repos case, AUR helper should decide which call to perform: pacman -Syu or pacman -Syu package, depending on package presence so pacman doesn't fail. But how AUR helper supposed to know that before refresh? -- Kitsunyan (talk) 12:18, 20 March 2018 (UTC)
You can simply use the old database(s) to cover the 99.9% cases when the packages are not moved and deal with the rest when something fails - seems better than risking partial upgrades in all cases. Or if you insist on always having an up-to-date databases for the decision, use a separate sync database like the checkupdates script. -- Lahwaacz (talk) 22:52, 20 March 2018 (UTC)
What partual upgrades risk? If AUR helper runs -Su after -Sy, pacman will do full system upgrade. AFAIU partial upgrade is installing packages after -Sy without -Su, but that's not the case. Could you explain the difference between -Syu and -Sy + -Su then? -- Kitsunyan (talk) 23:59, 20 March 2018 (UTC)
Partial upgrades are exactly about the delays between -Sy and -Su. If they follow right after each other, in which case you wouldn't need to split them, then it's fine. If you do things which could fail by themselves or be aborted by the user, like querying the AUR, downloading sources and presenting PKGBUILDs to the user (this is what "batch interaction" means, right?), then you have a problem. I hate to break it to you, but if you query the AUR before -Su and cache it until -Su is complete, you should query it again because -Su might take hours on a slow network, so the first query is useless. -- Lahwaacz (talk) 07:25, 21 March 2018 (UTC)
So, partial upgrade happens if operation was aborted for some reason, right? But pacman can fail itself: pacman -Syu qweasd (package qweasd is not exist, so pacman will also fail to upgrade). User can abort pacman during refresh (repos will be synced partially) or during packages download. The only thing I do is querying AUR database and asking about ignored packages; git clones and "view file" asks are deferred. I think I can defer the rest questions (about ignored packages/etc) as well. In this case, will -Sy + AUR query without fails and asks + -Su be still considered as partial upgrade? -- Kitsunyan (talk) 13:12, 21 March 2018 (UTC)
In any event, such behavior will warrant a yellow label at best, as suggested by @Actionless. If you want something green, consider a more clever mechanism like the JSON interface or make it optional behind a flag.
Note: to me it seemed that pakku also has a whole lot of pacman reimplementation going on through some home-made libalpm interface but I could be mistaken. If not that definitely warrants a red label. -- Alad (talk) 16:25, 22 March 2018 (UTC)
Pakku uses libalpm for queries used in calculations. It's much faster to call necessary functions than parsing pacman/expac/etc's output, which ultimately leads to the same result. Pakku doesn't alter database. -- Kitsunyan (talk) 21:08, 22 March 2018 (UTC)
About the "we can't know our package present in repos" part - the official repos, much like the AUR, have a JSON interface so you could just query that to get the latest information, independent of mirrors or synced dbs. -- Alad (talk) 01:47, 21 March 2018 (UTC)