Talk:Systemd/Timers

From ArchWiki
Jump to navigation Jump to search

Counterproductive Suggestions

The suggestions on this page are counterproductive. By merging all units under a single target, you're intentionally creating a stampede when the timer goes off. One of the great features of timer units is that a combination of OnCalendar and AccuracySec can trivially mitigate the overlap that's associated with classic cron while still maintaining regularity. This is distinctly better than classic cron which would force a human to schedule the task jitter manually.

In addition, it isn't reasonable to queue these on basic.target as one wants a fully operational system before cron will run. Much like how cron daemons were treated like any other daemon and brought up in runlevel 3 under sysvinit, it makes much more sense to order timers on multi-user.target. Falconindy (talk) 23:38, 4 May 2014 (UTC)

Your remarks seem to make perfect sense and are corroborated by how the old /etc/cron.*/ files provided by packages in [base] group were converted to systemd.timer files (see https://mailman.archlinux.org/pipermail/arch-dev-public/2014-March/026044.html and files present on one's system)).
So to sum it up:
  • A different /usr/lib/systemd/system/*.{timer,service} couple for every unit wanted should be created (or should /etc/systemd/system/ be used instead?);
  • Then a symbolic link to each custom /usr/lib/systemd/system/*.timer file should be made in /usr/lib/systemd/system/multi-user.target.wants/;
I've had a go at it for the reflector service that I was running as a daily (ana)cron job, taking after the new logrotate systemd.timer service converted by Thomas Bächler. Any comments appreciated.
Step one: create timer and service files for every different unit
/usr/lib/systemd/system/reflector.service
[Unit]
Description=Update pacman mirrorlist

[Service]
Type=oneshot
ExecStart=/usr/bin/reflector --protocol http --latest 5 --sort rate --save /etc/pacman.d/mirrorlist
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7
Note: As for now this service file doesn't work. See my edit below for error log. systemd gurus, your corrections are welcome.
/usr/lib/systemd/system/reflector.timer
[Unit]
Description=Daily update of pacman mirrorlist

[Timer]
OnCalendar=daily
AccuracySec=12h
Persistent=true
Step 2: Order timer on multi-user.target
# ln -s /usr/lib/systemd/system/reflector.timer /usr/lib/systemd/system/multi-user.target.wants/reflector.timer

If the above is correct, this will lead to quite a massive overhaul of the wiki page... Waiting for other comments to proceed.

Edit 2014-05-13: the example service file I wrote for Reflector doesn't work. I get a failed status with systemctl:

# systemctl status -l reflector
● reflector.service - Update pacman mirrorlist
   Loaded: loaded (/usr/lib/systemd/system/reflector.service; static)
   Active: failed (Result: exit-code) since mer. 2014-05-14 00:00:10 CEST; 49min ago
  Process: 6149 ExecStart=/usr/bin/reflector --protocol http --latest 5 --sort rate --save /etc/pacman.d/mirrorlist (code=exited, status=1/FAILURE)
 Main PID: 6149 (code=exited, status=1/FAILURE)

mai 14 00:00:10 arch-clevo reflector[6149]: error: failed to retrieve mirror data: (The read operation timed out)
mai 14 00:00:10 arch-clevo systemd[1]: reflector.service: main process exited, code=exited, status=1/FAILURE
mai 14 00:00:10 arch-clevo systemd[1]: Failed to start Update pacman mirrorlist.
mai 14 00:00:10 arch-clevo systemd[1]: Unit reflector.service entered failed state.

Does anyone know how I should edit this file? I'll be trying different options taken from systemd.service manpage, but if somebody is knowledgeable enough to spare us some time that's all right too... -- Neitsab (talk) 11:44, 8 May 2014 (UTC), edited 23:25, 13 May 2014 (UTC)

Well, I would say that it is a better idea to wait before editing this page. Falconindy, I remember some talk about AccuracySec not working properly on the arch-general ML. Has it been dealt with? - Genghizkhan91 (talk) 05:41, 13 May 2014 (UTC)

Not sure if this is helpful, since it calls a script that does a bunch of things (including reflector) and e-mails me the output, but the following works for me. Except for a couple differences, the systemd service and timer files are largely the same as above. (I had to use "Type=forking", otherwise my MTA would complain about receiving a SIGTERM and would kill the message.) I can post the content of my /usr/local/bin/pacman-sync too if you like, but it's probably not that interesting. After writing these files, I enabled and started pacman-sync.timer.

/etc/systemd/system/pacman-sync.service
[Unit]
Description=Update pacman mirrorlist, sync, and download packages

[Service]
Type=forking
ExecStart=/usr/local/bin/mail-output root "%H: pacman-sync" /usr/local/bin/pacman-sync
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7
/etc/systemd/system/pacman-sync.timer
[Unit]
Description=Daily update of pacman mirrorlist, sync, and download of packages

[Timer]
OnCalendar=daily
AccuracySec=12h
Persistent=true

[Install]
WantedBy=multi-user.target
/usr/local/bin/mail-output
#!/usr/bin/bash

RECIPIENT="$1" ; shift
SUBJECT="$1" ; shift
CMD="$@"

# Mails the output of the given CMD to the given RECIPIENT with the given SUBJECT.
# No mail is sent if no output is produced.
"$@" 2>&1 | /usr/bin/mail -E -s "${SUBJECT}" "${RECIPIENT}"

-- Liujed (talk) 05:43, 23 May 2014 (UTC)


Finally found how to make sure it works Moviuro (talk) 18:47, 11 June 2014 (UTC)

This unit makes sure that at the time it is fired up the host is available. If it isn't, it will keep trying.

/etc/systemd/system/reachable-retry@.service
[Unit]
Description=Test if %i is reachable
# Customize to your own needs, with network.target or whatever works less worse with you
After=systemd-networkd-wait-online.service
 
[Service]
Type=forking
ExecStart=/usr/bin/ping -c1 %i
Restart=on-failure

[Install]
WantedBy=multi-user.target
/etc/systemd/system/timer-daily.target.wants/reflector.service
[Unit]
Description=Update mirorlist
# Fire up the tester
Requires=reachable-retry@www.archlinux.org.service
# Wait for it to return SUCCESS
After=reachable-retry@www.archlinux.org.service

[Service]
# Copied from wiki
Nice=19
IOSchedulingClass=2
IOSchedulingPriority=7
Type=oneshot
# Customize to your own needs
ExecStart=/usr/bin/reflector -i (fr|nl|be). -f 5 --save /etc/pacman.d/mirrorlist

And voilà Moviuro (talk) 18:47, 11 June 2014 (UTC)

Edited

I think the arguments against unified timers makes sense, so I went ahead and removed that section. I then combined info from other sections to form a more complete example. This left the service examples orphaned (since they all relied on the unified timers) so I moved them (when possible) to their appropriate pages. Silverhammermba (talk) 21:01, 28 July 2014 (UTC)

A bit like unified timer

Solution that solves everything I think (though not perfect as it only works on single services):

/etc/systemd/system/timer-daily@.timer
[Unit]
Description=Daily Timer for the %i service

[Timer]
OnCalendar=daily
Persistent=true
Unit=%i.service

[Install]
WantedBy=multi-user.target

and then

systemctl enable timer-daily@myservice.timer

And Voilà! Moviuro (talk) 21:59, 1 September 2014 (UTC)

That is definitely nicer than the old unified timer example! However it still has the problem of all services using this timer starting simultaneously, which I think was the main complaint originally. Though based on the discussion about AccuracySec being random per-host (and thus useless for timer jitter), we still have no better solution than just per-service timers with manually staggered triggers. Silverhammermba (talk) 22:47, 1 September 2014 (UTC)

Workaround for the particular issue "they all fire up at the same time":

/etc/systemd/system/timer-daily@myservice.timer.d/time.conf
[Timer]
# man 7 systemd.time
# adjust for each service that should not run with all others
OnCalendar=*-*-* 1:30:0
# Disable Persistence if you don't want the timer to "catch up"
#Persistent=false

This way, the last systemctl command still works and we can set a different time in the override configuration. Still a bit of a heavy solution, though Moviuro (talk) 07:38, 2 September 2014 (UTC)

The above discussion shows how the current content of the article was developed, but it might be better to start a fresh item in case related questions arise again. We could close it and, once removed, still refer to it via the history if needed.
Or is there demand for keeping this discussion open? --Indigo (talk) 18:02, 9 January 2015 (UTC)

Mailing output

Are there any built-in facilities for e-mailing the output of timed services? It doesn’t look as if there are. Maybe the wiki could outline a good practice for achieving that effect. A simple workaround would be having the service pipe output into sendmail, but maybe there’s a more elegant way (e.g., maintaining this would be cumbersome for many services). --Eigengrau (talk) 10:37, 14 May 2014 (UTC)

I just noticed that the Gentoo wiki has an example using a failure-email@.service which is started using OnFailure=. This seems like the right way to do it, but I don't fully understand their code. --Silverhammermba (talk) 15:14, 6 October 2014 (UTC)

Hi, I rewrote this for inclusion in systemd-cron mail_on_failure It nows checks for MAILTO=, like vixie-cron. --A-detiste (talk) 08:02, 31 October 2014 (UTC)

mail_on_failure's behaviour doesn't match vixie-cron's. Whereas mail_on_failure only sends mail when a job has failed, vixie-cron will send mail containing the job's output regardless of the job's exit status. (Same probably goes for the OnFailure= approach suggested by the Gentoo wiki, described above.) --Liujed (talk) 20:12, 2 November 2014 (UTC)
Indeed, the taste of users varies on this; OnFailure= was absolutely needed and easy to implement "mail on output" is not so straightforward. I've already discussed this elsewhere DebianBug and here systemd-devel .
Putting in a wrapper script between ExecStart= and the actual script is an added point of failure; using OnFailure is more robust. I even have this mailer run with an underprivilieged user
So to there remains work to get "mail on output", here is some options:
  • use some ExecStartPre= & ExecStartPost= magic that parse journalctl output with '--cursor' and save it somewhere
  • read $MAINPID in ExecStartPost= and do something with it
  • use a wrapper ("ExecStart=/usr/lib/mail_on_output /usr/bin/run-parts /etc/cron.daily/")
--A-detiste (talk) 11:31, 4 November 2014 (UTC)

Parallelization section is confusing

The section on parallelization is not at all about parallelization. Systemd timers are already parallel: you can schedule them at any time regardless of other timers. The section is really about staggering timers to prevent large numbers of timers from elapsing at the exact same time. More importantly, the section doesn't convey a clear message. The first paragraph mentions that there is no good way to stagger timers using AccuracySec, then the second paragraph seems to argue against staggering timers. So which is it?

Secondly, the whole discussion about AccuracySec is not clearly relevant. It is only mentioned as it seems like it might be a solution even though it isn't. So then why do we need to argue for the design of it and mention energy consumption? Is it useful or is it not? How exactly does AccuracySec help with power consumption? Why should we care? Silverhammermba (talk) 15:01, 6 October 2014 (UTC)

Hi, your questions are triggered by my edit [1] in the section Systemd/cron functionality#Parallelization yesterday? Let me add: I looked into the section, because we have a report on a related edit [2] and I was working on how we can handle it. Let's see to fix the topic so the report is dealt with as well. Thanks.
You write above that the option "seems like it might be a solution even though it isn't." - can you explain to me why? You do not see it useful at all related to timing events or only not for 'parallelization'? (yes, I have read above talk items, but I'm still puzzling what you mean; we can come back to your above questions right after that). --Indigo (talk) 12:24, 7 October 2014 (UTC)
Let's start from scratch to ensure that we're on the same page. Suppose you have a bunch of timers all set to OnCalendar=weekly. Now that really means "Monday at midnight", so all of your timers will activate their services at the exact same time. In cron talk, this "stampede" of jobs is generally frowned upon and there are several mechanisms that have been created to avoid it (e.g. the RANDOM_DELAY variable). The original motivation of the "parallelization" section was to point out that systemd timers do not have an equivalent mechanism.
AccuracySec is often mentioned as such a mechanism, but systemd.timer(5) states "the expiry time... is synchronized between all local timer units". So it sounds like if you were to add AccuracySec=1h to all of your OnCalendar=weekly timers, you would still get a stampede - just not necessarily right at midnight. Perhaps AccuracySec does serve some useful purpose, but if it has no use for randomly staggering timers to prevent stampedes then that discussion does not belong in the parallelization section.
If AccuracySec does help with stampedes, then I really need someone to explain what the hell "synchronized between all local timer units" means if the timers are not activated at the same time. Alternatively, if such stampedes are not a problem with systemd, the article should explain why - as it is a rather common viewpoint.
--Silverhammermba (talk) 13:48, 7 October 2014 (UTC)
Ok, thanks! You most definetely have more practical experience with it, but let me contrast your summary with my understanding of it: With default configuration the example in Systemd/cron_functionality#Realtime timer with OnCalendar=Wed, 23:15 would kick off accordingly. (with an accuracy of 1 minute == default configuration). A 'stampede' in the historic cron sense would arguably only happen if you force all timers to realtime ("exact same time") on calender with something like AccuracySec=1usec, which we do not do i.e. default applies.
Now, let's say you add another timer event with the same OnCalendar=Wed, 23:15, it would be started within the same minute but not the same usec as the first one (anti-stampede ala cron's randomdelay). I might be wrong on something, but I believe that is also what falconindy referred to in his first paragraph, first talk item.
Of course if both those timed events are intensive on the same resource (e.g. bandwith, cpu, disk i/o), one has to control that (Nice, IOSchedulingClass, ...) or even spread them out more - but that is specific to the timed tasks. However, if resource restraints are not the primary concern, one can set something like OnCalendar=daily, 23:15 and even expand to AccuracySec=23h (or 24h, not sure now). This should lead to all timers scheduled daily at any time be staggered close to each other. Taking this example further (I have not tried this): Let's say the system is a small homeserver and auto-suspended. If you additionally use WakeSystem in the timers, the AccuracySec=23h should make it wake up only once daily to excute timed events (energy consumption).
Are we getting closer? Am I wrong somewhere?
Apart from coming to the same understanding, I think you are clearly right that that the section needs more verbose to explain and "parallelization" is the wrong section title for it.
--Indigo (talk) 18:30, 7 October 2014 (UTC)
I think that, "it would be started within the same minute but not the same usec as the first one" is the point of disagreement. I read "synchronized between local timers" as meaning that timers with the same On... option and the same AccuracySec would activate at practically the exact same time. I have tested this by creating three services that all run date +%s.%N to display the current time in seconds and nanoseconds and three timers all with OnCalendar=*-*-* *:*:00 (every minute) and AccuracySec=30s. Here are the results:
Oct 07 16:48:22 h5g-max date[5214]: 1412714902.218033473
Oct 07 16:48:22 h5g-max date[5215]: 1412714902.218698707
Oct 07 16:48:22 h5g-max date[5217]: 1412714902.219415650

Oct 07 16:49:15 h5g-max date[5224]: 1412714955.163915339
Oct 07 16:49:15 h5g-max date[5225]: 1412714955.165712079
Oct 07 16:49:15 h5g-max date[5226]: 1412714955.167225757

Oct 07 16:50:22 h5g-max date[5340]: 1412715022.189751381
Oct 07 16:50:22 h5g-max date[5341]: 1412715022.192302896
Oct 07 16:50:22 h5g-max date[5342]: 1412715022.193473285
So even though all three services do start at a random point between 00 and 30, they are all started within milliseconds of each other (sometimes even less)! This is certainly not the same sort of behavior as cron's RANDOM_DELAY. Furthermore, if we change the timers to AccuracySec=1us (the most accurate, according to the man page), we get the exact same millisecond-separation between services:
Oct 07 16:59:00 h5g-max date[5458]: 1412715540.037773807
Oct 07 16:59:00 h5g-max date[5460]: 1412715540.039115540
Oct 07 16:59:00 h5g-max date[5459]: 1412715540.038436570

Oct 07 17:00:00 h5g-max date[5464]: 1412715600.006456259
Oct 07 17:00:00 h5g-max date[5465]: 1412715600.008399768
Oct 07 17:00:00 h5g-max date[5466]: 1412715600.013211128

Oct 07 17:01:00 h5g-max date[5473]: 1412715660.008787964
Oct 07 17:01:00 h5g-max date[5474]: 1412715660.009727101
Oct 07 17:01:00 h5g-max date[5475]: 1412715660.011859585
From this I conclude that AccuracySec is not useful for staggering multiple services with the same start time. In fact, my new theory is that its purpose is exactly the opposite - to create stampedes! That makes sense with your example of waking a machine once per day. For example, if we have a timer with OnCalendar=*-*-* 10:00:00, AccuracySec=4h and another with OnCalendar=*-*-* 12:00:00, AccuracySec=1h, systemd can choose to activate both timers at the same time and thus only wake the system once. I have no evidence that it does this, but if that is indeed the behavior it might be worth mentioning in some other section of the article.
--Silverhammermba (talk) 21:26, 7 October 2014 (UTC)
That's an interesting test and results! So my above assumption about setting AccuracySec=1us was clearly wrong. I wrote that because I read that the option behind AccuracySec, TimerSlackNSec defaults to 50us. In our config I don't see that value though, not sure what that means. The grouping/coalescing if the events we talk about is done in sd-event.h [3], maybe it is defined there. I'm still not fully convinced we would call it a stampede. It would be interesting if something changes with the date timers, if TimerSlackNSec is set to a greater slack (e.g. 5ms) to test.
Related as a reference I saw the following bug: [4]
--Indigo (talk) 15:25, 8 October 2014 (UTC)
Thanks for the bug links!
I just did another test, this time with hourly timers starting at 00, 20, and 30 minutes but with AccuracySec 40m, 20m, 10m, respectively. I had hoped to see some kind of coalescing (since a single start time between 30 and 40 minutes would satisfy every timer). Instead the timers started at 12:00:17, 12:20:17, and 12:30:17. So again it looks like they all got the same AccuracySec. This also seems to backup Lennart's comment that "it will only really place the wakeup within 1min of range at max. If you set larger values it won't care". I'm still utterly confused about the purpose of AccuracySec but seeing as it clearly has nothing to do with staggering start times between timers, I will update the article accordingly.
--Silverhammermba (talk) 16:37, 8 October 2014 (UTC)
I also saw Lennart's comment about the minute, but did not take that for current because the systemd.conf option he proposes for it at the same time was already added in 216. Nonetheless, your second test above sure indicates it does not work as intended/described. Thanks! I'll refrain from adding in something about a 'coalescing wakeup to reduce energy consumption' again until I have witnessed it works ^^ --Indigo (talk) 19:35, 8 October 2014 (UTC)
I believe this talk item was implemented by reworking the "Parallezation" subsection to Systemd/Timers#As_a_cron_replacement. As such I propose to close this item and open a fresh one which states a reference to it. The fresh item because I believe the test results by Silverhammermba would be a very useful base to reconsider the item once there are relevant changes to systemd and its handling of the AccuracySec parameter. I don't use timers enough myself to follow it up right away again.
Objections against closing this and using a reference instead? --Indigo (talk) 18:11, 9 January 2015 (UTC)