107.2 Automate system administration tasks by scheduling jobs¶

Weight: 4

Candidates should be able to use cron and systemd timers to run jobs at regular intervals and to use at to run jobs at a specific time.

Objectives

Manage cron and at jobs.
Configure user access to cron and at services.
Understand systemd timer units.

Terms

/etc/cron.{d,daily,hourly,monthly,weekly}/, /etc/at.deny, /etc/at.allow, /etc/crontab, /etc/cron.allow, /etc/cron.deny, /var/spool/cron/, crontab, at, atq, atrm, systemctl, systemd-run

Three tools for scheduling¶

An administrator automates the jobs that repeat: backups, updates, cleanups. Linux gives you three tools:

Tool	Use it for
cron	jobs that repeat on a schedule, like every night at 03:00
at	a job that runs once, at a future time
systemd timers	the systemd way to do both

cron¶

cron is a daemon that runs all the time and wakes up every minute to check its tables, called crontabs, for jobs to run. A job runs only if the machine is on at that moment, so cron suits servers that never switch off. (For machines that are often off, there is anacron, which runs missed jobs at the next boot. It is outside the LPIC-1 scope.)

There are two kinds of crontab:

Kind	Where	Edited with	Fields
user crontab	one per user, under `/var/spool/cron/`	`crontab -e`	5 time fields + command
system crontab	`/etc/crontab` and the files in `/etc/cron.d/`	any editor, as root	5 time fields + user + command

The crontab format¶

Each line of a user crontab has six fields, separated by spaces. The first five say when, and the rest of the line is the command:

 .---------------- minute        (0 - 59)
 |  .------------- hour          (0 - 23)
 |  |  .---------- day of month  (1 - 31)
 |  |  |  .------- month         (1 - 12) OR jan,feb,mar,apr ...
 |  |  |  |  .---- day of week   (0 - 7, Sunday is 0 or 7) OR sun,mon,tue,wed,thu,fri,sat
 |  |  |  |  |
 *  *  *  *  *  command to be executed

Month and weekday also accept the first three letters of the name. Each time field can hold more than one value:

Symbol	Means	Example	Matches
`*`	any value	`*` in the hour field	every hour
`,`	a list	`8,18`	8 and 18
`-`	a range	`1-5` in the weekday field	Monday to Friday
`/`	a step	`*/5` in the minute field	every 5 minutes

Read these until each one is obvious:

0 10 * * *          /home/frank/foo.sh    # every day at 10:00
5 0 * * *           $HOME/bin/daily.job   # every day at 00:05
15 14 1 * *         $HOME/bin/monthly     # 14:15 on the 1st of every month
0 22 * * 1-5        $HOME/bin/reminder.sh # 22:00, Monday to Friday
23 0-23/2 * * *     $HOME/bin/check.sh    # 00:23, 02:23, 04:23 ... every day
5 4 * * sun         $HOME/bin/weekly.sh   # 04:05 every Sunday
*/5 * * * *         $HOME/bin/poll.sh     # every 5 minutes
42 8,18 * * 1-5     $HOME/bin/report.sh   # 08:42 and 18:42 on weekdays
0,15,30,45 08 * * 2 /home/frank/bar.sh    # Tuesdays at 08:00, 08:15, 08:30 and 08:45
30 20 1-15 1,6 1-5  /home/frank/foobar.sh # 20:30, weekdays, days 1-15 of January and June

Two traps to watch for:

A * in the minute field means every minute. * 3 * * * runs 60 times between 03:00 and 03:59, not once.
42 8 1 1 0 runs only when the 1st of January is also a Sunday.

Special shortcuts¶

In place of the five time fields you can write one shortcut:

Shortcut	Runs
`@reboot`	once, after each boot
`@hourly`	at the start of every hour
`@daily` (or `@midnight`)	every day at midnight
`@weekly`	every week, at midnight on Sunday
`@monthly`	at midnight on the 1st of every month
`@yearly` (or `@annually`)	at midnight on the 1st of January

@reboot        echo "runs after the reboot"

Variables in a crontab¶

A crontab can set variables before the jobs. The common ones:

Variable	Sets	Default
`SHELL`	the shell that runs the commands	`/bin/sh`
`PATH`	where to look for commands	set by the system
`HOME`	the directory where commands run	the user's home
`MAILTO`	who gets the job's output by mail. Several addresses can be comma separated. Empty means send no mail	the crontab's owner

Whatever a job prints is mailed to its owner (or to MAILTO). A common habit is to send normal output to /dev/null and leave errors alone, so you get a mail only when something goes wrong:

30 01 * * * root /root/barfoo.sh >/dev/null

Or keep both in log files to read later:

30 01 * * * root /root/barfoo.sh >>/root/output.log 2>>/root/error.log

User crontabs: the crontab command¶

Each user has their own crontab, named after the account. The file lives under /var/spool/cron/ (for example /var/spool/cron/crontabs/ or /var/spool/cron/tabs/, depending on the distribution). Do not edit it directly. Use crontab, which checks your changes and loads them:

Command	Does
`crontab -e`	edit your crontab (it is created if needed)
`crontab -l`	list your crontab
`crontab -r`	remove your crontab
`crontab -u frank -e`	edit frank's crontab (root only)

crontab -e opens the editor named in the VISUAL or EDITOR variable. Some distributions ask you to choose one the first time:

$ crontab -e
no crontab for frank - using an empty one

Select an editor.  To change later, run 'select-editor'.
  1. /bin/ed
  2. /bin/nano        <---- easiest
  3. /usr/bin/emacs24
  4. /usr/bin/vim.tiny

Choose 1-4 [2]:

Add one line, save, and the job is scheduled. To run ~/foo.sh every day at 10:00:

0 10 * * * /home/frank/foo.sh

The stored file starts with a warning, which is why you always go through crontab -e:

# cat /var/spool/cron/tabs/frank
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.khObLu installed on Thu Oct 29 22:04:43 2023)
(...)
* * * * * date >> /tmp/date.cron.txt

Be careful with crontab -r: it deletes the whole crontab without asking.

System crontabs¶

/etc/crontab and every file in /etc/cron.d/ are system crontabs. Only root can change them, and you edit them with a normal editor, not with crontab. They have one extra field, the user that runs the job, between the time and the command:

 minute  hour  day  month  weekday  USER  command

30 01 * * * root /root/barfoo.sh >>/root/output.log 2>>/root/error.log

A typical /etc/crontab sets its variables, then runs the jobs of the hourly, daily, weekly and monthly directories:

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed

Another /etc/crontab, with the line that runs the hourly directory:

# cat /etc/crontab
SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin
MAILTO=root

17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly

The first lines set the shell, the search path, and who receives the output e-mail. run-parts runs every script in the given directory.

Programs drop their own crontabs into /etc/cron.d/, in the same 7-field format. For example the one installed by the monitoring tool munin:

$ cat /etc/cron.d/munin
#
# cron-jobs for munin
#

MAILTO=root

*/5 * * * *     munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi
14 10 * * *     munin if [ -x /usr/share/munin/munin-limits ]; then /usr/share/munin/munin-limits --force --contact nagios --contact old-nagios; fi

The cron.hourly, daily, weekly and monthly directories¶

The easiest way to schedule a script: drop it into one of these directories and it runs at that interval. No time fields needed:

Directory	Scripts run
`/etc/cron.hourly/`	every hour
`/etc/cron.daily/`	every day
`/etc/cron.weekly/`	every week
`/etc/cron.monthly/`	every month

$ ls /etc/cron.daily
apt  dpkg  logrotate  man-db  mlocate  passwd

To schedule a daily job, you just copy your executable script into /etc/cron.daily/. This keeps routine cleanup and backup scripts tidy.

$ sudo tree /etc/cron*
/etc/cron.d
├── munin
├── php5
└── sendmail
/etc/cron.daily
├── apt
├── dpkg
├── logrotate
├── man-db
├── mlocate
└── passwd
/etc/cron.hourly
/etc/cron.monthly
/etc/cron.weekly
├── fstrim
└── man-db

These are ordinary scripts. For example, logrotate (108.2) runs daily because its script sits in /etc/cron.daily/. Some distributions use /etc/cron.d/daily and similar instead, so check which directories your system has.

Who may use cron¶

Two files control which users may use crontab. Each holds one user name per line:

Situation	Who may use cron
`/etc/cron.allow` exists	only the users listed in it
no `cron.allow`, but `/etc/cron.deny` exists	everyone except the users listed in it. An empty `cron.deny` allows everyone
neither file exists	depends on the distribution

cron.allow wins: if it exists, cron.deny is ignored. These rules apply to normal users; root can always use cron.

at: run a job once¶

at schedules a job to run one time. Type at and a time, then the commands at the at> prompt, and finish with Ctrl+D:

$ at now +5 minutes
warning: commands will be executed using /bin/sh
at> date
at> <EOT>
job 12 at Sat Sep 14 09:15:00 2019

Like cron, at mails you the job's output. The atd daemon must be running for at to work.

Time formats¶

You type	Runs
`at 17:30`	at 17:30. If that time has passed today, tomorrow
`at 09:30 AM`	12-hour format with AM or PM
`at now +5 minutes`	`now +` a number of minutes, hours, days or weeks
`at noon`, `at midnight`	at 12:00 or 00:00
`at teatime`	at 16:00
`at 17:30 tomorrow`	`today` or `tomorrow` after a time
`at 07:15 AM Jan 01`	on a date. Also `MMDDYY`, `MM/DD/YY`, `DD.MM.YY` and `YYYY-MM-DD`

Listing and removing jobs¶

Command	Same as	Does
`atq`	`at -l`	lists your pending jobs: ID, date, time, queue, user. Root sees all users' jobs
`atrm 14`	`at -d 14`	deletes job 14. Several IDs can be given
`at -c 14`		prints the commands of job 14

$ at 09:30 AM
warning: commands will be executed using /bin/sh
at> ./foo.sh
at> <EOT>
job 13 at Sat Sep 14 09:30:00 2019
$ at now +2 hours
warning: commands will be executed using /bin/sh
at> ./bar.sh
at> <EOT>
job 14 at Sat Sep 14 11:10:00 2019
$ atq
14      Sat Sep 14 11:10:00 2019 a frank
13      Sat Sep 14 09:30:00 2019 a frank
12      Sat Sep 14 09:15:00 2019 a frank
$ atrm 14
$ atq
13      Sat Sep 14 09:30:00 2019 a frank
12      Sat Sep 14 09:15:00 2019 a frank

The a is the queue. More at options:

Option	Does
`-f file`	read the commands from a file instead of typing them
`-m`	send a mail when the job ends, even with no output
`-q letter`	choose a queue, `a` to `z` or `A` to `Z` (default `a` for at, `b` for batch). Later letters run with a higher nice value
`-v`	show the time the job will run

batch is like at, but runs the job only when the system load is low enough.

Who may use at¶

/etc/at.allow and /etc/at.deny work exactly like the cron files:

Situation	Who may use at
`/etc/at.allow` exists	only the users listed
no `at.allow`, but `/etc/at.deny` exists	everyone except the users listed. Empty means everyone
neither exists	depends on the distribution

systemd timers¶

On systemd systems, timers are an alternative to cron. A timer is a unit file ending in .timer. When it fires, it starts another unit, by default the service with the same name: foobar.timer starts foobar.service.

 /etc/systemd/system/foobar.timer   --- when it fires, starts --->   /etc/systemd/system/foobar.service
   [Timer] OnCalendar=...                                              [Service] ExecStart=the job

The schedule goes in the [Timer] section. This runs foobar.service at 05:30 on the first Monday of every month:

[Unit]
Description=Run the foobar service

[Timer]
OnCalendar=Mon *-*-1..7 05:30:00
Persistent=true

[Install]
WantedBy=timers.target

Then enable and start the timer, as root:

# systemctl enable foobar.timer
# systemctl start foobar.timer
# systemctl enable --now myservice.timer        # both in one step

After changing a timer file, run systemctl daemon-reload so systemd reads it again.

Real-time timers: OnCalendar¶

OnCalendar= works like a cron time, with this format:

OnCalendar=DayOfWeek Year-Month-Day Hour:Minute:Second

DayOfWeek is optional, written as Mon or Monday.
*, / and , mean the same as in cron.
.. gives a range: 1..7 is days 1 to 7. So Mon *-*-1..7 means "a Monday in the first 7 days of the month", which is the first Monday.

Instead of the full form you can use a shortcut:

Shortcut	Runs
`hourly`	at the start of every hour
`daily`	every day at midnight
`weekly`	at midnight on Monday (cron's `@weekly` is Sunday)
`monthly`	at midnight on the 1st of the month
`yearly`	at midnight on the 1st of January

A real timer from a Fedora system, updating the locate database every day:

# cat /etc/systemd/system/timers.target.wants/plocate-updatedb.timer
[Unit]
Description=Update the plocate database daily

[Timer]
OnCalendar=daily
RandomizedDelaySec=12h
AccuracySec=20min
Persistent=true

[Install]
WantedBy=timers.target

Monotonic timers¶

Monotonic timers fire a set time after some event, not at a clock time:

Option	Counts from
`OnBootSec=`	the system boot
`OnActiveSec=`	the moment the timer was activated
`OnStartupSec=`	the start of the service manager (useful for user services, started at login)
`OnUnitActiveSec=`	the last time the service was activated
`OnUnitInactiveSec=`	the last time the service was deactivated

Times can use units like seconds, minutes, hours, days, weeks:

[Timer]
OnUnitActiveSec=5minutes 20seconds

This runs the service again 5 minutes and 20 seconds after each time it was last started.

Checking timers¶

systemctl list-timers lists the active timers, sorted by when they fire next. Add --all to include inactive ones. Timers log to the journal, so journalctl shows what they did. As a normal user working with your own timers, add --user to systemctl and journalctl.

systemd-run: one-time jobs¶

systemd-run creates a temporary timer for a command, without writing any unit files. It is the systemd alternative to at:

# systemd-run --on-calendar='2019-10-06 11:30' date
# systemd-run --on-active="2m" ./foo.sh
# systemd-run --on-active="2hours" /usr/local/bin/helloworld.sh

The first runs date at 11:30 on 2019-10-06. The second runs ./foo.sh two minutes from now, and the third runs the script once, two hours from now. Like the other systemd tools, add --user to run it as your own user, and read its output with journalctl.

More examples¶

The same commands once more, with other names and values, as a quick reference:

A  B  C  D  E  command and arguments

5 0 * * *        $HOME/bin/daily.job      # 00:05 every day
15 14 1 * *      $HOME/bin/monthly        # 14:15 on the 1st of each month
0 22 * * 1-5     echo "it is 10pm"        # 22:00 on weekdays (Mon-Fri)
23 0-23/2 * * *  echo "23 min past every 2nd hour"
*/5 * * * *      echo "every 5 minutes"
42 8,18 * * 1-5  echo "08:42 and 18:42 on weekdays"
@reboot          echo "runs once after each reboot"

$ crontab -l        # list my cron jobs
$ crontab -e        # edit them (validated, then loaded)
$ crontab -r        # remove my crontab

A  B  C  D  E  USER  command and arguments

$ at now + 1 min
warning: commands will be executed using /bin/sh
at> touch /tmp/at
at> <EOT>
job 3 at Thu Oct 29 22:12:00 2015

$ atq
5    Fri Oct 30 22:15:00 2015 a nagato
4    Fri Oct 30 16:00:00 2015 a nagato
$ atrm 4

/etc/cron.allow   /etc/cron.deny
/etc/at.allow     /etc/at.deny

# /etc/systemd/system/myservice.timer
[Unit]
Description=Run my service

[Timer]
OnCalendar=Sat *-*-1..7 02:42:00    # 02:42 on the first Saturday each month
Persistent=true

[Install]
WantedBy=timers.target

# systemctl daemon-reload
# systemctl enable --now myservice.timer
# systemctl list-timers          # add --all to include inactive ones

Summary¶

I pick the tool by the need: cron for repeating jobs, at for one-time jobs, and systemd timers for either kind. cron wakes every minute and reads the crontabs. A user crontab line has five time fields, minute, hour, day of month, month and day of week (where 0 and 7 are Sunday), followed by the command; * means any, , lists, - gives a range and / a step, and shortcuts like @reboot, @daily and @weekly replace the five fields. I manage my own crontab only through crontab -e, -l and -r, root edits other users' with -u, and the files themselves sit under /var/spool/cron/. System crontabs, /etc/crontab and /etc/cron.d/*, are edited directly and have an extra user field, and dropping a script into /etc/cron.hourly, .daily, .weekly or .monthly schedules it at that interval with no crontab line. A job's output is mailed to its owner or to MAILTO, so I redirect what I do not need.

at runs a job once: at now +5 minutes, at 17:30 tomorrow or at teatime, typing the commands and finishing with Ctrl+D. atq lists pending jobs, atrm deletes them, and the atd daemon must be running. Access to both services follows the same rule: an .allow file is a whitelist, so if it exists only the users in it may schedule jobs, otherwise the .deny file is a blacklist and everyone except those in it may, using /etc/cron.allow, /etc/cron.deny, /etc/at.allow and /etc/at.deny.

A systemd timer is a .timer unit that starts the service with the same name. OnCalendar= sets real times in the form DayOfWeek Year-Month-Day Hour:Minute:Second, with .. for ranges and shortcuts like daily and weekly, while monotonic options like OnBootSec= and OnUnitActiveSec= count time from an event. I enable and start the timer with systemctl, run systemctl daemon-reload after edits, check timers with systemctl list-timers and read their logs with journalctl. For a quick one-time job, systemd-run --on-active= or --on-calendar= builds a temporary timer for me.