How to Monitor systemd Timers and Get Alerted on Missed Runs

To monitor a systemd timer, add an ExecStartPost= line to its service that pings a heartbeat URL, and alert when that ping does not arrive on schedule. For a Type=oneshot service, ExecStartPost= only runs after ExecStart= exited successfully, so the ping means "the job ran and worked". Everything else, from a failed script to a timer that was never enabled, shows up as a missing ping.

I'm Léo, I build Hyperping. This guide covers the ways systemd timers fail without anyone noticing, the native commands to check them, and the unit files I use with Hyperping healthchecks. I ran every unit below on systemd 249 (Ubuntu 22.04) with a real timer, and checked the man pages of systemd 262, the current release.

Key takeaways

• ExecStartPost= in a Type=oneshot service runs only when ExecStart= succeeded. Prefix it with - so a failed ping never marks a successful job as failed.
• A oneshot service has no start timeout by default (TimeoutStartUSec=infinity). A hung run stays active, and the timer cannot start the next one.
• OnCalendar= accepts an IANA timezone since systemd 235: *-*-* 02:00:00 Europe/Paris.
• Persistent=true runs a missed job once at boot. Without it, a run missed while the machine was off is skipped.
• OnFailure= reacts to failures, but not to a timer that never fired. A heartbeat catches both.

How systemd timers fail silently

A timer unit (backup.timer) starts a service unit (backup.service) at the times set by OnCalendar=. Errors land in the journal, and the journal does not page anyone.

The timer was never enabled

systemctl enable "does not have the effect of also starting any of the units being enabled", per the systemctl man page. You need systemctl enable --now backup.timer, on the timer, with WantedBy=timers.target in its [Install] section. Enabling backup.service does nothing for the schedule. After a server rebuild from a config management role that forgot the --now or the .timer, the job just never runs.

User timers (systemctl --user) have a second trap: the user manager "is generally started on first login only". Without loginctl enable-linger <user>, the timer stops when the user logs out.

A hung oneshot blocks the next runs

The systemd.timer man page says that if "the unit to activate is already active at the time the timer elapses it is not restarted, but simply left running". By default, the timer triggers again as soon as the service finishes, since the missed elapse is now in the past. systemd 257 added DeferReactivation=true to wait for the next scheduled time instead.

Combine that with the oneshot default I measured on systemd 249:

$ systemctl show backup.service -p Type -p TimeoutStartUSec
Type=oneshot
TimeoutStartUSec=infinity

A backup stuck on a dead NFS mount stays in activating forever. Each night the timer elapses, finds the service active, and does nothing. systemctl --failed stays empty because nothing failed.

Failures stay in the journal

When ExecStart= exits non-zero, the unit goes to failed, the journal records Failed with result 'exit-code', and the timer keeps firing on schedule. Unless something reads the journal, the next failure is just as silent.

Missed runs while the machine is off are skipped

Without Persistent=true, a run scheduled while the machine was powered off or rebooting is not made up. With it, "the service unit is triggered immediately if it would have been triggered at least once during the time when the timer was inactive". It catches up once, not once per missed run, and it only works with OnCalendar=.

The timezone is the server's

OnCalendar=*-*-* 02:00:00 uses the system's local timezone. On servers set to UTC, a job meant for 2:00 in Paris runs at 3:00 or 4:00 local time depending on the season. Since systemd 235, you can append the timezone. systemd-analyze shows what it means:

$ systemd-analyze calendar --iterations=2 '*-*-* 02:00:00 Europe/Paris'
Normalized form: *-*-* 02:00:00 Europe/Paris
    Next elapse: Fri 2026-10-09 02:00:00 CEST
       (in UTC): Fri 2026-10-09 00:00:00 UTC
       From now: 4h 9min left
       Iter. #2: Sat 2026-10-10 02:00:00 CEST
       (in UTC): Sat 2026-10-10 00:00:00 UTC
       From now: 1 day 4h left

How to check systemd timers with systemctl and journalctl

# Every timer, its next and last run, and the service it starts
systemctl list-timers --all

# Result of the last run: success, exit-code, timeout...
systemctl show backup.service -p Result -p ExecMainStatus -p ActiveState

# Output and errors of the last runs
journalctl -u backup.service -n 50

# Units in the failed state
systemctl --failed

After a successful run, list-timers reads like this:

NEXT                         LEFT    LAST                         PASSED UNIT         ACTIVATES
Fri 2026-10-09 02:00:00 CEST 23h left Thu 2026-10-08 02:00:00 CEST 2s ago backup.timer backup.service

LAST tells you the timer fired. It does not tell you the service succeeded, and none of these commands send anything anywhere.

How to monitor systemd timers with Hyperping

A Hyperping healthcheck gives the job a secret URL. If the success ping does not arrive by the scheduled time plus a grace period, Hyperping opens an incident and alerts you. It resolves on the next successful ping. Healthchecks are included on every plan, Free included.

1. Create a healthcheck from the OnCalendar expression

In Hyperping, open Healthchecks, click Create healthcheck, and pick Cron. Translate OnCalendar= into a five-field cron expression and pick the same timezone:

For monotonic timers (OnBootSec=, OnUnitActiveSec=1h), use simple mode with the same interval: every 1 hour.

2. Store the ping URL in an EnvironmentFile

sudo install -d -m 0700 /etc/hyperping
echo 'HC_URL=https://hc.hyperping.io/tok_your_backup_token' | sudo tee /etc/hyperping/backup.env
sudo chmod 0600 /etc/hyperping/backup.env

The URL is the only credential for that healthcheck, so keep it out of world-readable unit files.

3. Ping /start in ExecStartPre and success in ExecStartPost

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly database backup

[Service]
Type=oneshot
EnvironmentFile=/etc/hyperping/backup.env
TimeoutStartSec=1h
ExecStartPre=-/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null ${HC_URL}/start
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=-/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null ${HC_URL}

# /etc/systemd/system/backup.timer
[Unit]
Description=Run the nightly backup

[Timer]
OnCalendar=*-*-* 02:00:00 Europe/Paris
Persistent=true

[Install]
WantedBy=timers.target

The systemd.service man page is explicit: "ExecStartPost= commands are only run after the commands specified in ExecStart= have been invoked successfully, as determined by Type= (i.e. ... the last ExecStart= process exited successfully for Type=oneshot...)". The - prefix matters on both curl lines: without it, a failed ping is a failed command, and an unreachable endpoint would mark a good backup as failed.

Here is what I got on systemd 249, with the timer set one minute ahead in another timezone and a local listener in place of Hyperping:

${HC_URL}/start expands correctly inside ExecStartPre=, and the timer fired at the right minute with Asia/Tokyo in OnCalendar=.

4. Add a start timeout and enable the timer

TimeoutStartSec=1h in the service above turns a hung run into a failed one, which frees the timer for the next night. Size it well above the job's longest normal run.

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer
systemctl list-timers backup.timer

If you want a local reaction on failure too, OnFailure= starts another unit when the service enters the failed state. I tested it with a template unit, OnFailure=backup-failed@%n.service, which received the failing unit's name as %i. Use it to save the journal or clean up a lock file. The alert itself does not depend on it: the missing success ping is enough.

OnSuccess= (systemd 249 and later) can run the ping from a separate unit instead of ExecStartPost=. I prefer ExecStartPost= because it works on every systemd still in support and keeps the whole job in one file.

5. Size the grace period and route the alerts

In cron mode, Hyperping expects the success ping by the scheduled time plus the grace period. Cover the job's run time, plus RandomizedDelaySec= if you set one, plus a margin: AccuracySec= defaults to one minute. A 15 minute backup with no random delay is fine with 30 minutes of grace. The healthcheck's ping history shows the duration of each run between /start and success, so you can tighten it after a week.

With Persistent=true, a run missed while the server was off alerts at 02:00 plus the grace period, and the catch-up run at boot closes the incident.

Alerts go to every channel connected to the project at once: email and SMS to every member, Slack, Discord, Telegram, PagerDuty and Opsgenie. Healthchecks do not use escalation policies or on-call schedules, so send them to PagerDuty or Opsgenie when a missed backup should page the person on call.

Test it once by hand

Run sudo systemctl start backup.service and check that both pings appear in the healthcheck's last pings, with a curl user agent. Then make the script exit 1, start it again, and wait for the alert after the grace period. Restore the script and start the service: the incident closes.

The same heartbeat covers jobs scheduled elsewhere: Kubernetes CronJobs, the Laravel scheduler, GitHub Actions scheduled workflows and Windows Task Scheduler. If the timer runs on a server you manage, my Ubuntu server monitoring guide covers the host metrics that usually explain a failed job. The job these timers run most often is a backup: monitoring database backups shows how to check the dump before the success ping.