How to monitor restic backups and alert on failed scheduled jobs

The obvious failure is a restic process returning non-zero. The more dangerous one is absence: cron was removed, a timer was disabled, the host was down, credentials expired before the script started, or the monitoring wrapper itself stopped running. There is no failed command to alert on in those cases.

Track two signals for each backup set: the outcome of the last attempt and the timestamp of the last success. The first gives fast feedback on a bad run. The second catches a job that never ran and a string of failures that somebody acknowledged but did not fix.

Set the stale threshold from the schedule, normal runtime, and a small grace period. A daily backup that usually takes 35 minutes might alert after 27 hours, not at 24 hours and one second. Record the agreed threshold next to the alert rule so an operator can tell whether the delay is intentional.

Restic documents its exit codes for scripts. Exit code 0 is success. Code 3 can mean that backup could not read some source data, which is not a clean backup even if restic stored everything else it could read. Codes for a missing repository, a lock, a wrong password, cancellation, and unknown future failures also need attention.

Publish a new last-success signal only after the entire workflow that defines a usable backup has completed. Evaluate that signal outside the protected host, otherwise a dead server can silence both the job and its monitor.

Cron can run the script, but systemd gives the job a named service, journal logs, dependency handling, and a queryable result. Persistent=true tells a calendar timer to trigger a missed run after the machine comes back. RandomizedDelaySec spreads load when many servers share a schedule or storage endpoint.

The timer still cannot report that its own host is permanently down. Pair local service history with an external stale-success check and test that a missed run becomes visible without relying on the server being protected.

A dead-man check must be evaluated somewhere that survives the server you are protecting. It can be a hosted heartbeat service, a small endpoint in your own monitoring system, or a central backup control plane. Send the success signal only after the backup workflow is complete. Alert when that timestamp becomes older than the recovery-point window.

Keep immediate failure and staleness as separate alerts. A failure notification says what just broke and can carry the exit code or a short error. A stale-backup notification says there has been no usable recovery point for too long. Separating them makes routing and escalation much clearer.

Test the absence path. Temporarily point a non-production timer at a harmless failing command, then disable that timer and wait through an accelerated stale threshold. If both alerts arrive and resolve as expected, you have tested more than the happy path.

Grafana is useful when Prometheus and Grafana already operate your servers. The essential fleet signal is the age of the last successful recovery point, accompanied by the latest attempt result and duration.

That approach still requires collectors, stable labels, alert rules, and deliberate No Data behavior. A vanished host must not look healthy just because its metric disappeared.

• Configure Grafana's No Data behavior deliberately. A vanished host or collector must not look healthy because the query returns nothing.
• Route data-source errors separately from backup failures so a Prometheus outage does not masquerade as hundreds of broken repositories.
• Link the alert to the service journal, backup run log, host, repository, and a short recovery runbook.

Rested records scheduled and manual runs in one history with their outcome, logs, snapshots, and expected schedule. It can notify operators when a backup or connected server needs attention.

Current channels and notification policies belong in the product documentation, where they can track releases. Rested remains a backup control plane rather than a replacement for general host and service monitoring.

A useful page names the host, backup set, last successful recovery point, failed stage, and where to find the unredacted local log. It should not include repository passwords, storage secrets, database passwords, or an entire noisy command transcript.

Run periodic alert drills alongside restore testing. Confirm that the expected notification reaches the right person, the runbook still works, and the alert resolves after recovery. Delivery is part of the backup system; an untested channel is another silent dependency.