Why a successful backup can still be unrecoverable

A backup dashboard shows ninety consecutive successful runs. Then an operator deletes a customer workspace and discovers that the restored application has no recent uploads. The job was configured to protect /srv/app, but a deployment moved uploads to a new Docker volume two months earlier. Every green result was truthful: the tool read and stored the path it was given.

Backup software can report command completion, transferred bytes, and a new snapshot. It cannot infer whether the selected paths represent the whole business service. “Successful” is therefore evidence about execution, not a blanket guarantee of recovery.

Reliable recovery needs several layers to agree: correct scope, application consistency, repository integrity, credential availability, compatible infrastructure, and a procedure that operators can execute. A failure in any one can leave healthy-looking backup history and no usable service.

Scope errors are common because applications change while backup definitions stay still. Upload directories move, containers gain volumes, databases are renamed, a new service appears, or a mounted disk is missing when the job runs. Old paths often remain readable, so the backup does not fail.

Look beyond the top-level directory. Confirm that separate filesystems were mounted, symbolic links behaved as expected, container mounts are covered, exclusions did not grow too broad, and permissions allowed the backup identity to read every required file. Restic documents a distinct exit status when source data could not be read, but operators still need to treat partial reads as failed coverage rather than an acceptable warning.

Compare the protected inventory with current application reality after deployments and migrations. Then inspect restored content for a recognizable recent record or file. That content-level check catches stale-but-valid sources that byte and job counters cannot.

Live databases coordinate writes across data files, logs, and memory. Copying their storage at arbitrary moments can capture a state the database engine cannot safely replay. The filesystem backup may be internally perfect while the database inside it refuses to start or contains inconsistent application state.

Use the database engine’s supported backup procedure. PostgreSQL’s pg_dump, for example, creates a consistent logical backup while the database remains in use. Physical and point-in-time methods have their own requirements, including version compatibility and complete write-ahead-log history.

Consistency also spans components. A database recovered from noon and uploads recovered from midnight can produce records that reference files that do not exist. Define whether components need a coordinated recovery point, and validate their boundaries together during a rehearsal.

Encrypted backups deliberately separate stored ciphertext from the knowledge required to decrypt it. With restic, losing every working repository password means the data is irrecoverably lost. A storage bucket full of intact pack files cannot compensate for missing key access.

Storage credentials and encryption credentials solve different problems. The first authorizes access to repository objects; the second unlocks the encrypted repository. Both need a recovery path that survives loss of the protected server, the primary operator, and the normal identity provider where practical.

Test credential custody without exposing secrets in the test record. A second authorized person should be able to locate approved recovery material, access the storage account, and open the repository from a clean environment. If the procedure depends on a former employee’s phone or one browser session, the backup is not team-owned.

Recent recovery points protect against immediate failure; history protects against problems discovered late. Slow corruption, malicious changes, and accidental deletion may remain unnoticed for weeks. A short retention window can leave many successful snapshots that all contain the same damaged state.

Set retention from detection time and recovery needs, not only storage price. Keep multiple time horizons where the risk justifies them, and understand how policies group snapshots. Test policy changes before they delete history. Retention is a destructive lifecycle operation, so its success is not the same as backup success.

Independence matters too. Backups in the same provider account or writable with the same compromised credential can be deleted along with the server. Versioning, object-lock controls, separate credentials, or a second copy can reduce that shared failure domain, depending on the threat model.

Applications depend on ownership, permissions, symbolic links, service accounts, configuration, secrets, certificates, network names, and compatible software. Recovering only content can leave a directory that looks complete but cannot be served safely.

Database dumps have context as well. Roles, extensions, locale behavior, server versions, and object ownership can affect restoration. PostgreSQL notes that owners and granted users may need to exist before a dump is restored. An operator needs to know which prerequisites are part of the artifact and which must be recreated.

Record software and infrastructure dependencies alongside the backup set. Prefer reproducible definitions for rebuildable infrastructure, but test them with recovered data. A runbook copied from three releases ago can be as damaging as a missing file when it applies the wrong migration or starts an obsolete service.

A restore command can finish successfully while the chosen snapshot is wrong, the database is empty, or the application cannot process a real request. Verification should move from storage to service: inspect files, load the database, start the application in isolation, and perform representative read operations.

Choose validation that reflects business use. Count important records, open recent uploads, confirm expected schema, and check that authentication and encryption dependencies work. Keep real outbound integrations disabled until old queued work has been examined.

Measure the whole exercise. If obtaining credentials and rebuilding infrastructure pushes recovery beyond the promised time, the backup may be technically recoverable but operationally inadequate. Recovery objectives are proven by timed tests, not inferred from upload speed.

Use a ladder of evidence rather than one green badge. First, monitor every run and treat partial source reads as failures. Second, alert when no usable recovery point appears on schedule. Third, run repository integrity checks. Fourth, restore representative files and databases. Finally, rehearse the complete application on a clean target.

Each layer catches a different defect and none replaces the others. Repository checks cannot know that a path was omitted. Application validation cannot cover every stored pack. A full disaster exercise run once three years ago cannot detect last week’s deployment drift.

When a test fails, preserve the result and assign corrective work. The uncomfortable finding is the value of the exercise. A backup system becomes trustworthy through repeated evidence and repaired gaps, not through an uninterrupted streak of status icons.