VPS backup restore test checklist: prove recovery before an outage

A VPS backup restore test should answer one operational question: **can your team restore a working service from a known recovery point before the outage becomes unacceptable?** Downloading a few files or confirming that a backup job succeeded does not answer it. A useful test rebuilds the service in an isolated target, restores the required state, and checks the behavior users depend on.

Start with a plausible scenario. For example: a deployment at 14:20 corrupts the production database; the VPS still boots, but the application serves errors and the last trustworthy transaction happened shortly before the deployment. This forces you to choose a recovery point, account for data that may be lost, restore the application’s dependencies, and decide when the recovered service is safe to use.

Write the scenario in one sentence before the test starts. State what failed, what is assumed unavailable, which backup copy remains reachable, and what “recovered” means. A marketing site may be recovered when pages, assets, and its contact form work. A billing system may also require authentication, representative customer records, scheduled work, and payment reconciliation checks.

Use a non-production destination. A rehearsal must not overwrite the live server or allow a recovered application to contact production queues, email providers, payment gateways, webhooks, or third-party APIs. Isolation makes errors cheaper, prevents old jobs from being replayed, and gives the team room to inspect restored data.

• Good scenario: “The database is corrupted after a deployment; recover the service using the last safe backup and report the resulting data-loss window.”
• Weak scenario: “Restore a random folder and confirm that files exist.”
• Failure signal: the team cannot explain which components must be restored, who may approve recovery, or what evidence proves the service is usable.

Define the pass criteria before looking at the newest backup. Your recovery point objective (RPO) is the maximum data-loss window the service can accept. Your recovery time objective (RTO) is the maximum acceptable time to restore service. They are different constraints: taking backups more often can improve RPO, while faster infrastructure provisioning, smaller data sets, practiced procedures, and automation can improve RTO. For help setting these limits, see [RPO vs RTO for small teams](/blog/rpo-vs-rto-small-teams-backup-frequency).

Make the pass condition observable. For a typical application, that might mean the operating system and required runtime are available; configuration is present; the database artifact restores without errors; representative uploads render; an administrator can sign in; a read-only report returns expected data; and outbound integrations remain disabled. Include any business-specific reconciliation needed before a production cutover.

The selected recovery point may expose an RPO gap. If the incident began at 14:20 and the newest safe database export is from 11:15, the exercise has demonstrated a three-hour-and-five-minute data-loss window. Record that result honestly. Do not rewrite the scenario around the available backup merely to turn a gap into a pass.

• RPO: the maximum amount of data created since the chosen recovery point that you can afford to lose.
• RTO: the time from declaring the incident to reaching the approved recovered state, including access, provisioning, restoration, validation, and approval.
• Record: scenario start time, chosen recovery-point timestamp, target RPO and RTO, pass criteria, and the operator responsible for the test.

Choose a specific recovery point and state why it is safe. “Latest” is ambiguous when several hosts, database exports, tags, or backup sets share a repository. The newest filesystem snapshot might include an incomplete deployment, while an earlier database dump is the last known-good application state. Record the timestamp, source host, included paths, database artifact name, snapshot ID or tag, and reason for selection.

Inspect the backup scope before restoring. A working application commonly needs more than its code directory: configuration, secrets or a documented way to obtain them, database-consistent backup artifacts, user uploads, infrastructure definitions, and recovery documentation. Use a current [Linux server backup scope checklist](/blog/what-to-back-up-linux-server) if the team is unsure what belongs in the recovery set.

Confirm that the recovery operator can independently reach the repository from the test environment. Storage credentials, repository location, and the repository password or recovery key are separate dependencies. A backup is not operationally recoverable if all of those details exist only on the failed VPS or in one unavailable person’s account. Protect recovery-key ownership deliberately; an encrypted repository cannot be recovered if its required key is lost.

A clean repository integrity check is useful evidence about stored data, but it does not prove that the backup scope is complete or that the application will start. Read [what restic repository health checks prove—and what they do not](/blog/restic-repository-health-integrity-checks) before treating a successful check as a complete recovery test.

• Record the snapshot or backup-artifact identifier rather than relying on a relative label such as “last night.”
• Record where the repository is, how the test operator receives storage access, and where the recovery key is held without placing secrets in the test record.
• Failure signal: the selected backup has no documented host, scope, timestamp, or safe relationship to the failure scenario.

Provision an empty target that resembles a machine you could realistically obtain during an incident, rather than a carefully prepared copy of production. Record the operating-system version, architecture, disk capacity, network restrictions, package repositories, container runtime if relevant, and required private image-registry access. A recovery test should reveal hidden dependencies on a particular package version, firewall rule, image, or operator workstation.

Keep the target clean enough that restored data cannot be confused with leftovers. Old test databases, cached uploads, mounted volumes, and forgotten environment files can make an incomplete restore appear successful. Label the environment with the test date and scenario. Do not attach production DNS, production credentials, or a load balancer route that can send real traffic to it.

Block or redirect side effects before application startup. Disable workers and schedulers, use non-production email and webhook endpoints, remove payment-provider credentials, and prevent queue consumers from running. If the application cannot be started safely without production integrations, that is a recovery design issue to document and fix.

Start timing before infrastructure work begins. Recovery time includes declaring the incident, finding documentation, obtaining access, provisioning a target, restoring data, starting services, validating behavior, and receiving approval. Measuring only backup download time produces a number that will not help during a real outage.

• Create a separate project, network, or account boundary where practical.
• Use a fresh restore directory and an empty database service for the rehearsal.
• Restrict inbound access to the test team and restrict outbound access to only what the test requires.
• Failure signal: a restored service could send customer email, consume a live queue, process payments, or receive production traffic.

Restore into a new, intentional directory first. This protects the test target, preserves the recovered evidence for inspection, and makes it easier to identify missing paths, unexpected permissions, or an incorrect snapshot. Do not point a restore at a production path or use a command that overwrites current data simply to save a later copy step.

The illustrative restic workflow below assumes the recovery target has the restic binary, network access to the configured repository, repository credentials supplied through your approved secret mechanism, sufficient local disk space, and a directory dedicated to the rehearsal. Adapt it to your repository backend and test it outside production first. Do not paste passwords, access keys, or recovery keys into shell history, tickets, or command transcripts.

First, list and inspect the candidate snapshots. The expected result is a snapshot identifier, host, timestamp, and paths that match the recovery record. If the expected host or paths are absent, stop and correct the recovery-point selection rather than restoring an arbitrary newer snapshot.

bash # Illustrative only: run on the isolated recovery target. # Supply RESTIC_REPOSITORY and RESTIC_PASSWORD_FILE through your approved # secret-handling method; do not put secret values in this command. restic snapshots restic ls <snapshot-id>

• Check that the listed paths contain the expected application configuration, persistent files, and database backup artifacts—not only application source code.
• If repository health has not recently been checked, plan an appropriate restic check according to repository size and available bandwidth; a data read check can take significant time and storage-provider requests.
• Failure signal: the only usable snapshot is older than the agreed RPO, contains a different host’s paths, or lacks a component required by the runbook.

Create an empty directory owned by the test operator, then restore the exact snapshot into it. The restore command below writes a recovered filesystem tree below the target directory; review restic’s path layout before moving anything into service locations. The expected result is a successful command exit and a recovered tree that can be inspected without affecting the running system.

bash # Illustrative only: run on an isolated target after verifying <snapshot-id>. # Ensure /srv/recovery/restore is a new, disposable directory with enough space. sudo install -d -m 0700 /srv/recovery/restore restic restore <snapshot-id> --target /srv/recovery/restore # Inspect the restored tree before copying or mounting any recovered data. sudo find /srv/recovery/restore -maxdepth 3 -type f | sed -n '1,80p' sudo du -sh /srv/recovery/restore

Inspect ownership, permissions, symbolic links, and expected files. Depending on the source and restore target, you may need to map paths, restore as a privileged user, or deliberately set service-account ownership before startup. Do not treat a nonzero file count as validation: a restored tree can still lack a required upload directory, an environment file, a mounted volume, or a database export.

For containerized services, map Compose named volumes and bind mounts before restoring. A Compose file can recreate containers, but it does not recreate the persistent state around them. Use [Docker Compose backup strategy: volumes, databases, and restore order](/blog/docker-compose-backup-strategy) and [how to detect Docker volume and mount coverage drift](/blog/detect-docker-volume-mount-backup-coverage-drift) to compare the currently deployed mounts with the paths your backup actually captured.

• Check available disk space before the restore and record restore duration separately from provisioning time.
• Confirm that restored files are from the selected point in time, especially database exports and recent user uploads.
• Failure signal: the restore command succeeds but expected paths are empty, files have unusable ownership, or the recovered tree does not match the application’s current deployment layout.