Take it, prove it, restore it.

How a backup run works

A policy’s schedule queues a run, or you do (ark backups run). Each run writes one backup per target the policy places it on. An agent that has the source type’s programs claims each one, encrypts before upload and reports a signed manifest. TillArk records the manifest, never the data.

run ─▶ an agent that can run it claims the job
        │
        ├─ opens the job's sealed credentials (in memory only)
        ├─ confirms the target's object lock, when the target requires one
        ├─ dumps or streams the source      restic, or WAL-G or pgBackRest for point in time,
        │                                   or clickhouse-backup for ClickHouse table parts
        ├─ encrypts in the agent, uploads   the target only ever holds ciphertext
        └─ signs the manifest               Ed25519 + ML-DSA-65
                │
                ▼
        TillArk verifies both signatures, records the manifest (never the bytes)
        status: succeeded · size · dedup · run epoch

Deduplication and compression come from restic, so a nightly dump stores only what changed. TillArk adds scheduling, retention, the manifest, and a prune that never races a running backup: see Schedules and upkeep. Backups of several sources queued together share a run epoch, so ark backups ls --epoch <n> shows the set taken together. A backup carries a status through pending → running → succeeded, then verifying → verified once a verification passes. A failed backup says why on the Backups page, under the table of ark backups ls, and as error in the API. Each backup also records when it read its data, shown on its row and returned as capture: see One moment.

Schedules and upkeep

A policy’s backup and verification schedules are five-field cron expressions, evaluated in UTC. A schedule that can never fire, such as 0 0 30 2 *, is refused when you save it. While an agent is connected, a slot is queued within about a minute of its time.

tilldev ark policies ls                  # CRON (UTC), NEXT, and any schedule error
tilldev ark policies get <policy-id>
tilldev ark policies update <policy-id> --cron "15 2 * * *"   # 02:15 UTC daily
tilldev ark policies update <policy-id> --disable

The Policies page shows the same next run, last scheduled run and schedule error. In the API, they are next_run_at, last_scheduled_for and schedule_error on each policy.

To manage schedules, retention and the rest for every source from one reviewed file, see Policy files.

Backups that retention no longer keeps are deleted from their target by an agent. That delete never runs while a backup or restore of the same source is using the same target, and those wait while a delete is under way. If the agent doing it goes away, another agent takes the delete over after 30 minutes.

Restore-verification

On your policy’s verification schedule, an agent restores a real backup into a throwaway server of its own, checks what came back against the signed manifest, records how long it took, and removes the server. The server listens only on a private socket (Meilisearch on loopback with a random key), so nothing in your network can reach it, and your systems are never touched.

verify (on the policy's schedule, or on demand)
   │
   ▼
 a restorable backup                 the newest, or in turn an older one
   │                                 never one still being written
   ▼
 check the manifest signatures ──✗──▶ failed · nothing is restored
   │ ✓
   ▼
 restore into a throwaway server on the agent
   │
   ▼
 run the checks for its source type  (below)
   │
   ├─ any check fails, or none ran ──▶ failed
   │
   ▼
 read one part of the repository     scheduled, restic: every blob against its hash
   │
   ├─ damage found ──▶ failed · the damaged pack files are named
   │
   ▼
 passed · the backup becomes "verified" · the restore time is recorded
   │
   ▼
 the throwaway server and its files are removed

Every verification checks the manifest’s signatures and that the stored bytes match its signed hash and size. Then, per source type:

Run one by hand with tilldev ark verify <backup-id> and read the outcome with tilldev ark restores.

Verification at scale

Restoring the newest backup proves the newest backup. Three policy settings make scheduled verification cover everything you keep, without every source restoring at once:

A read locks its repository, so it waits while a backup, a prune or another read is using it, and new backups and restores of that repository wait while it reads. A read that can’t start within 20 minutes is left to a later verification, which the result names; the verification still passes on its restore. Damage a read finds fails the verification, with the damaged pack files named in the repository-read check. Proving an older backup doesn’t move the recovery point, which the newest backup sets.

tilldev ark policies update <policy-id> --verify-spread 120 --verify-older-every 7 --verify-read-parts 60
tilldev ark policies get <policy-id>      # next verify, and what each verification covers
tilldev ark restores                      # PROVED: newest or older, and the part read
tilldev ark status                        # COVERED: kept backups proven, and data read

The Policies page shows each policy’s next verification and its settings, and the Restores page shows which backup each verification proved and the part it read. In the API they are verify_spread_min, verify_older_every, verify_read_parts and next_verify_at on each policy, and sampling on each restore. How far it has got is on the Recovery panel of the TillPulse overview and in tilldev ark status: how many kept backups a verification has proven, when the oldest unproven one was taken, and for each restic repository how many parts are read, any whose read failed, and, once all are read, since when every stored byte has been; coverage in GET /api/ark/status. Reading a part needs an ark-agent from this release or later; an older one restores the backup and reads no part, and a later verification by a current agent reads it.

Real restores are dangerous — and gated

A real restore overwrites a destination and, done carelessly, is a data-exfil vector (restore-to-arbitrary-target). So it carries the strongest gates in the product:

• Four-eyes approval. The requester must name a second, different owner/admin of the same org as approver, plus a non-empty approval_ref (a change ticket / incident id). The server re-verifies the approver against the org’s members — the client’s claim is never trusted, and a token cannot approve its own request.
• Allowlisted destination. You can only restore into a registered source in your own org — not an arbitrary URL.
• Verify before trust. The agent checks the manifest’s signatures, the snapshot it is about to read against the signed one, and the stored bytes against their signed hash before it touches the destination.

# A REAL restore overwrites a destination, so it is four-eyes gated:
# a DIFFERENT owner/admin's user id + an approval reference, and the
# destination must be an ALLOWLISTED registered source (exfil guard).
tilldev ark restore <backup-id> \
  --into <destination-source-id> \
  --approved-by <second-approver-uuid> \
  --approval-ref INC-4821
# → authorized real restore <uuid> → source <dest> (pending)
#   the agent verifies the manifest signature before trusting a byte

How a restore lands depends on the source type; in every case, what the backup doesn’t hold is left alone.

• PostgreSQL: one transaction, so it lands whole or not at all.
• Meilisearch: each index loads beside its live copy and all of them swap in at once.
• ClickHouse: tables load into a private staging database, are checked, then swap in one by one. With clickhouse-backup, the copied parts are first restored and checked in a throwaway server on the agent, and the rows it reads load the same way.
• MongoDB: collections are staged, then swapped in one at a time.
• MySQL and Redis: tables or keys are replaced one at a time.
• Files: restored in place under the agent’s TILLARK_FILE_ROOTS.
• Bucket: every object is written under the destination’s prefix with its content type and metadata, each write checked against the bucket’s MD5. The destination can also be a directory under TILLARK_FILE_ROOTS.

Recover PostgreSQL to any moment

With the WAL-G or pgBackRest engine, the agent streams your database’s write-ahead log to every target that holds its backups, all the time. A restore can then land on any moment between your oldest base backup and a few minutes ago, not only on the moment a backup was taken. Each WAL stream reports how far it reaches: every commit before its recoverable-to time, on the database’s own clock, is archived.

# On the database host, as the postgres OS user (the engine reads the data directory):
TILLARK_ENGINES=restic,walg                # or restic,pgbackrest (pgBackRest 2.51 or newer)
TILLARK_LOCAL_SOURCES=<source-id>          # the sources whose WAL this host archives

-- The role in the source's connection string:
ALTER ROLE ark REPLICATION;
GRANT pg_read_all_settings TO ark;
GRANT EXECUTE ON FUNCTION pg_backup_start(text, boolean), pg_backup_stop(boolean) TO ark;
--   before PostgreSQL 15: pg_start_backup(text, boolean, boolean), pg_stop_backup(boolean, boolean)
GRANT EXECUTE ON FUNCTION pg_switch_wal() TO ark;
-- pg_hba.conf: allow it a replication connection, e.g.
--   host  replication  ark  127.0.0.1/32  scram-sha-256
-- postgresql.conf: cap what a stalled stream may hold back
--   max_slot_wal_keep_size = 20GB

The stream to a target starts with its first base backup, which waits for the stream before it begins so its own WAL is kept. Pausing a source stops new base backups but keeps WAL streaming, so point-in-time recovery stays current and the slot never grows unbounded. Pausing a target stops its stream.

Restore to a moment from the Restores page (Real restore → A point in time) or the CLI. TillArk refuses, with the reason, a moment in the future, before the chosen backup finished, past what is archived, or across a gap. The agent recovers in a private sandbox and copies the database into the destination in one transaction only when recovery succeeds, then reports the last commit it replayed. A moment with no later commit, such as a quiet night, restores too: once the archive provably holds every commit up to it, recovery finishes at the end of the archive. Verifications always recover to the end of their backup.

# Recover to a moment. TillArk picks the newest base backup that finished
# before it and whose engine's WAL archive reaches it, and replays WAL up to it. Four-eyes gated, as above.
tilldev ark restore --source <source-id> --at 2026-09-28T14:05:00Z \
  --into <destination-source-id> \
  --approved-by <second-approver-uuid> \
  --approval-ref INC-4822

tilldev ark wal        # per target and engine: state, recoverable-to time, lag, gaps
tilldev ark restores   # TO: the moment asked for, or the last commit replayed

If you stop archiving a source on a host, remove it from TILLARK_LOCAL_SOURCES and drop its inactive tillark_ replication slots there, or Postgres keeps their WAL.

Incremental and differential base backups

A large database need not be copied whole for every base backup. On WAL-G and pgBackRest sources, a policy can have each base backup store only what changed:

# Each base backup stores what changed since the one before it; a full one at least weekly.
tilldev ark policies update <policy-id> --incremental incremental --full-every 7d

# Or only what changed since the last full backup: a restore reads two backups at most.
tilldev ark policies update <policy-id> --incremental differential

tilldev ark backups ls --source app-db     # KIND: full, incremental on 3f2a91c0, …

A full backup is taken at least every full_every days (default 7), and once a chain holds 64 backups on WAL-G or 1000 on pgBackRest. TillArk picks what each backup builds on from the signed, unquarantined backups of the same source, target, engine and key: the newest full one for a differential backup, the newest in that full one’s chain for an incremental one. Before building on it, the agent checks the stored backup (with WAL-G, every backup its chain holds) against its signed manifest. If it has changed, or is gone, the agent takes a full backup instead, and the catalog says why. The signed manifest records what each backup builds on, and TillArk accepts it only when it is the backup it handed out.

A restore or verification reads every backup its backup builds on and checks each file it reads against the signed manifest, so a file changed anywhere in the chain fails it. A backup stays while anything kept builds on it, whatever its retention says. A backup that builds on a quarantined one is quarantined with it, and dismissing the alarm releases them together. A backup started by hand follows the source’s policies when they agree, and is full when they don’t. Chains need an ark-agent from this release; an older agent takes full backups.

Copy ClickHouse table parts

On a self-managed ClickHouse server, the clickhouse-backup engine (chbackup) copies each MergeTree table’s data parts instead of reading every row, which takes far less of the server on large tables. It freezes the tables, a snapshot made of hard links that copies nothing on the server, then packs the parts, the other tables’ rows and every definition into one backup, encrypted in the agent like any other.

# On the database host, as a user that can read and write ClickHouse's data directory:
TILLARK_ENGINES=restic,chbackup
TILLARK_LOCAL_SOURCES=<source-id>          # the ClickHouse sources this host backs up
TILLARK_CHBACKUP_BIN_DIR=/usr/local/bin    # where clickhouse-backup is, if not on PATH

-- The user in the source's connection string (not readonly):
GRANT SELECT, SHOW, ALTER FREEZE PARTITION ON *.* TO ark;

tilldev ark sources update events --engine chbackup

Back up a TillForge git node

A TillForge git node source backs up a git server’s repositories with its own tillforge backup: every repository’s history, its Git LFS objects and deleted repositories awaiting purge, each sealed under its organization’s key before the agent reads it, then encrypted in the agent like any other backup. The manifest signs how many refs and LFS objects each repository holds.

# On the git node, as the node's user:
TILLARK_LOCAL_SOURCES=<source-id>          # the git nodes this host backs up
TILLARK_FORGE_BIN_DIR=/usr/local/bin       # where tillforge is, if not on PATH

tilldev ark sources create --name git-eu-1 --type forge --provider self --unmanaged --with-dsn
#   tillforge:///etc/tillforge/tillforge.env
tilldev ark policies create --source <source-id> --cron '0 1 * * *' --verify-cron '0 4 * * *' \
  --residency eu --allow-same-region

Restore a run’s stores together

Backups queued for the same scheduled slot form a run. A run’s page (open it from the Backups page), tilldev ark runs or /api/ark/runs/{epoch} verifies or restores its stores as one set, says whether they come back as of one moment, and checks the references you declare between them.

One moment

A run is consistent when every store comes back as of the same moment, the run’s point. How each store was read decides whether it can:

The point is the moment of the one store read as a snapshot, when there is one; otherwise the latest moment every versioned bucket covers, or the latest base backup’s end. A run holding a store read over time, or two stores each read as its own snapshot, isn’t consistent: the run’s page, tilldev ark runs show and the API name each such store and say why, and the others still come back at the point.

The agent times every backup on TillArk’s clock, measured on each request it makes, and reads each database’s and bucket’s own clock, so a point-in-time database and a versioned bucket recover to the point on their own clocks. The plan shows how far apart any two stores’ data may be from clock uncertainty: tens of milliseconds between databases, and about a second with a bucket, whose clock is read to the second. A versioned bucket restores each object’s version current at the point; an object gone by then comes back from its nearest version within that uncertainty. So when your application stores an object before the row that names it, and removes the row before the object, no restored row names a missing object, and the references you declare check exactly that.

The plan shows each store’s moment before anything starts. A store whose backup can’t be restored is left out, with the reason. When a base backup finished after the point, an older one that reaches it is used; when none does, that database comes back to its own end, and the plan says so. A run passes when every store passed and every reference was checked and found whole.

References

A reference declares that a column of a PostgreSQL source names objects in an object source (a bucket or a directory). Declare it on the database’s source page or with tilldev ark sources refs.

A verification checks a reference in the database’s sandbox against the object backup, after checking the object backup against its signed hash. A restore checks it once both stores are restored, on an agent that reaches both destinations. The agent reads the column from one read-only snapshot, as text, never through a cast the database defines. Neither side has to fit in memory.

A restore’s check reads as the role its database’s source connects with. When row-level security would hide rows from that role, the check fails and says so rather than checking only the rows it could see: connect as the table’s owner or a role that bypasses row-level security.

Each check reports the rows and values it read, values that name no object (empty, or without the strip prefix), the objects listed, the missing keys and the rows naming them, with up to five examples. Any missing key fails the run and raises a run.dangling signal; every finished run raises run.passed or run.failed.

# A PostgreSQL column that names objects in an object source:
tilldev ark sources refs app-db add --name avatars --column users.avatar_url \
  --to uploads --strip https://cdn.example.com/ --key 'avatars/{value}'

tilldev ark runs                        # each run, and when its stores last came back together
tilldev ark runs verify 1790208000      # every store in a sandbox, references checked
tilldev ark runs show 1790208000        # the plan, each store's outcome, missing objects
tilldev ark runs restore 1790208000 \
  --into app-db=app-db-restore,uploads=uploads-restore \
  --approved-by <second-approver-uuid> --approval-ref INC-4823

Recovery point, measured

Each source shows its recovery point: the newest moment a restore could bring back right now. It comes from the newest backup that is signed, not quarantined and restorable, or, for a source on WAL-G or pgBackRest, from how far its WAL archive reaches past a base backup. A backup that failed, or one you can’t restore, never counts.

tilldev ark sources ls                  # RECOVERY POINT: age / objective
tilldev ark sources get <source-id>     # recovery point, objective, breached since
tilldev ark policies update <policy-id> --rpo 7200

GFS retention

Retention is grandfather-father-son: keep the last N, plus a thinning tail of hourly / daily / weekly / monthly / yearly snapshots. Retention depth is a correctness property — it’s what lets you roll back past a logical corruption (a “time-bomb” a fresh backup would faithfully preserve).

Retention applies per target: each target keeps its own full set, so a second target adds depth rather than splitting it. With several policies on one source, a backup is deleted only when every policy would let it go. A backup that another kept backup builds on stays until nothing kept builds on it. Retention is applied about every 10 minutes.

# GFS defaults — override any bucket with a --keep-* flag on the policy.
tilldev ark policies create --source <id> \
  --keep-last 7 --keep-hourly 24 --keep-daily 7 \
  --keep-weekly 4 --keep-monthly 12 --keep-yearly 3

# Change one bucket later; the others stay as they are.
tilldev ark policies update <policy-id> --keep-daily 14

Limits by source type