A small binary that runs where your data lives.

Claim, run, sign, report

start ─▶ register keys ─▶ wait for approval ─▶ heartbeat ─▶ claim a job
                                                (what this host can run)
      ┌──────────────────────────────┬──────────────────────────────┐
    backup                         verify                        restore
      │                              │                              │
  open this job's credentials    check the manifest signatures  check the manifest signatures
  check the bucket's object lock restore into a throwaway server check the stored hash
  dump or stream the source      run the checks, then remove it restore into the destination
  encrypt, upload                                                    │
  sign the manifest                  │                              │
      └──────────────────────────── report ────────────────────────┘

Every heartbeat reports which source types the host can back up, verify and restore, from the programs it finds. TillArk sends a job only to an agent that can run it, and routes WAL-G, pgBackRest and clickhouse-backup work for a database on a particular host to the agent on that host. A reference check after a run is restored goes to an agent that reaches both restored stores.

Install

Builds are published for Linux and macOS on amd64 and arm64: one static binary, no runtime to install. The install script picks the build for your host, verifies the release signature and the checksum, and stops if either fails. It needs curl and OpenSSH 8.2 or newer.

curl -fsSLO https://tilldev.dev/downloads/ark-agent/install.sh
less install.sh     # read it first
sh install.sh       # installs /usr/local/bin/ark-agent

Pin a version with ARK_AGENT_VERSION=<version> sh install.sh, or install somewhere else with ARK_AGENT_DIR=$HOME/bin. To verify by hand:

V=$(curl -fsSL https://tilldev.dev/downloads/ark-agent/LATEST)
F=ark-agent_${V}_linux_amd64           # or linux_arm64, darwin_arm64, darwin_amd64
curl -fsSLO https://tilldev.dev/downloads/ark-agent/allowed_signers \
     -O https://tilldev.dev/downloads/ark-agent/$V/SHA256SUMS \
     -O https://tilldev.dev/downloads/ark-agent/$V/SHA256SUMS.sig \
     -O https://tilldev.dev/downloads/ark-agent/$V/$F

# The signature proves TillDev built this list; the checksum ties your file to it.
ssh-keygen -Y verify -f allowed_signers -I releases@tilldev.dev -n ark-agent-release \
  -s SHA256SUMS.sig < SHA256SUMS
grep " $F\$" SHA256SUMS | sha256sum --check    # macOS: shasum -a 256 --check
sudo install -m 0755 "$F" /usr/local/bin/ark-agent

Run it as a service

Mint a token with tilldev ark tokens create --scope operator. An operator token lets the agent claim and report jobs but not change your sources, targets or policies. Run the agent as its own unprivileged user: PostgreSQL and MySQL verification refuse to run as root.

# /etc/systemd/system/ark-agent.service
[Unit]
Description=TillArk agent
After=network-online.target
Wants=network-online.target

[Service]
User=ark-agent
EnvironmentFile=/etc/ark-agent/env
ExecStart=/usr/local/bin/ark-agent
Restart=on-failure
RestartSec=5
StateDirectory=ark-agent
StateDirectoryMode=0700
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=full

[Install]
WantedBy=multi-user.target

sudo useradd --system --home-dir /var/lib/ark-agent --shell /usr/sbin/nologin ark-agent
sudo install -d -m 0700 /etc/ark-agent
sudo tee /etc/ark-agent/env >/dev/null <<'EOF'
TILLARK_CP_URL=https://ark.tilldev.dev
TILLARK_AGENT_TOKEN=tark_…
EOF
sudo chmod 0600 /etc/ark-agent/env
sudo systemctl daemon-reload
sudo systemctl enable --now ark-agent
journalctl -u ark-agent | grep key_fingerprint

If the agent restores files, add each of its TILLARK_FILE_ROOTS to the unit as ReadWritePaths=. Outside systemd, run ark-agent with the same environment; it stops cleanly on SIGTERM.

Work only TillArk signed

The agent acts only on work signed by TillArk’s control-plane key: jobs, WAL archiving leases and clone teardowns. Each reply is signed with Ed25519 and ML-DSA-65 over its body and a nonce the agent sent, so a reply can’t be altered, replayed or moved to another request. Both signatures must verify.

On first contact the agent records the key in cp-key.pin in its data directory and logs the fingerprint. From then on it refuses work under any other key. To pin it before first contact, set TILLARK_CP_KEY to the fingerprint under TillArk → Agents or from tilldev ark cp-key; the install steps on the Agents page put it in the run command.

Environment variables

The startup log prints the configuration without the token.

What each source type needs on the host

Dumps and files are stored with restic 0.17 or newer, which must be on PATH. PostgreSQL sources on the WAL-G engine use WAL-G 3.0 or newer instead, the PostgreSQL build (wal-g-pg), and those on the pgBackRest engine use pgBackRest 2.51 or newer. ClickHouse sources on the clickhouse-backup engine use clickhouse-backup 2.5 or newer, and their archives are stored with restic. Each source type then needs:

PostgreSQL’s pg_dump must be at least as new as the server. TillArk → Agents shows what each host can run; a job goes only to an agent that has its programs.

Keys that stay on your hosts

A source can be put under a customer key: 32 random bytes kept in a file on your agent hosts. The agent derives each repository’s password from it, so TillArk never holds a key that opens those backups. See Security for what that protects.

# As the agent's user, so the agent can read the key
sudo -u ark-agent ark-agent byok-keygen     # prints the key's fingerprint
sudo -u ark-agent ark-agent byok-list       # every key this host holds

# Copy it to each other host that runs the source's jobs
sudo cat /var/lib/ark-agent/byok/<fingerprint>.key              # on this host
sudo -u ark-agent ark-agent byok-import < <fingerprint>.key     # on the other host

# Then put the source under it
tilldev ark sources update <source-id> --byok <fingerprint>

• Run the commands as the user the agent runs as. A key file is readable by its owner only, so a key written by root is one the agent can’t use. If you set TILLARK_AGENT_DATA_DIR or TILLARK_BYOK_DIR, set it for these commands too.
• Every host that runs the source’s backups, verifications, restores or WAL archiving needs the key. The agent reports the keys it holds with each heartbeat, so a new key counts from its next heartbeat, with no restart. TillArk sends a source’s jobs only to agents holding its key, and refuses to put a source under a key no approved agent online holds.
• byok-list names any file it can’t use (another user can read it, it isn’t named after its fingerprint, or its contents don’t match) and exits 1. The agent logs the same and skips it.
• byok-import reads the key as base64 or hex. The file byok-keygen writes is base64.

On a host that must never run a job under a key TillArk holds, set TILLARK_BYOK_ONLY=1. TillArk then sends it only customer-key work, and TillArk → Agents marks it as customer-key only.

Storage on your network

A target can be a MinIO, another S3-compatible store or an SFTP server on your own network: a private address, or a name outside public DNS (a single label, or one ending in .internal, .local, .lan, .home.arpa or .corp). List it on each agent host that can reach it:

# Storage at 10.20.0.7, or anywhere in that network
TILLARK_PRIVATE_TARGETS=10.20.0.0/16
# An internal name is listed by name
TILLARK_PRIVATE_TARGETS=minio.internal,10.20.0.0/16

• Each agent reports its list, and TillArk sends a private target’s backups, verifications, restores and deletions only to agents whose list covers it. Under TillArk → Targets, each private target names the agents its jobs go to, or says that none can take them.
• List a name ending in .internal or .local by name: the agent refuses it before resolving it, so an address range doesn’t cover it. Other names are covered by their name, or by the network they resolve into.
• For a name that looks public but resolves to a private address, mark the target private (TillArk → Targets, or --private on tilldev ark targets create) and list the name or its network.
• The host that archives a source’s WAL, the one listing it in TILLARK_LOCAL_SOURCES, must reach every target the source uses.
• A bucket source on such a host follows the same rule: its backups, and real restores into it, go only to agents that list its endpoint. Mark it private under TillArk → Sources, or with --private on tilldev ark sources create.

Loopback, link-local and cloud metadata addresses can’t be targets. The agent never connects to a link-local or metadata address, even one inside a network you list.

Hosting masked clones

An agent can build masked clones of PostgreSQL backups and serve them to your team. Rows are masked on this host as they are copied into the clone; they never pass through TillArk.

# On a host with the PostgreSQL server installed
TILLARK_CLONE_LISTEN=0.0.0.0          # the address clones listen on
TILLARK_CLONE_HOST=clones.internal    # the name your team connects to
TILLARK_CLONE_PORTS=55400-55499       # the default
TILLARK_CLONE_MAX=2                   # clones served at once; the default

• The host needs what verifying PostgreSQL needs: pg_restore and the server (initdb, postgres), 12 or newer and at least as new as the sources you clone, and restic in TILLARK_ENGINES.
• Each clone’s certificate names TILLARK_CLONE_HOST, and the connection TillArk hands out checks it. Leave it unset only when TILLARK_CLONE_LISTEN is one address people reach directly.
• A clone takes one port from the range. Open the range to the machines that use clones and no further. Each clone accepts only TLS, only its own user and database, with a password.
• Clones live under <data directory>/clones and keep running across restarts of the agent. One that expires, or that someone expires early, is deleted at the agent’s next heartbeat.
• Restart the agent after setting these. TillArk → Agents then shows the host as hosting clones.

Security properties