Fence first. Promote only what has caught up.

How a failover runs

A failover starts with a request that an owner or admin makes and a second, different owner or admin approves, or, once you have armed automatic failover, when the observers agree the primary is down. From there the steps below run in order; a step that can’t complete safely ends the failover as failed rather than going on.

  request · an admin plus a second approver,
            or observers agreeing the primary is down
        │                           no replica in bounds ──▶ FAILED
        ▼                           (nothing is fenced)
 ┌──────────────┐  the agent watching the source fences the old
 │   SUSPECT    │  primary, so it takes no more writes
 └──────┬───────┘
        │ fenced                          fence fails ──▶ FAILED
        ▼                                 (nothing is promoted)
 ┌──────────────┐  picks a replica measured in the last 5 min and
 │   FENCING    │  at most 30 s behind (at the outage, in one);
 └──────┬───────┘  the agent re-measures it just before promoting,
        │          and a client switch moves clients to it first
        │ promoted                  no replica in bounds ──▶ FAILED
        ▼
 ┌──────────────┐  clients follow the switch, or you repoint your
 │   PROMOTED   │  application; point backups at the new primary
 └──────┬───────┘                            finish ──▶ HEALTHY
        │ you reseed the old primary        (the new primary stays)
        │ from the new one
        ▼
 ┌──────────────┐  the old primary replicates from
 │ REVERSE-SYNC │  the new one               finish ──▶ HEALTHY
 └──────┬───────┘
        │ you request a failback (four-eyes)
        ▼
 ┌──────────────┐  the agent proves the old primary keeps up, fences
 │   FAILBACK   │  the current one, waits for every committed write,
 └──────┬───────┘  and promotes the old primary
        │               fails before promoting ──▶ REVERSE-SYNC
        ▼               fails after promoting  ──▶ FAILED
 ┌──────────────┐
 │   HEALTHY    │  the old primary is the primary again
 └──────────────┘

What a failover refuses to do

Set up the agent

Failover runs from a TillArk agent that can reach the source’s servers. Their credentials stay on that host, in its environment; the control plane never sees them. Describe each source in a config file named by TILLARK_FAILOVER_CONFIG_PATH. One file can hold a neon block and a postgres block; each source is in one of them.

Neon

The agent needs your Neon API key, and a connection string for the replica to measure its lag:

{
  "provider": "neon",
  "neon": {
    "sources": {
      "<source-id>": {
        "project_id": "<neon-project-id>",
        "primary_endpoint_id": "ep-primary-123",
        "primary_branch_id": "br-primary-456",
        "role": "app_user",
        "replica_branch_id": "br-replica-789",
        "replica_endpoint_id": "ep-replica-012",
        "replica_dsn_env": "TILLARK_REPLICA_DSN"
      }
    }
  }
}

TILLARK_FAILOVER_CONFIG_PATH=/etc/tillark/failover.json
TILLARK_NEON_API_KEY=<neon-api-key>
# A read-only login on the replica endpoint, for the lag probe:
TILLARK_REPLICA_DSN=postgres://probe:<password>@ep-replica-012.<region>.aws.neon.tech/<db>?sslmode=require

The agent measures lag every heartbeat as the time since the replica last replayed a transaction. An idle primary therefore also shows as lag, since nothing new reaches the replica, and a replica with no connection string configured reports health but no lag, so it can’t be promoted within the bound.

Self-managed PostgreSQL

Servers you run: a primary and its replicas, kept in step by streaming replication (physical standbys) or by logical replication (a publication and a subscription). A streaming source has two to nine nodes; a logical one exactly two. The agent connects to each, works out which is the primary from the servers themselves on every step, and follows the roles through a failover. Streaming replication, three nodes:

{
  "postgres": {
    "sources": {
      "<source-id>": {
        "replication": "physical",
        "nodes": [
          { "provider": "hetzner", "region": "fsn1", "dsn_env": "ORDERS_FSN1_DSN", "replication_dsn_env": "ORDERS_FSN1_REPL" },
          { "provider": "hetzner", "region": "hel1", "dsn_env": "ORDERS_HEL1_DSN", "replication_dsn_env": "ORDERS_HEL1_REPL" },
          { "provider": "ovh", "region": "gra", "dsn_env": "ORDERS_GRA_DSN", "replication_dsn_env": "ORDERS_GRA_REPL" }
        ],
        "fence_roles": ["app"],
        "fence_command": ["/usr/local/bin/stop-db-host", "orders"],
        "reseed_command": ["/usr/local/bin/rebuild-standby", "orders"],
        "heartbeat": true,
        "max_slot_wal_bytes": 21474836480
      }
    }
  }
}

Logical replication:

{
  "postgres": {
    "sources": {
      "<source-id>": {
        "replication": "logical",
        "nodes": [
          { "provider": "aws", "region": "eu-west-1", "dsn_env": "LEDGER_AWS_DSN", "subscription_dsn_env": "LEDGER_AWS_FROM_HEL" },
          { "provider": "hetzner", "region": "hel1", "dsn_env": "LEDGER_HEL_DSN", "subscription_dsn_env": "LEDGER_HEL_FROM_AWS" }
        ],
        "publication": "ledger_pub",
        "subscription": "ledger_sub",
        "fence_roles": ["ledger_app"],
        "fence_command": ["/usr/local/bin/stop-db-host", "ledger"],
        "max_slot_wal_bytes": 21474836480
      }
    }
  }
}

TILLARK_FAILOVER_CONFIG_PATH=/etc/tillark/failover.json
ORDERS_FSN1_DSN=postgres://tillark:<password>@10.0.1.5:5432/orders?sslmode=verify-full
ORDERS_HEL1_DSN=postgres://tillark:<password>@10.0.2.9:5432/orders?sslmode=verify-full
ORDERS_GRA_DSN=postgres://tillark:<password>@10.8.0.7:5432/orders?sslmode=verify-full
# How standbys stream from each node once it is the primary. No application_name: each
# standby keeps its own. Leave the password out; each standby reads it from its ~/.pgpass.
ORDERS_FSN1_REPL=host=10.0.1.5 port=5432 user=replicator sslmode=verify-full
ORDERS_HEL1_REPL=host=10.0.2.9 port=5432 user=replicator sslmode=verify-full
ORDERS_GRA_REPL=host=10.8.0.7 port=5432 user=replicator sslmode=verify-full

The agent’s role

A superuser works. A role with only what failover needs is better; with logical replication, create it on both servers, since roles and grants aren’t replicated:

-- PostgreSQL 15 or later. Before 15, the agent's role must be a superuser.
CREATE ROLE tillark LOGIN PASSWORD '<password>' CREATEROLE;
GRANT pg_signal_backend, pg_read_all_stats TO tillark;
GRANT EXECUTE ON FUNCTION pg_reload_conf() TO tillark;
GRANT ALTER SYSTEM ON PARAMETER default_transaction_read_only TO tillark;
GRANT app TO tillark WITH ADMIN OPTION;       -- each fence role
GRANT CREATE ON DATABASE orders TO tillark;   -- for the tillark schema
ALTER ROLE tillark REPLICATION;               -- to create and drop replication slots

-- Streaming replication:
GRANT EXECUTE ON FUNCTION pg_promote(boolean, integer) TO tillark;

-- With replication_dsn_env, so standbys follow a promoted node (PostgreSQL 13 or later):
GRANT pg_read_all_settings TO tillark;
GRANT ALTER SYSTEM ON PARAMETER primary_conninfo TO tillark;

-- Logical replication, PostgreSQL 16 or later, on both servers:
GRANT UPDATE ON ALL SEQUENCES IN SCHEMA public TO tillark;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO tillark;
GRANT ledger_owner TO tillark WITH INHERIT FALSE;  -- each replicated table's owner
ALTER PUBLICATION ledger_pub OWNER TO tillark;     -- on both servers, unless FOR ALL TABLES
ALTER SUBSCRIPTION ledger_sub OWNER TO tillark;    -- on the replica

-- To reseed a logical replica, also on both servers:
GRANT pg_create_subscription TO tillark;
GRANT TRUNCATE ON ALL TABLES IN SCHEMA public TO tillark;

-- With synchronous replication (on both servers with logical replication):
GRANT ALTER SYSTEM ON PARAMETER synchronous_standby_names TO tillark;

A replica whose role lacks one of these is reported with what is missing and can’t be promoted, so a failover refuses before fencing rather than stopping halfway. Without REPLICATION, slots are left for you to drop, and the Failover page says so.

With logical replication, the subscription applies changes as each table’s owner, which is why its owner needs those roles, and its connection needs a password, since its owner isn’t a superuser. Before PostgreSQL 16, use a superuser.

The agent keeps a small tillark schema in the database: a heartbeat row per source, and a record of which node TillArk last promoted. With logical replication it creates both tables on the replica first, then adds the heartbeat to the publication.

Lag

With the heartbeat, lag is the time since the last heartbeat the replica applied. Without it (streaming replication only), it is the time since the standby replayed a commit, so an idle primary looks behind. With logical replication, a table the publication has but the subscription doesn’t, or one still on its initial copy, would be lost in a failover: the replica’s lag is reported unknown until ALTER SUBSCRIPTION … REFRESH PUBLICATION has copied it.

What fencing does

On the old primary, the fence roles can’t log in, default_transaction_read_only is turned on with ALTER SYSTEM and the configuration reloaded, and every client session is ended except the agent’s own. A primary the agent can’t reach is stopped with fence_command, or, with assume_fenced_when_down, treated as fenced. With neither, the fence fails and nothing is promoted.

Replication slots

A replica’s slot keeps WAL on the primary until the replica confirms it. The Failover page shows how much each slot holds. With max_slot_wal_bytes set, the agent drops a slot that holds more, so a replica that stopped confirming can’t fill the primary’s disk; the replica then has to be reseeded, and the drop is in the audit trail. A slot that lost WAL its replica still needs is reported down.

Request a failover

From the Failover page of a source in the dashboard, or from the CLI or API:

# Four-eyes: a different owner/admin approves, with a reference.
tilldev ark failover promote --source <source-id> \
  --approved-by <second-approver-uuid> --approval-ref INC-4821

# Follow it, with each replica's lag and whether it can be promoted:
tilldev ark failover status --source <source-id>

The failover records which replica was promoted, its lag at promotion and the time from the request to the promotion, and every step lands in the audit trail. Fence and promote steps reach the agent signed with TillArk’s key (Ed25519 and ML-DSA-65), and the agent runs only a step whose signatures verify. An agent bound to its client certificate is answered only over that certificate.

On self-managed PostgreSQL

When the fenced primary can still be reached, the replica first applies everything it committed. Then a streaming standby is promoted; a logical replica stops subscribing, and the old primary’s slot for it is dropped. Logical replication doesn’t carry sequences, so each one moves sequence_gap past the highest value used: on the old primary when it can be read, and in the columns it fills. Last, the fence roles can log in to the new primary and it takes writes. The failover lists what each step did, such as the sessions ended, the sequences moved and any slot left to drop by hand.

Three or more nodes

With streaming replication and more than one standby, the failover promotes the replica with the least lag, and keeps the rest:

• If another standby received more from the old primary, the chosen replica first streams the difference from it, so nothing that standby holds is lost.
• Before the promotion the other standbys stream from the chosen replica, so none gets ahead of it; after it they follow it onto its new timeline, keeping the name each connects as.
• A standby that was down during the failover is pointed at the new primary once it is back, if it still points at a node that isn’t the primary.

This needs replication_dsn_env on every node, and on each standby pg_read_all_settings and ALTER SYSTEM on primary_conninfo for the agent’s role (see its grants), with PostgreSQL 13 or later. A standby that can’t follow is named in the failover’s notes and keeps streaming from the old primary until you reseed it. A failback moves the other standbys the same way, and refuses before changing anything if one of them couldn’t follow.

When the old primary can’t be reached, a replica is still proven to hold every commit it acknowledged if TillArk kept synchronous replication set: each commit waited for sync_count of the named standbys, so once the agent reaches all but sync_count − 1 of them and the replica has received as much as each, it holds every one. The failover then records a lag of zero and says why.

When the primary is gone

In an outage the replica’s lag keeps growing, because it stops hearing from the primary. When the observers agree the primary is down, a failover is judged by how far behind the replica was when the primary went down, so a replica that had caught up can still be promoted minutes later. Without observers, after half a minute no replica is within the bound. To promote anyway, allow data loss:

tilldev ark failover promote --source <source-id> --allow-data-loss \
  --approved-by <second-approver-uuid> --approval-ref INC-4821

Fail over when observers agree the primary is down

Agents can observe a source’s primary: every 5 seconds each connects to it and runs one query. Only a failure of the server itself counts as down: a refused or reset connection, a timeout, or the server shutting down. An observer that can’t log in, resolve the name or verify the certificate reports unknown, which never counts toward a failover. Observers must be approved agents. List several sources in TILLARK_OBSERVE separated by commas; the connection strings stay in the agent’s environment.

# On each agent that observes the primary:
TILLARK_OBSERVE=<source-id>=ORDERS_PRIMARY_DSN
ORDERS_PRIMARY_DSN=postgres://observer:<password>@ep-primary-123.<region>.aws.neon.tech/<db>?sslmode=require

# And on the one that runs where your application does:
TILLARK_OBSERVER_VANTAGE=app

Once enough observers report, arm automatic failover from the source’s Failover page or the CLI. It then opens a failover when the quorum of observers, including one where your application runs, has not reached the primary for the whole confirm window. The failover runs the same steps as one you request: fence, then promote within the lag bound.

# Who observes, and what automatic failover would decide now:
tilldev ark failover auto --source <source-id>

# Arm it (four-eyes):
tilldev ark failover auto --source <source-id> --arm \
  --approved-by <second-approver-uuid> --approval-ref CHG-2231

# Tighten or turn it off (no approver needed):
tilldev ark failover auto --source <source-id> --confirm-window 60
tilldev ark failover auto --source <source-id> --disarm

Arming it, re-arming it, and making any setting less strict while it is armed need a second approver. Disarming and tightening don’t. It holds off when:

How far behind, in an outage

When the observers agree the primary is down, a replica is judged by how far behind it was at that moment: the time from its last replayed commit to when the last agreeing observer saw the primary go down, rounded up to the second. After that the replica has nothing more to replay, so its growing lag loses nothing. If it replayed commits more than two seconds after that moment (lag is measured to the second), the primary was still writing, and the replica is judged by its lag now. A failover you request while the observers agree the primary is down is judged the same way.

Every automatic failover, stand-down and settings change is in the audit trail and the notification bell. Observers can’t see a client that reaches the primary when none of them can, which is why one should run where your application does; fencing still stops the old primary before anything is promoted.

When TillArk is unreachable

TillArk decides automatic failover, so an outage that also cuts the agents off from TillArk would leave the primary down. Turn on autonomy and the agent that watches the replica, the executor, can fail the source over on its own. It acts only after it has not heard from TillArk for 60 seconds, and only on the same rules: the quorum of observers, including one where your application runs unless you dropped that rule, has not reached the primary for the whole confirm window, and the cooldown and cap allow it. It fences the primary, then promotes the replica if it was at most 30 s behind when the primary went down, judged as above.

It acts under a grant TillArk signs, with Ed25519 and ML-DSA-65, for the executor and the observers it names: which node is the primary, which replica may be promoted, and the settings. A grant lasts 24 hours and is re-signed every hour, and whenever what it names changes. TillArk revokes it as soon as it could no longer be met: autonomy or automatic failover is turned off, a failover or failback is under way, the source has failed over, a failover failed, or the cap is reached. Observers send their signed reports straight to the executor, which counts only reports from observers in its grant.

The executor needs a port observers can reach. It serves TLS 1.3 there with the certificate it signs in to TillArk with, and observers check that certificate before they send anything.

# On each agent that watches a replica: where observers reach it.
TILLARK_PEER_LISTEN=0.0.0.0:7443
TILLARK_PEER_ADDR=replica-host.example.internal:7443

Turning autonomy on needs automatic failover armed and a second approver; so does arming with it on. The source’s Failover page and the CLI say whether it is in force and, when it is not, what is missing: an observer that does not hold the grant or cannot reach the executor, an agent that refused the grant and why, or an executor that has stopped sending heartbeats.

# Turn it on, once automatic failover is armed (four-eyes):
tilldev ark failover auto --source <source-id> --autonomy \
  --approved-by <second-approver-uuid> --approval-ref CHG-2240

# Whether it is in force, and what each agent holds, hears and decides:
tilldev ark failover autonomy --source <source-id>

# Turn it off; the grant is revoked at once (no approver needed):
tilldev ark failover auto --source <source-id> --no-autonomy

When the executor reaches TillArk again it reports what it did, with the observers’ signed reports. TillArk records the failover and rechecks that evidence the way the executor had to judge it; a failover the evidence does not support is marked so in the failover, the audit trail and the notification bell. It takes the place of any failover opened for the source meanwhile that did nothing, and the replica it promoted becomes the primary. A report that arrives after a later failover has acted is kept as history and changes nothing; check which node takes writes.

While an agent may hold a grant for a source, TillArk sends failover steps for that source only to it, even after revoking the grant, until the agent confirms it dropped the grant or the grant expires. So two agents never promote at once.

tilldev ark failover autonomy release --source <source-id> \
  --approved-by <second-approver-uuid> --approval-ref INC-4830

An executor cut off from both TillArk and your application can’t tell a network split from an outage by itself. The observers decide that, which is why autonomy keeps the same quorum and application observer, and why the old primary is fenced before anything is promoted.

Moving clients

Without a client switch, TillArk promotes the replica and pointing your application at it is yours to do. With one, the agent moves your clients itself, after the old primary is fenced and before the promoted node takes writes: it points Cloudflare DNS records at the node, calls a webhook of yours, or both. A failback moves them the same way. Add switch to a source in the agent’s failover config:

{
  "postgres": {
    "sources": {
      "<source-id>": {
        "replication": "physical",
        "nodes": [
          { "provider": "hetzner", "region": "fsn1", "dsn_env": "ORDERS_FSN1_DSN" },
          { "provider": "ovh", "region": "gra", "dsn_env": "ORDERS_GRA_DSN" }
        ],
        "fence_roles": ["app"],
        "switch": {
          "dns": { "provider": "cloudflare", "zone_id": "<cloudflare-zone-id>", "names": ["db.example.com"], "ttl": 60 },
          "webhook": { "url": "https://ops.example.com/tillark/switch" },
          "nodes": {
            "hetzner:fsn1": { "A": "203.0.113.10" },
            "ovh:gra": { "A": "198.51.100.20" }
          }
        }
      }
    }
  }
}

# A Cloudflare API token with Zone · DNS · Edit on this zone only:
TILLARK_CLOUDFLARE_API_TOKEN=<cloudflare-api-token>
# At least 32 characters; your webhook checks each request with it:
TILLARK_SWITCH_WEBHOOK_SECRET=<random-secret>

Create the API token in Cloudflare under My Profile → API Tokens with one permission, Zone · DNS · Edit, limited to the zone that holds the names. Neon routes connections by the endpoint in the host name, so a DNS record can’t move clients between Neon endpoints: on Neon, use a webhook.

DNS records

Before fencing anything, the agent reads the records; if it can’t, the failover stops with the old primary still taking writes. After fencing, it sets each record to the promoted node’s contents, reads them back, and only then promotes the node. If a record can’t be set or read back, or the promotion fails, it sets back every record it changed, and the failover lists what it did.

The agent reads the records once a minute and reports where they point. The Failover page, tilldev ark failover status and the notification bell say when they point away from the primary outside a failover, for example after someone changed them by hand, and when the agent can’t read them. A problem that won’t pass on its own, such as a revoked token, a missing record or two records of one type for a name, makes every replica unpromotable until it is fixed, so a failover never fences a primary it couldn’t move clients away from.

When the records drift, an owner or admin can have the agent point them at the node it finds taking writes. The agent never takes the target from the request, and refuses when both nodes, or neither, take writes. The request lapses if the agent doesn’t pick it up within 15 minutes, and its outcome is in the audit trail.

# Where the records point, and any drift:
tilldev ark failover status --source <source-id>

# Point them at the node taking writes (admin):
tilldev ark failover repoint --source <source-id>

The webhook

The agent posts JSON to your URL. event is one of:

POST /tillark/switch
Content-Type: application/json
User-Agent: tillark-agent
X-TillArk-Event: switch
X-TillArk-Delivery: 5f0c9d2e41a87b6c3d1e0f9a8b7c6d5e
X-TillArk-Signature: t=1790680504,v1=9c1d…

{
  "event": "switch",
  "kind": "failover",
  "source_id": "<source-id>",
  "failover_id": "<failover-id>",
  "from_node": "hetzner:fsn1",
  "to_node": "ovh:gra",
  "records": { "A": "198.51.100.20" },
  "sent_at": "2026-09-29T10:15:04Z"
}

kind is failover, failback or repoint; a repoint has no failover_id. records is present when nodes lists the node. Each attempt waits up to 10 seconds, and a retry after a timeout can repeat an event you already handled, so make each one idempotent. X-TillArk-Delivery is a random id per attempt, for your logs.

Check X-TillArk-Signature before acting on a request: it is t= the Unix time it was sent and v1= the hex HMAC-SHA256 of the time, a dot and the raw body, keyed with the secret. Refuse a request more than 5 minutes old.

import { createHmac, timingSafeEqual } from 'node:crypto'

// raw is the request body exactly as received, before any JSON parsing.
export function verifyTillArkSwitch(raw: Buffer, header: string, secret: string): boolean {
  const parts = new Map(header.split(',').map((p) => p.split('=', 2) as [string, string]))
  const t = Number(parts.get('t'))
  if (!Number.isInteger(t) || Math.abs(Date.now() / 1000 - t) > 300) return false
  const want = createHmac('sha256', secret).update(`${t}.`).update(raw).digest()
  const got = Buffer.from(parts.get('v1') ?? '', 'hex')
  return got.length === want.length && timingSafeEqual(got, want)
}

After the promotion

With a client switch, your clients already point at the promoted database. Without one, repoint your application at it. Either way, point the source’s backups at it too:

# Point backups at the new primary (prompts for the connection string):
tilldev ark sources update <source-id> --with-dsn

On PostgreSQL you run, if the old primary comes back taking writes, for example one treated as fenced while it was unreachable, the agent fences it the next time it observes the source, going by TillArk’s record of which node it promoted last, and the audit trail shows it. Of several nodes taking writes, every one but the node promoted last is fenced; if the records can’t tell them apart, none is, and the Failover page asks you to fence all but one. Then either fail back to it or finish the failover. Until you do, a new failover can’t be requested and automatic failover holds.

Rebuild a node from the primary

A reseed replaces everything on a node with a copy of the current primary: a replica whose slot was dropped or that fell too far behind, or the old primary after a failover. It is for PostgreSQL you run. Like a promotion it needs an owner or admin and a second approver, and the agent that watches the source runs it.

# Which nodes can be reseeded (RESEED column):
tilldev ark failover status --source <source-id>

# Four-eyes, like a promotion:
tilldev ark failover reseed --source <source-id> --node hetzner:hel1 \
  --approved-by <second-approver-uuid> --approval-ref CHG-1204

# The old primary after a failover whose replica wasn't proven caught up:
tilldev ark failover reseed --source <source-id> --node aws:eu-west-1 --discard-writes \
  --approved-by <second-approver-uuid> --approval-ref CHG-1204

The Failover page and tilldev ark failover status show each replica as ready to reseed, or unconfigured until the agent’s failover config says how, and list recent reseeds with what the agent did.

Logical replication

Set subscription_dsn_env on each node. The agent removes the node’s subscription, drops its replication slot on the primary, empties the node’s copies of the published tables, and subscribes it again, which copies every row. It finishes when the copy has.

# The connection each node subscribes with when it is the replica, to the other node:
LEDGER_AWS_FROM_HEL=host=10.9.0.4 dbname=ledger user=tillark password=<password> sslmode=verify-full
LEDGER_HEL_FROM_AWS=host=10.2.0.6 dbname=ledger user=tillark password=<password> sslmode=verify-full

Nothing changes, and the reseed says why, when:

• a published table doesn’t exist on the node, since logical replication doesn’t copy the schema;
• a table that isn’t published has a foreign key to one that is, since emptying it would break;
• another connection is using a replication slot with the subscription’s name;
• the node is neither the replica nor a fenced old primary.

The new subscription has the default subscription options; set any others again afterwards. Its slot holds WAL on the primary while the copy runs, and max_slot_wal_bytes still applies, so set it above what a full copy needs.

Streaming replication

Set reseed_command to a script that rebuilds the node as a standby of the primary, with pg_basebackup or pg_rewind, and starts it. The agent runs it on its own host without a shell, for up to 24 hours, with TILLARK_SOURCE_ID, TILLARK_RESEED_NODE and TILLARK_PRIMARY_NODE set to the source, the node being rebuilt and the primary, as provider:region. Of the agent’s own environment it inherits only PATH, HOME, TMPDIR, LANG and LC_ALL, so read passwords from a file such as ~/.pgpass. After it exits 0, the node has 2 minutes to be streaming from the primary as a copy of it, or the reseed fails.

What a reseed refuses to do

• Rebuild the primary, or a node while it’s unclear which node is the primary.
• Start while a failover is in progress; a failover that starts before a requested reseed runs cancels it.
• Run two reseeds of a source at once.
• Delete writes without saying so. An old primary replaced by a replica that wasn’t proven caught up may hold writes the new primary never received, and the reseed deletes them: copy them out first, then pass --discard-writes.

A node being reseeded is never fenced for or promoted: a failover picks another replica, or refuses. If the agent stops while a reseed runs, the reseed fails and the node may be half-copied; reseed it again.

Failback: catch up first, then switch

Never resume replication in the old direction: it would overwrite the writes taken during the outage. Failing back is for PostgreSQL you run, and goes the other way:

1. Reseed the old primary from the new one. Once its agent reports it as a replica, the failover moves to reverse_sync.
2. Request the failback. Like a promotion, it needs an owner or admin and a second approver.
3. Once the state is healthy, your clients follow the client switch back to the old primary, or you repoint your application at it.

# Once the old primary replicates from the new one (state reverse_sync):
tilldev ark failover status --source <source-id>

# Four-eyes, like a promotion:
tilldev ark failover failback --source <source-id> \
  --approved-by <second-approver-uuid> --approval-ref CHG-1190

The agent that watches both nodes checks that the old primary applies a fresh write from the current one within 1 second, fences the current primary the way a failover does (without fence_command), waits up to 30 seconds until the old primary has every write it committed, and promotes it. With logical replication it sets the sequences past every value used first, and the demoted node then subscribes to the promoted one without a copy, since both hold the same rows. With streaming replication the demoted node stays fenced and is reseeded from the promoted one with your reseed_command, under the failback’s approval.

If anything fails before the promotion, the current primary takes writes again and the failover returns to reverse_sync with the reason. If it fails after, the failover ends as failed: check which node is the primary before doing anything else. What each step did is on the Failover page and in tilldev ark failover status.

Failback needs, with logical replication, the publication on both servers and subscription_dsn_env on both nodes, so each can follow the other; with streaming replication, reseed_command and heartbeat, so an idle primary doesn’t look behind.

Finish instead

Finishing keeps the new primary and ends the failover, so a new one can be requested and automatic failover resumes. Nothing changes on your database: the old primary stays as it is until you reseed it, and that reseed still needs --discard-writes if it may hold writes the new primary never received. It needs an owner or admin, without a second approver. A failover of a Neon database ends this way.

# Keep the new primary and end the failover:
tilldev ark failover finish --source <source-id>

Synchronous replication

With synchronous replication, PostgreSQL acknowledges a commit on the primary only once a standby has it, so promoting that standby loses no acknowledged commit. With a quorum, a commit waits for any k of several standbys, so it survives the loss of all but one of them. On PostgreSQL you run, the agent reports it for every source, and can keep it set for you.

What the agent reports

On the primary: its synchronous_standby_names, the weakest synchronous_commit the fence roles commit with (set for the role, the database or the server), and how many commits are waiting for a standby and for how long. Once a commit has waited 5 seconds the primary is reported down: its writes are stalled.

On the replica: whether it is synchronous, and if not, why. It may not be connected, still be catching up, wait behind another standby, be one of a quorum a commit may not include, not be named in synchronous_standby_names, or a fence role may commit with synchronous_commit set to local or off, so its commits don’t wait. A synchronous replica’s mode is sync; one in a quorum, where every acknowledged commit is on some of its members but not necessarily this one, is quorum.

The agent tells the replica’s connection apart from others by its replication slot. A streaming standby without a slot is found by the name it connects as: the application_name in its primary_conninfo, otherwise its cluster_name. Give each standby its own name, or it is reported as one that can’t be told apart.

The Failover page shows it in its own panel, and the notification bell reports stalled writes and a replica that should be synchronous and isn’t.

tilldev ark failover status --source <source-id>
# MODE reads sync for a synchronous replica, quorum for a quorum member; below the table:
# Synchronous replication: commits on hetzner:fsn1 wait for synchronous_standby_names = ANY 1 ("hel1", "gra") (synchronous_commit on, managed by TillArk: preferred)

Failover and failback

A standby copied from a primary with synchronous_standby_names set has the same setting, and once promoted its commits would wait for a standby that doesn’t exist yet. So when the agent promotes a node, in a failover or a failback, it clears synchronous_standby_names there with ALTER SYSTEM unless a standby already streams from it that satisfies it. The step’s notes and the audit trail record it. Without synchronous set, set it again yourself once the other nodes replicate from the new primary. The cleared value is in postgresql.auto.conf, which overrides postgresql.conf; ALTER SYSTEM RESET synchronous_standby_names returns to the value in postgresql.conf.

This needs ALTER SYSTEM on synchronous_standby_names for the agent’s role (see its grants). Without it, a replica with the setting can’t be promoted and a failover refuses before fencing. The agent’s own sessions commit with synchronous_commit = local, so fencing a primary whose writes are stalled doesn’t wait for a standby.

Let TillArk keep it set

Set synchronous in the source’s failover config. Every 2 seconds the agent checks the primary; once enough replicas stream, it sets synchronous_standby_names to name every replica that isn’t async or being reseeded, and reloads the configuration. One replica gives FIRST 1 ("<replica>"); more give ANY <sync_count> ("<a>", "<b>", …), a quorum. Each change is in the audit trail.

{
  "postgres": {
    "sources": {
      "<source-id>": {
        "replication": "physical",
        "nodes": [
          { "provider": "hetzner", "region": "fsn1", … },
          { "provider": "hetzner", "region": "hel1", … },
          { "provider": "ovh", "region": "gra", … },
          { "provider": "aws", "region": "sa-east-1", …, "async": true }
        ],
        "fence_roles": ["app"],
        "synchronous": "preferred",
        "sync_count": 1
      }
    }
  }
}

The agent then owns synchronous_standby_names on the primary: don’t set it elsewhere at the same time.

Three durability tiers

TillArk judges what protects a source’s committed writes in three tiers, each on its own, from what its agents report. A tier is held, lapsed when the source has it but it doesn’t hold now, or none when the source doesn’t have it.

The Failover page shows the three side by side, with the replica or archive target that holds each or comes closest, and tilldev ark failover status lists them under the replicas. A lapsed tier opens an alarm that reaches your alert rules and the notification bell. While a failover moves roles, or when no replica can be promoted at all, the tier 2 alarm waits; the tier 3 alarm waits while the source is paused.