Roles move. Machines don't lie about it.

The failover / failback lifecycle

The reducer is a pure, tested core — illegal transitions are no-ops. It encodes the guardrails below directly, so “never promote without fencing” is a property of the machine, not a hope about the operator.

   ┌──────────┐  primary unhealthy for N sec across M observers
   │  HEALTHY  │  (≥1 observer positioned where the APP is)
   └────┬─────┘─────────────────────────────────────────────┐
        │ manual / auto (opt-in)                             ▼
        │                                             ┌──────────────┐
        │                                             │   SUSPECT     │
        │                                             │ confirm window│
        │                                             └──────┬───────┘
        ▼                                                    │ quorum agrees
 ┌──────────────┐  fence old primary                        │
 │   FENCING     │◀──────────────────────────────────────────┘
 └──────┬───────┘  STONITH / Neon suspend + role-password rotate
        │           pre-flight: REFUSE to promote a too-stale replica
        ▼
 ┌──────────────┐  promote replica → it holds the PRIMARY role;
 │  PROMOTED     │  repoint app (DNS flip default | proxy opt-in);
 └──────┬───────┘  backups now sourced from the new primary
        │ old node returns
        ▼
 ┌──────────────┐  old node re-attaches as REPLICA of the new primary
 │ REVERSE-SYNC  │  ⚠ NEVER resume the old direction (would overwrite
 └──────┬───────┘     new writes) · physical: pg_rewind · logical: re-seed
        │ lag → 0
        ▼
 ┌──────────────┐  DELIBERATE clean switchover: quiesce writes,
 │  FAILBACK     │  drain the last bytes, flip the role back.
 └──────────────┘  Catch-up FIRST, flip SECOND → zero data loss.
                   Failback is manual / scheduled — never auto (anti-flap).

What the machine refuses to do

Fencing — even a DB you don't control

Fencing (STONITH) makes the old primary stop accepting writes before a replica is promoted. On self-managed Postgres that’s Patroni + pg_rewind. On a managed database you don’t own — Neon, RDS, Supabase, PlanetScale — TillArk uses a provider-native substitute: rotate the role password and suspend the compute endpoint via the provider API, so the old primary rejects writes even if it is still alive.

Fencing itself is an authenticated, audited action; the commands the agent executes are hybrid-signed and mTLS-authenticated between agent and control plane. The fence_status (none → pending → fenced) is tracked on the failover row.

Promote a replica

Full-auto failover is explicit opt-in, never the default. A manual promote is a high-privilege, RBAC-gated action that additionally requires four-eyes approval — the same discipline as a real restore.

# Promote a replica. High-privilege + four-eyes: a DIFFERENT owner/admin
# approver and a reference. The server re-verifies the approver; a token
# can never approve its own promote.
tilldev ark failover promote --source <source-id> \
  --to <replica-node> \
  --approved-by <second-approver-uuid> --approval-ref INC-4821

# Watch the state + replica lag:
tilldev ark failover status --source <source-id>
#  state    fencing → promoted
#  ROLE     REGION   LAG(s)  PROMOTABLE  HEALTH
#  primary  eu-west  0       yes         healthy

On promotion the replica takes the primary role, the app is repointed (DNS flip by default, or an ark-managed proxy for enterprise, which removes the DNS-TTL wait), and backups are sourced from the new primary. Logical-replication caveats are handled — sequences are bumped past the primary on promote (or you use UUIDv7 / ULID and avoid a shared counter), and DDL is fanned to both sides through a TillArk-aware migration path.

Failback: catch up first, flip second

When the old node returns it re-attaches as a replica of the new primary — the old direction is never resumed, because that would overwrite the writes taken during the outage. Only once the returning node’s lag reaches zero can you run a deliberate, scheduled failback: quiesce writes, drain the last bytes, then flip the role back. Failback is manual / scheduled — never automatic, to avoid flapping.

# Failback is a deliberate, scheduled switchover — refused while the
# returning node is beyond the failback staleness bound.
tilldev ark failover failback --source <source-id> \
  --lag 0 \
  --approved-by <second-approver-uuid> --approval-ref CHG-1190

The agent can act if the control plane is down

The control plane going down must never take your ability to fail over with it. The agent caches its policy and last-known topology and can execute a pre-authorized failover while the control plane is unreachable — deferring entirely to the control plane whenever it is reachable. TillArk is never a single point of failure for your recovery.