A secret nothing should use

Three kinds of decoy

Set a canary

Under Secrets → Canaries, choose New canary, or use the CLI. The value is made on the server and never shown. Only its public parts come back: the access key id and IAM user, or the host and user name.

# An AWS key pair: a real IAM user that can do nothing. Also stores AWS_ACCESS_KEY_ID.
tilldev secrets gen AWS_SECRET_ACCESS_KEY --env prod --type canary --shape aws

# A Postgres connection string whose host refuses every sign-in and reports it.
tilldev secrets gen DATABASE_URL --env prod --type canary --shape postgres --pg-database billing

# A live-mode Stripe key. Trips inside TillDev only.
tilldev secrets gen STRIPE_SECRET_KEY --env prod --type canary --shape stripe

Put canaries where an intruder would look: in a production environment, next to real keys, under names that fit. Export them with the rest of the environment (tilldev secrets pull or a sync target) to plant them in a .env file, a CI variable or a config repository. Making a canary under a key that already holds a real secret is refused. Making one where a canary already exists replaces it.

What trips it inside TillDev

The first touch by someone raises an anomaly. Further touches by the same agent or person while it is open count as sightings of the same anomaly, not new alerts.

Who hears about it

A hit appears under Secrets → Anomalies as a high Canary anomaly, and every owner and admin gets an email. To page someone, add a TillPulse alert rule on secrets_anomaly and limit it to "kinds": ["canary"]. A hit from outside TillDev has the agent outside.

# Which secrets are canaries. Waits for your approval every time.
tilldev secrets canaries              # --all or --retired for retired ones

# Stop watching one: deletes the decoy and removes its AWS user.
tilldev secrets canaries rm <id>

# Page someone on any canary hit.
tilldev secrets anomalies route --project checkout --kinds canary --min-severity high --pagerduty <integration-key>

{
  "source": "tillsecrets",
  "type": "secrets_anomaly",
  "rule": "TillSecrets canaries",
  "agent": "outside",
  "severity": "high",
  "message": "Canary checkout/prod/AWS_SECRET_ACCESS_KEY was used against AWS (sts, eu-west-1)",
  "anomalies": ["Canary checkout/prod/AWS_SECRET_ACCESS_KEY was used against AWS (sts, eu-west-1)"],
  "kinds": ["Canary"],
  "anomaly_ids": ["…"],
  "link": "https://tilldev.dev/acme/secrets/anomalies"
}

AWS canaries

AWS canaries are made in an AWS account you choose, by an IAM identity whose key you store in TillSecrets. TillDev reads that key on the server only to make, check and remove decoy users. Each of those reads is audited; the scheduled checks once a day.

# 1. In an AWS account that holds nothing else, create an IAM user with this policy
#    and one access key. Store both halves in TillSecrets.
# 2. Point canary settings at them. Saving checks the key with one IAM call.
tilldev secrets canaries settings \
  --aws-key-id-ref secretref://ops/prod/CANARY_AWS_ACCESS_KEY_ID \
  --aws-secret-key-ref secretref://ops/prod/CANARY_AWS_SECRET_ACCESS_KEY

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "iam:CreateUser", "iam:DeleteUser",
      "iam:PutUserPolicy", "iam:DeleteUserPolicy",
      "iam:CreateAccessKey", "iam:DeleteAccessKey",
      "iam:GetAccessKeyLastUsed"
    ],
    "Resource": "arn:aws:iam::*:user/*"
  }]
}

Every ten minutes TillDev asks AWS when each decoy key was last used. A key with a deny-all policy still records its use, so a call that fails still trips. Retiring or regenerating a canary removes its user. If that fails, the canary is still retired and TillDev tells you which user to remove by hand.

Postgres canaries

The host in a Postgres canary resolves to a TillDev listener that speaks the Postgres protocol. It accepts TLS, reads the user and database, asks for the password, and answers password authentication failed, the same error a real server sends. The attempt is reported with the client's address. The password is compared with the canary's by hash, so the alert says whether it matched without storing what was typed.

The default host is a random name under a domain TillDev runs. To make it read like yours, point a host name of yours (CNAME) at the name shown in canary settings, then set it there or pass --pg-host. If Secrets → Canaries says Postgres canaries aren't available, no listener is configured where your workspace is served.

Keeping them hidden

Nothing but the canary list says a secret is a canary. Secret listings, the API, agent tokens and sync targets treat it like any other secret. Its value is audited as an ordinary write, and the separate canary.create entry doesn't name the secret. Seeing the list takes an owner or admin's approval (canary.view) every time, and each look is audited. Retiring a canary (canary.retire) and changing canary settings (canary.configure) wait for approval too, so an agent using your login can't switch a tripwire off. Deleting a canary's secret retires the canary.

What it can and can’t tell you

• A Stripe canary trips only inside TillDev. Stripe tells no one else that a key was used.
• AWS reports a key's last use minutes to hours late, with the service and region but not the caller's address. For that, turn on CloudTrail in the canary account.
• AWS canaries are checked by the scheduler. If a check fails, for example because the IAM key was revoked, the canary list shows the error and the canary trips only inside TillDev until it is fixed.
• Reading a whole environment doesn't trip anything. Someone who pulls it is caught when they use the canary, not when they copy it.
• Postgres clients that insist on SCRAM authentication hang up before sending a password. The attempt is still reported, without one. The address is whoever connected, which may be a VPN or proxy.
• An admin who reads the audit log closely can match a canary's creation time to its secret's first write. An admin who approves a look at the list sees them all.

API

POST   /api/secrets/environments/{envId}/secrets/generate
       {"key": "AWS_SECRET_ACCESS_KEY", "type": "canary", "shape": "aws" | "postgres" | "stripe",
        "aws_user": "svc-billing-export", "pg_database": "billing", "pg_host": "db-replica.example.com"}

GET    /api/secrets/canaries?status=active|retired|all        needs approval: canary.view
DELETE /api/secrets/canaries/{id}                              needs approval: canary.retire
PUT    /api/secrets/canaries/settings                          needs approval: canary.configure
       {"aws_access_key_id_ref": "secretref://…", "aws_secret_access_key_ref": "secretref://…",
        "pg_host": "db-replica.example.com", "verify": true}

Canary routes are for signed-in owners and admins. A write or rollback that targets a canary answers 409. See the API reference.