An agent spending unlike itself
BetaEvery time an agent spends a secret through the broker, TillSecrets records the host, the secret reference and the time, never the value. It compares each agent with its own last 14 days, so a deploy bot that has only ever called one API stands out the first time it calls another.
What counts
| Anomaly | When | Severity |
|---|---|---|
| New host | A spend reaches a host the agent has not used in 14 days. | High |
| New secret | The agent uses a secret it has not used in 14 days. | Medium |
| Volume spike | More spends in the last hour than three times its busiest hour (never fewer than 20). | Medium; high past twice that |
| Unusual hour | A spend in a UTC hour the agent has never been active in. | Low |
| Refusals | Five or more refused spends in 15 minutes, for example a missing grant or a blocked host. | High |
The same list shows canary hits, leaks (vault values found in a push, a TillPulse event or a check) and service tokens used without their host key.
A new agent has no history, so for its first 20 spends over 3 days only volume (more than 100 an hour) and refusals are judged. Hours are judged once it has been active on 7 days. The baseline ends an hour ago, so a burst is never compared with itself.
Who hears about it
Anomalies are checked within about a minute of a spend. Each one appears under Secrets → Anomalies and in the notification bell for owners and admins. Medium and high anomalies are emailed to the admins behind the spends, or to the owners when none of them is an admin any more. The same thing seen again on the same UTC day counts as another sighting, not another email.
To page someone, add a TillPulse alert rule with the trigger TillSecrets: an agent spending unlike itself (secrets_anomaly). It sends to Slack, Microsoft Teams, PagerDuty, email or a webhook. By default it sends medium and high. You can raise or lower that, limit it to some agents (a trailing * matches a prefix) and set a quiet period. Rules from every project in the workspace receive the workspace's anomalies. A webhook is signed with the rule's own secret, shown once when you add it; verify it before you act on a delivery.
# Send high anomalies from deploy agents to Slack, at most every 30 minutes.
tilldev secrets anomalies route --project checkout \
--slack https://hooks.slack.com/services/… \
--min-severity high --agents 'agent:deploy-*' --quiet-minutes 30
# Or PagerDuty, Teams, email or any webhook.
tilldev secrets anomalies route --project checkout --pagerduty <integration-key>{
"delivery_id": "…",
"rule_id": "…",
"source": "tillsecrets",
"type": "secrets_anomaly",
"rule": "Secrets anomalies",
"agent": "agent:deploy-bot",
"severity": "high",
"message": "agent:deploy-bot spent a secret at paste.example, a host it has not used in 14 days",
"anomalies": ["agent:deploy-bot spent a secret at paste.example, a host it has not used in 14 days"],
"kinds": ["New host"],
"anomaly_ids": ["…"],
"link": "https://tilldev.dev/acme/secrets/anomalies"
}Resolve it
Look at the agent's baseline, decide, then resolve the anomaly as expected, revoked or investigated, with an optional note. Resolving waits for your approval (anomaly.resolve), so an agent using your login can't silence its own alarm. Agent tokens can't list or resolve anomalies.
# Open anomalies. --all adds resolved ones; --agent narrows to one agent.
tilldev secrets anomalies
# What the detector compares an agent with.
tilldev secrets anomalies baseline agent:deploy-bot
# Close one. Waits for your approval.
tilldev secrets anomalies resolve <id> --revoked --note "token rotated"
# or --expected, or --investigatedWhat it can and can’t tell you
- It is a heuristic. An anomaly is a reason to look, not proof of misuse, and a careful attacker who stays inside an agent's habits won't trip it.
- The baseline moves. An agent that drifts slowly to new behaviour teaches the baseline as it goes.
- Only agent tokens (
agent:<name>) give an agent an identity it can't choose. Other agent keys are labels the caller sends, so two programs using one label share a baseline. - Hours are UTC. An agent that follows local business hours across a daylight-saving change may show one unusual hour.
- Detection runs on the scheduler, so it lags a spend by up to about a minute. It warns; it doesn't block. To block, use grants, destinations and approvals.
API
GET /api/secrets/anomalies?status=open|resolved|all[&agent=agent:deploy-bot]
GET /api/secrets/anomalies/baseline?agent=agent:deploy-bot
POST /api/secrets/anomalies/{id}/resolve {"resolution": "expected" | "revoked" | "investigated", "note": "…"}
POST /api/alerts {"project_id": "…", "name": "Secrets anomalies", "trigger_type": "secrets_anomaly",
"conditions": {"min_severity": "high", "agents": ["agent:deploy-*"]},
"actions": [{"type": "slack", "webhook_url": "https://hooks.slack.com/services/…"}]}Anomaly routes are for signed-in owners and admins. Resolving an anomaly that is already resolved answers 409. See the API reference.
anomaly.detect when one is raised and anomaly.resolve when a person closes it, with the resolution.