Customer keys

Where to find it

Under Customer keys in workspace settings, or with tilldev customer-keys. Everyone in the workspace can see which keys exist and what they protect. Only owners can add, change or remove a key, or put a product under one, and API keys can’t do any of it.

What a key protects

Put a key on one product at a time. While the key refuses, everything listed for that product stops working:

Each product sits under at most one key, and a workspace holds up to 3 keys.

How it works

• TillDev never holds your key or a credential for your cloud. For each call it signs a short-lived identity token, your cloud checks it against a trust you set up, and hands back access that lasts minutes.
• What your key seals names the workspace and the product it belongs to, so a value can’t be opened for another workspace or moved between products.
• Before a product is switched over, the value is sealed with your key and opened again, and nothing changes unless it comes back the same.
• Rotating a product’s own key keeps it under your key: the new key is sealed with yours before it is stored.

Setting up your cloud

Add the key in TillDev first. Its Setup view, or tilldev customer-keys show, gives the exact trust and permissions to paste, with the issuer and the subject for that key filled in. In short:

Then press Test. A key only protects anything once it has sealed and opened a test value.

Putting a product under your key

Press Put under next to the product, or run tilldev customer-keys protect <key> <product>. You see the list of what goes offline if the key ever refuses, confirm it, then approve the change, so a signed-in session on its own can’t make it. Taking a product off works the same way and needs the key to still work, since its value has to be opened one last time.

A key can only be removed once it protects nothing. Removing it in TillDev doesn’t touch the key in your cloud.

Your users’ sign-in

TillAuth is the one product where a refusing key reaches your own users. Every app’s signing key, authenticator secrets, social sign-in secrets and webhook secrets sit under the app’s data key, so while your key refuses, your users can’t sign in, their sessions stop refreshing, and webhooks wait.

While the key refuses, sign-in and token calls answer 423. Webhook deliveries are held and retried rather than failed, for up to three days. TillAuth never falls back to a shared signing key.

Sealed namespaces

With TillCache under a key, every namespace in the workspace is sealed. The cache edge encrypts values, hash values, set and sorted-set members, queue messages and pub/sub messages before they reach storage, with a key derived for that namespace alone, and opens them again on the way out. Your code doesn’t change.

• Key names, hash field names and sorted-set scores stay readable, so TTLs, ranges by score and counts keep working
• APPEND, STRLEN, GETRANGE, SETRANGE, INCR, DECR, INCRBY, DECRBY, INCRBYFLOAT, HINCRBY, HINCRBYFLOAT, and ZRANGE or ZREVRANGE with BYLEX, are refused, since they need the value in the clear
• New writes are sealed from about a minute after you turn it on, once every edge has the change; data already stored, pub/sub history included, is then sealed in the background, and reads return both sealed and unsealed values correctly until that finishes

Set and sorted-set members are sealed the same way each time, so membership checks and removals still work. Sorted sets with equal scores come back in a different order than they would unsealed. While your key refuses, every call to a sealed namespace answers 423. Taking TillCache off the key unseals stored data in the background the same way.

Sealed event detail

With TillPulse under a key, the full detail of every event is sealed: stack frames and local variables, breadcrumbs, device, app and user context. Events are sealed at the edge the moment they arrive, so even the queue waiting to be processed holds them sealed, and uploaded source maps, dSYMs and ProGuard mappings are sealed in storage.

• Issue titles, culprits, counts and the analytics columns behind charts, search and questions stay readable, so they keep working while the key refuses
• Events that arrive while the key refuses are held, sealed, for up to 7 days and processed once it answers again
• New events are sealed as soon as you turn it on; events and uploads already stored are sealed in the background, and both read correctly until that finishes
• Security history recorded before you turned it on keeps its evidence readable until that history ages out
• Hunts, detection rules and playbooks see app security events without their sealed evidence, so they can’t match on what is inside it

Issue pages, crash lines in TillForge, release correlation and TillMind open the detail they need as they run. While your key refuses, an issue still shows its title, counts and charts, with a notice in place of the stack and breadcrumbs, and calls that need the detail answer 423. Taking TillPulse off the key opens stored data again in the background.

When you revoke

A product’s key, once opened, stays in memory for the window you choose: 5, 10, 15 minutes. Shorter means a revocation takes hold sooner; longer means fewer calls to your cloud. After you disable the key or take away access, TillDev stops serving that product’s data within the window, plus about a minute.

From then on, calls that need the data answer 423 with code: KEY_LOCKED and the kind of refusal, and the workspace shows a banner. TillDev keeps asking, so the data comes back on its own once the key works again. Nothing is deleted. If your cloud doesn’t answer at all, calls get 503 with Retry-After, since that is likely a passing outage rather than a decision.

Health checks

TillDev checks each key every 6 hours, and straight after any change: it seals and opens a test value, reads the key’s state, and opens what each product has stored with it. Owners get an email when a key starts refusing, and when your cloud says the key is set to be deleted or expire within 30 days, so you hear about it before anything goes offline.

The record

The workspace audit log records customer_key.added, customer_key.updated, customer_key.tested, customer_key.attached, customer_key.detached and customer_key.removed, with who did it. Your cloud’s own logs show every call TillDev makes with the key.

From the terminal

tilldev customer-keys                                # your keys, their health and what each protects
tilldev customer-keys add --name "Prod key" --provider aws \
  --key-arn arn:aws:kms:eu-west-1:111122223333:key/… \
  --role-arn arn:aws:iam::111122223333:role/TillDevCustomerKey
tilldev customer-keys show "Prod key"                # the trust and policies to set up in your cloud
tilldev customer-keys test "Prod key"
tilldev customer-keys protect "Prod key" tillsecrets  # lists what goes offline, asks, then needs an approval
tilldev customer-keys protect "Prod key" tillauth --yes --acknowledge "my users cannot sign in while the key refuses"
tilldev customer-keys window "Prod key" 5
tilldev customer-keys unprotect "Prod key" tillsecrets
tilldev customer-keys rm "Prod key"

From the API

The endpoints are under Customer keys in the API reference. An API key with org.read can read keys and their health; every change needs an owner signed in.

curl -s https://tilldev.dev/api/org/customer-keys -H "authorization: Bearer $TOKEN"

curl -s -X POST https://tilldev.dev/api/org/customer-keys/$KEY/test \
  -H "authorization: Bearer $TOKEN"