Runbook — encryption / HMAC key rotation

Keys in use

Env var	Purpose	Rotation cadence
SECRETS_CIPHER_KEY	AES-256-GCM DEK for SecretsCipher (TOTP secrets, refresh tokens, API-key material).	12 months.
PII_CIPHER_HMAC_KEY	HMAC-SHA256 deterministic search hash for PII columns (client*Hash).	Only on suspected compromise — rotating breaks duplicate lookup on legacy rows without a re-hash migration.
PAYMENT_WEBHOOK_HMAC_SECRET	HMAC signing key for outbound payment-provider webhooks + inbound verification.	6 months, coordinated with provider.
SIEM_EXPORT_SECRET	HMAC signing key for ${ts}.${body} on SIEM push.	12 months, coordinated with SIEM operator.
JWT_ACCESS_SECRET / JWT_REFRESH_SECRET	JWT HS256 signing keys.	12 months. Refresh-token family revocation cushions the switch.

Standard rotation (planned)

1. Announce the window. Notify compliance + partner integrations 72h in advance.
2. Generate the new key. 32 random bytes, base64:

   openssl rand -base64 32

3. Add as a versioned secret. In the secret store add SECRETS_CIPHER_KEY_NEXT alongside the current value.
4. Deploy code that reads both. Ciphers already accept KEY (primary) + KEY_NEXT (decrypt-only fallback). Confirm at boot: GET /api/v1/health — the ciphers.status field should read dual.
5. Flip primary. In the next deploy, promote KEY_NEXT → KEY, keep the old value as KEY_PREV.
6. Backfill. Run:

   npm run script -- rotate-secrets --from KEY_PREV --to KEY

   This re-encrypts every column in-place inside a serializable transaction.
7. Remove KEY_PREV. After the backfill's audit event fires, delete the old secret from the store and redeploy.
8. Verify. SELECT COUNT(*) FROM audit_events WHERE action = 'crypto.rotation.completed' should show one new row.

Emergency rotation (compromise suspected)

1. Revoke all refresh-token families — POST /api/v1/auth/revoke-all as super-admin.
2. Rotate JWT signing keys immediately. Existing access tokens will fail signature check; clients re-authenticate.
3. Rotate the compromised key. Skip steps 1-4 above and go straight to promotion + backfill.
4. Notify. Compliance officer files a data-incident notification if PII decryption keys were exposed. Include chainSeq range of audit events written during the exposure window in the disclosure.
5. Post-incident. File a follow-up ticket to add detection for the exposure vector.

Rollback

If the backfill script errors partway:

• The script commits per-batch, so partial rows are re-encrypted under KEY.
• Re-run the script — it is idempotent (checks a version marker column).
• Do not delete KEY_PREV until the script exits 0.