Runbook — incident response

Severity

Sev	Definition	Response time
SEV-1	Data breach, wrong-money payout, regulator-visible outage.	Page immediately.
SEV-2	API 5xx > 1% for 5 min, MFA broken, payment provider outage.	15 min ack.
SEV-3	Single-feature degradation, background job backlog.	Business hours.

First 5 minutes

1. Ack the page and open an incident channel in Slack (#inc-<yyyy-mm-dd>-<short>).
2. Snapshot state. In the channel post:
   • GET /api/v1/health output
   • Prometheus links: 5xx rate, latency p95, DB connection saturation, queue depth
   • Sentry link for the top error in the last 15 minutes
3. Nominate an Incident Commander. IC coordinates; everyone else executes.

Common playbooks

API 5xx spike

• Check DB — SELECT COUNT(*) FROM pg_stat_activity WHERE state = 'active'. Saturated? Scale connections or shed traffic.
• Check Redis — BullMQ backlog on notifications or siem_export. If workers dead, restart.
• Roll back last deploy if error rate rose after it: helm rollback insurelink-api <prev-rev>.

Audit chain broken

• If AuditService.record starts logging chain integrity violation, freeze writes immediately:

  kubectl scale deploy/insurelink-api --replicas=0

• Investigate whether it's a real tamper event or a chainSeq gap from a rollback. Compliance officer is decision-owner.
• Never truncate audit_events. If you must repair, insert a chain.reset marker event via a signed migration.

Payment webhook mismatch

• Symptom: PaymentsService rejects an incoming webhook with signature_mismatch.
• Verify sender clock skew < 5 minutes.
• Confirm the provider is signing with the current PAYMENT_WEBHOOK_HMAC_SECRET (not _PREV).
• Never disable signature verification. If provider is genuinely broken, quarantine and reconcile offline.

DB failover

• Managed Postgres will promote the standby automatically. The API will spam P1001 for ~30s during failover.
• After failover: run npx prisma migrate status — verify no drift.
• Trigger a SIEM export push (POST /api/v1/siem/export/push) to confirm the cursor advanced past failover.

Comms

• Regulator-visible outage > 30 min: Compliance officer notifies IRA per the Insurance Act reporting obligation.
• Data breach: Legal + Compliance drive comms. Engineering provides chainSeq window + affected user IDs (from audit trail).

Post-incident

• Draft an incident review within 3 business days.
• File follow-up tickets. No blameless-culture theatre — just: root cause, corrective action, owner, due date.
• Update the relevant runbook if the playbook was wrong.