# Runbook — incident response

## Severity

| Sev | Definition | Response time |
| --- | --- | --- |
| SEV-1 | Data breach, wrong-money payout, regulator-visible outage. | Page immediately. |
| SEV-2 | API 5xx > 1% for 5 min, MFA broken, payment provider outage. | 15 min ack. |
| SEV-3 | Single-feature degradation, background job backlog. | Business hours. |

## First 5 minutes

1. **Ack the page** and open an incident channel in Slack (`#inc-<yyyy-mm-dd>-<short>`).
2. **Snapshot state.** In the channel post:
   - `GET /api/v1/health` output
   - Prometheus links: 5xx rate, latency p95, DB connection saturation, queue depth
   - Sentry link for the top error in the last 15 minutes
3. **Nominate an Incident Commander.** IC coordinates; everyone else executes.

## Common playbooks

### API 5xx spike

- Check DB — `SELECT COUNT(*) FROM pg_stat_activity WHERE state = 'active'`. Saturated? Scale connections or shed traffic.
- Check Redis — BullMQ backlog on `notifications` or `siem_export`. If workers dead, restart.
- Roll back last deploy if error rate rose after it: `helm rollback insurelink-api <prev-rev>`.

### Audit chain broken

- If `AuditService.record` starts logging `chain integrity violation`, freeze writes immediately:
  ```bash
  kubectl scale deploy/insurelink-api --replicas=0
  ```
- Investigate whether it's a real tamper event or a `chainSeq` gap from a rollback. Compliance officer is decision-owner.
- Never truncate `audit_events`. If you must repair, insert a `chain.reset` marker event via a signed migration.

### Payment webhook mismatch

- Symptom: `PaymentsService` rejects an incoming webhook with `signature_mismatch`.
- Verify sender clock skew < 5 minutes.
- Confirm the provider is signing with the current `PAYMENT_WEBHOOK_HMAC_SECRET` (not `_PREV`).
- Never disable signature verification. If provider is genuinely broken, quarantine and reconcile offline.

### DB failover

- Managed Postgres will promote the standby automatically. The API will spam `P1001` for ~30s during failover.
- After failover: run `npx prisma migrate status` — verify no drift.
- Trigger a SIEM export push (`POST /api/v1/siem/export/push`) to confirm the cursor advanced past failover.

## Comms

- **Regulator-visible outage** > 30 min: Compliance officer notifies IRA per the Insurance Act reporting obligation.
- **Data breach**: Legal + Compliance drive comms. Engineering provides `chainSeq` window + affected user IDs (from audit trail).

## Post-incident

- Draft an incident review within 3 business days.
- File follow-up tickets. No blameless-culture theatre — just: root cause, corrective action, owner, due date.
- Update the relevant runbook if the playbook was wrong.