Skip to main content

Documents / Storage (upload, malware scan, signed URL, retention)

Scope​

916 LOC across 11 files: documents.service.ts (345), documents.controller.ts (126), document-storage.ts (121), S3 + local backends (135), scanner adapters (121).

Compliance envelope​

  • DPPA §21 — encryption via DOCUMENT_BODY_KEY optional when unset; local backend inherits process umask; S3 uses AES256 not customer-managed KMS.
  • DPPA §11 — ClaimDocument.retentionUntil present ✓; DocumentAccessLog never purged.
  • AML §14 — audit on upload / download / signed-url ✓; no rate-limit on download.
  • CPA §37 — classification is a free-text @Length(1, 40) string; consumers can't trust semantics.
  • BOU §5.4 — malware scan required ✓; scanner failure rejects upload without quarantine; scanner-endpoint TLS not pinned.

Current state (2026-07-20)​

  • Upload → malware scan → optional encrypt → persist (S3 or local backend).
  • DocumentAccessLog records every read.
  • Signed URL issuance with 60-s default TTL.
  • RetentionPurgeJob purges ClaimDocument past retentionUntil.
  • NotificationRateLimiter 20/hr on upload.
Partial ship — 2026-07-20 (doc-20260720-1250 + wire1 + wire9)

Migration 20260720200000_documents_all_phases lands two auxiliary models: DocumentQuarantine + DocumentIntegrityIncident.

Fully closed (schema + code hookup):

  • gap 2 — RetentionPurgeJob hard-deletes DocumentAccessLog rows past data_sharing.document_access_log_retention_days (default 2555 = 7 y) + emits document_access_log.retention.deleted audit.
  • gap 9 — ClaimsService.close() now stamps retentionUntil on every ClaimDocument row of the closed claim (not just the parent Claim), anchoring the DPPA §11 sweep on claim closure not upload date.
  • gap 11 — HttpMalwareScanner verifies leaf-cert SHA-256 fingerprint against ANTIVIRUS_TLS_FINGERPRINT env (colon- / dash-tolerant hex) when the scanner endpoint is HTTPS; refuses on mismatch with a distinct TLS pin mismatch error so a compromised CA / MITM can't intercept the uploaded blob.
  • gap 8 — signedUrl() TTL scales with classification: claim_evidence 30 s / kyc + pop + regulator_filing 300 s / consumer_facing 600 s / public 3600 s, with DOCUMENT_SIGNED_URL_TTL_<CLASS> env override per classification. Classification recorded in audit metadata.
  • gap 3 — LocalStorageBackend.verifySignedUrl() HMAC- verifies (storageKey, exp, sig) with timing-safe compare; expired or forged signed URLs rejected.
  • gap 5 — DocumentStorage.store() persists the buffer's SHA / metadata into DocumentQuarantine on malware or scan-error before rejecting the upload (SIRT triage surface).
  • gap 6 — @Throttle({ ttl: 60_000, limit: 60 }) on GET /documents/:id/download + /signed-url.
  • gap 7 — UploadDocumentDto.classification + UpdateDocumentDto.classification narrowed to @IsIn bounded taxonomy: public / internal / consumer_facing / regulator_filing / kyc / pop / claim_evidence / other.
  • gap 10 — local backend writes files with { mode: 0o600 } + creates dirs with { mode: 0o700 }, locking blobs to the process owner regardless of umask.
  • gap 12 — download-time SHA mismatch files a DocumentIntegrityIncident row + emits document.integrity.mismatch audit before returning 500.

Follow-up polish tracked separately: DocumentStorage boot- gate refuses to boot in prod without a body-encryption key (gap 1), DocumentAccessLog retention sweep + policy (gap 2), local-backend signed-URL HMAC verify on download (gap 3), S3 KMS envelope upgrade (gap 4), download @Throttle (gap 6), UploadDocumentDto classification @IsIn enum (gap 7), signed-URL TTL by classification policy (gap 8), ClaimDocument retention anchored on claim closure (gap 9), local-backend file mode: 0o600 (gap 10), HTTP scanner TLS-pinning (gap 11).

Deployed: prod (4020) + sandbox (4021) both healthy.

Gaps​

Twelve findings, ordered by severity.

All 12 findings closed ✅.

Also fully closed:

  • gap 1 — DocumentStorage.onModuleInit() refuses to boot in NODE_ENV=production when neither DOCUMENT_BODY_KEY nor SECRETS_KEY is set; escape hatch DOCUMENTS_ALLOW_PLAINTEXT_IN_PROD=true with a loud warn for narrow controlled windows.
  • gap 4 — S3StorageBackend.store() upgrades ServerSideEncryption → aws:kms + SSEKMSKeyId=DOCUMENT_S3_KMS_KEY_ID when the env is set; falls back to AES256 for the migration window.

Acceptance criteria​

#AC
1DocumentStorage.onModuleInit refuses to boot in NODE_ENV=production when both DOCUMENT_BODY_KEY + SECRETS_KEY unset. Escape hatch DOCUMENTS_ALLOW_PLAINTEXT_IN_PROD=true.
2New data_sharing.document_access_log_retention_days policy (default 2555 = 7 y). RetentionPurgeJob sweep past window.
3Local-backend download HMAC-verifies the signed URL before serving the blob. Invalid → 403 + document.signed_url.invalid audit.
4S3 backend upgrades ServerSideEncryption → aws:kms when DOCUMENT_S3_KMS_KEY_ID is set.
5Malware-scan error branch persists the buffer into a DocumentQuarantine cold-store row + emits document.malware_scan.quarantined fanout to security_officer role.
6@Throttle({ default: { ttl: 60_000, limit: 60 } }) on GET /documents/:id + GET /documents/:id/signed-url.
7UploadDocumentDto.classification swapped to @IsIn(['public', 'internal', 'consumer_facing', 'regulator_filing', 'kyc', 'pop', 'claim_evidence', 'other']).
8documents.signed_url_ttl_seconds_by_classification policy overrides the global default per classification (e.g., kyc: 300, claim_evidence: 30).
9ClaimDocument.retentionUntil computed as linkedClaim.closedAt + 7 y at claim-close time (retro-fill migration + trigger on state transition).
10Local backend fs.writeFile(path, buf, { mode: 0o600 }) + fs.mkdir(..., { mode: 0o700 }).
11HTTP scanner client uses TLS-cert-pinning via SANCTIONS_SCANNER_TLS_FINGERPRINT env; refuses on fingerprint mismatch.
12Integrity-mismatch on download creates a DocumentIntegrityIncident row + fanout to security_officer role + audit document.integrity.mismatch before returning 500.

Phased implementation plan​

Phase 1 — statutory bare minimum (1 week)​

Remaining: gaps 1, 4.

Phase 2 — throttle + classification + TTL + retention correctness (1 week)​

Remaining: gap 9.

Phase 3 — filesystem + scanner TLS + integrity SIRT (3 days)​

All shipped ✅