# Shield v1.3.0

**Release date:** 2026-06-13
**Code name:** "hardening-sprint"

This release fixes all P0 + most P1 items from the IMPROVEMENTS.md
backlog. The focus is real bugs (P0) and high-value hardening (P1);
P2/P3 polish items remain on the roadmap.

## What ships

### 1. `PUBLIC_BASE_URL` becomes the canonical public origin
- `services/email.js` now reads `publicBaseUrl()` (from
  `config/publicBaseUrl.js`) when building verification / password
  reset links. Accepts `PUBLIC_BASE_URL` → `APP_URL` → dev fallback.
- `config/assertProductionEnv.js` refuses to boot in
  `NODE_ENV=production` if `PUBLIC_BASE_URL`, `SESSION_SECRET`, or
  `TRUST_PROXY` are missing. Prevents the previous silent
  `undefined/auth/verify?token=...` bug.
- `.env.example` now lists both names with a comment explaining the
  rule.

### 2. `autoEscalate` predicate fixed (regression test included)
- File: `jobs/autoEscalate.js` — `COALESCE(sent_at, 0)` was returning 0
  for NULL rows, making `escalated_at < 0` always false → takedowns
  with `sent_at IS NULL` (chain-only rows, legacy data) never
  escalated. Fix: `COALESCE(sent_at, next_escalate_at - 1)` so the
  sentinel is always < any real `escalated_at`.
- `test/autoEscalate.test.js` has 5 cases including the regression
  test that would have caught the bug originally.

### 3. Rate limiters wired
- `scanLimiter` (10 req/min) now applies to **both** `/scan/*` and
  `/cases/:id/discover` (the most expensive endpoints).
- New `twofaLimiter` (20 req/min) on `POST /2fa/auth/login/2fa` to
  prevent 6-digit TOTP brute force.
- `apiLimiter` was already on `/api/v1/*`.

### 4. `/scan/video` is now crash-safe
- Wrapped in try/catch with structured `{error, ref, message}` envelope.
- Raises a severity=`error` alert with the ref, stack, and case_id.
- Every error path now returns a ref the operator can quote to support.

### 5. `autoEscalate` skips non-email chain steps
- Chain steps whose address is a URL (HackerOne, ICANN form) are
  skipped silently — the operator must escalate manually to those.

### 6. `POST /admin/production-pending/:id/send` validates the contact
- Refuses to send to non-email addresses (catches data-entry typos).
- Refuses to send to `example.com`, `test.com`, `.invalid`, etc.

### 7. `POST /cases/:id/discover` is now duplicate-guarded
- 409 + existing `scan_id` if there's a queued or running scan for
  this case in the last 5 minutes. Prevents spam-click waste.

### 8. 2FA wizard on the admin dashboard
- `public/js/admin.js` listens for `TWO_FA_REQUIRED` 401s and
  auto-renders an enrollment banner with QR code (via Google Chart
  API) + secret + verify form, or a verify-only banner if already
  enrolled.
- Backup codes are displayed once after enrollment and the page
  reloads automatically.
- No more "admin gets 401 and has no idea what to do" footgun.

### 9. `/health` endpoint (liveness + readiness)
- `GET /health` returns 200 if DB ok + workers running + at least one
  active search proxy + transactional email key; 503 otherwise. Body includes uptime,
  worker states, key counts, node_env, version.
- `GET /health/live` — always 200.
- `GET /health/ready` — 200 if DB is reachable, 503 otherwise.
- No auth, no CSRF. Mount for external monitoring.

### 10. Structured logger
- `middleware/logger.js` — JSON lines to stdout + `data/server.log`,
  with request-id correlation (`X-Request-Id` header).
- Auto-redacts `password`, `passwd`, `secret`, `token`, `key`,
  `api_key`, `Authorization`, `cookie`, `set-cookie`, `signature`
  fields. Long opaque strings (JWTs, API keys) get a tail-truncation
  with `len=` annotation.
- Wired in `app.js` as `requestLogger` middleware; replaces the
  error handler's `console.error` with `logger.error`.
- Old `console.log/warn` calls still work; new code should call
  `logger.info/warn/error` directly.

### 11. Evidence retention cron
- `jobs/evidenceRetention.js` — prunes `case_discovery_scans` rows
  older than `NCII_EVIDENCE_RETENTION_DAYS` (default 90) AND their
  matching `data/evidence/case_*/scan_*.json` files. Runs on startup
  + every 24h.
- Set `NCII_EVIDENCE_RETENTION_DAYS=0` to disable (legal hold mode).

### 12. Test suite
- `node test/run.js` — no test framework dep, just `node:assert`.
- 5 test files, 23 tests, all green:
  - `apiKey.test.js` — prefix/shape validation per service
  - `autoEscalate.test.js` — predicate correctness + the NULL-sent_at regression
  - `publicBaseUrl.test.js` — fallback chain + assertProductionEnv
  - `logger.test.js` — JSON emission + redaction
  - `health.test.js` — 200/503 logic + liveness
- `npm test` runs them all.

### 13. Schema migration unified
- The `takedowns` table now declares `discovery_json`,
  `escalation_json`, `escalated_at`, `next_escalate_at` in
  `config/db.js` (was previously added at runtime by
  `routes/takedown.js` on first deploy). `db.init()` adds the
  columns to legacy DBs that don't have them.
- `case_discovery_scans.updated_at` is also added by `init()` if
  missing.

## Bug fixes since v1.2.0
- **Email URLs were `undefined/...`** when only one of `APP_URL` /
  `PUBLIC_BASE_URL` was set. Now asserted at startup.
- **auto-escalation never fired** on rows with `sent_at IS NULL`.
- **`/scan/video` returned 500** with no `ref` and no alert on any
  unhandled exception.
- **Spam-clicking "Run deep discovery"** queued 5+ scans; now 409s
  on the 2nd.
- **Production-pending "Send all"** could send to `abuse@example.com`
  if data was dirty. Now validates email + refuses reserved TLDs.
- **CSP/CSRF for webhooks** was implicit; now there's a comment in
  `app.js` so the next dev doesn't break it.

## Operator guide

### Local dev
```sh
node app.js
# tests
npm test
```

### Production
- New required env vars: `PUBLIC_BASE_URL`, `SESSION_SECRET`,
  `TRUST_PROXY`. App refuses to boot without them.
- `NCII_REQUIRE_SUPERADMIN_2FA=0` to disable the admin 2FA gate for
  staging only.

## Files changed
- `app.js` — requestLogger middleware, assertProductionEnv,
  publicBaseUrl, /health route, scanLimiter wiring, evidenceRetention
  start, comment on webhook CSRF exemption
- `config/assertProductionEnv.js` — new
- `config/publicBaseUrl.js` — new
- `config/db.js` — added takedowns columns to schema + migrations
- `middleware/logger.js` — new
- `middleware/rateLimit.js` — added twofaLimiter
- `routes/health.js` — new
- `routes/scan.js` — try/catch + alert + ref in /video
- `routes/cases.js` — duplicate-scan guard on /:id/discover
- `routes/admin.js` — email + reserved-TLD check on production_pending send
- `routes/twofa.js` — twofaLimiter on /auth/login/2fa
- `jobs/autoEscalate.js` — predicate fix + isEmail chain-step check
- `jobs/evidenceRetention.js` — new
- `services/email.js` — uses publicBaseUrl()
- `public/js/admin.js` — 2FA wizard
- `test/_setup.js`, `test/run.js`, `test/apiKey.test.js`,
  `test/autoEscalate.test.js`, `test/publicBaseUrl.test.js`,
  `test/logger.test.js`, `test/health.test.js` — new
- `package.json` — bumped to 1.3.0, added `test` script
- `.env.example` — both URL vars with explanation

## Backlog status
- ✅ P0 #1, #2, #3, #4, #5 (skipped — cosmetic)
- ⏸ P0 #6 (CSP unsafe-inline extraction) — parked; needs ~200-file
     refactor with nonce middleware
- ✅ P1 #7, #8, #10, #11, #12, #13
- ⏸ P1 #9 (asyncRoute sweep) — parked; 30-route sweep, do per-bug-fix
- ✅ P2 #14, #15 (already shipped as docs), #17, #18, #20
- ⏸ P2 #16 (configurable keyword lists per case), #19 (pHash dedup
     for new scan-vs-finding) — improvements, not security
- ⚪ P3 — all parked