# Shield v1.5.0 — Operator workspace, webhooks, weekly digests

Shipped: 2026-06-16

## Headline

The operator's daily life just got easier. The dashboard now shows
**every takedown's full lifecycle state** in one table with health
probes, bulk actions, and a "Check now" button. Every state change
fires a **webhook** (Slack/Discord/Teams) and a **weekly email
digest** lands in the case owner's inbox so they never have to
log in to know what's happening.

And — **you can ditch transactional email**. The email transport is now
pluggable. Use transactional email, or direct SMTP to your own mail server,
or any of SendGrid / Mailgun / Postmark. The webhook path
needs **no email provider at all** — pure HTTP.

## What's new since v1.4.0

### 1. Pluggable email transport
`services/emailTransport.js` exposes a single `send()` interface
backed by one of:
- `resend` (default — no config change, backwards-compatible)
- `smtp` (direct SMTP via nodemailer — your own Postfix/Mailcow/Gmail)
- `sendgrid` / `mailgun` / `postmark` (HTTP API)
- `file` (writes .eml to `data/outbox/` for local dev / audit)

The active transport is stored in the `email_transport` table.
Switch at runtime via `POST /admin/email-transport` (or directly
via the env vars `SMTP_HOST` / `SMTP_PORT` / `SMTP_USER` / `SMTP_PASS`).

**Why this matters**: the operator was asking "can we do this
without going through transactional email?" The answer is yes. If transactional email's
quota runs out or you want to keep email fully on your own server
for compliance, just set the SMTP env vars and the system falls
back to SMTP automatically.

### 2. Webhooks (no email required)
`POST /admin/webhooks` creates an HTTP endpoint that receives
JSON payloads for: `takedown.sent`, `takedown.escalated`,
`takedown.removed`, plus any custom event. Optional HMAC-SHA256
secret for signature verification. Auto-disabled after 10
consecutive delivery failures.

Every takedown event in the system now calls
`services/notifier.notify('event.name', payload)` which fans out
to all matching webhooks. No SMTP or transactional email involved — this is
pure HTTP POST to your URL.

Slack example URL: `https://hooks.slack.com/services/...`
Discord example URL: `https://discord.com/api/webhooks/...`
Teams: use Power Automate or similar to expose a webhook URL.

### 3. Weekly + daily email digests
`/admin/digest-subscriptions` lets any user subscribe their email
to a summary: "every Monday 09:00 UTC, send me a recap of my cases
— new cases, takedowns sent, confirmed removed, still live, AI
classifications". The `services/digest.js` builder produces a
clean HTML + plaintext report and sends via the active email
transport. The `jobs/weeklyDigest.js` cron checks every hour
whether any subscription is due.

The Nisali case is the canonical example: she's the case owner
who shouldn't have to log in every day. The weekly digest tells
her "this week: 5 takedowns sent, 1 confirmed removed, 4 still
being escalated".

### 4. Per-takedown "Check now" button
`POST /takedown/:id/check-now` runs the same search proxy + classify
pipeline that the link-health worker uses, but synchronously, on
a single takedown. Capped at 6 calls/minute per user. Returns the
new health status (live / removed / blocked / soft_404 / dns_fail)
plus the updated takedown row. Cost: 1 search proxy credit per call.

The UI now shows a "🔄 Check" button on every takedown row.
Click → instant answer to "is the page still up right now?".

### 5. Bulk actions on the takedowns table
The dashboard now has a checkbox column on the takedown history.
Select 1-N rows, then:
- **Mark removed** — sets `status = 'removed'`, records `resolved_at`
- **Mark responded** — operator saw the platform reply
- **Mark failed** — escalation or platform error
- **Re-queue** — resets the 30h escalation window
- **Export CSV** — full takedown list, Excel-ready

`POST /takedown/bulk` does the update in one shot. Rate-limited
implicitly (max 200 IDs per request). Every update is audit-logged.

### 6. Global request timeout
`middleware/timeout.js` puts a hard deadline on every request.
Default 30s; longer for `/scan/*` (120s) and `/cases/*` (180s)
which can be slow. Returns a clean 504 with `{error, timeout_ms,
path}` envelope if the handler doesn't call `res.end()` in time.

Why: a hung handler (DB lock, search proxy timeout, regex infinite
loop) used to block the connection forever. Now the worst case
is a 504 in 30s.

### 7. Other improvements

- **Per-takedown health drilldown** — `GET /takedown/:id/health` returns full probe history with body-hash diffs. Operator can see "the page shrank by 95% on day 3 — that's when the platform nuked it".
- **`POST /takedown/export.csv`** — full takedown list as CSV, Excel-ready, for sharing with lawyers / auditors.
- **Better autoEscalate visibility** — when a chain step is a URL (not an email, e.g. ICANN lookup form), the worker raises an alert instead of silently skipping. The operator now sees "Auto-escalate blocked: fapedia.net → registrar step is a URL" instead of nothing.
- **Body-shrink detection** — if the page body suddenly shrinks by 80% (e.g. replaced with a 30-byte "OK" page), the classifier calls it `soft_404` even if the page returns 200.
- **gen_mail JS-obfuscation parser** — `extractEmailsFromHtml` now decodes the `gen_mail('lhs', 'rhs')` pattern (EroMe, many older PHP sites) so `contact@erome.com` is auto-extracted from `/s/report` without manual intervention.
- **search proxy quota classifier** — when search proxy returns "Monthly request limit exceeded", the health probe records it as `blocked` (we can't tell) instead of `dns_fail` (host is gone). Prevents false-positive "removed" alerts when the issue is our quota, not the platform.
- **Per-finding dashboard panel** with category badges, MHTML download, manual classify, open viewer.

## Schema additions (in config/db.js)

- `link_health_checks` (takedown_id, url, status, http_status, reason, body_size, body_hash, response_ms, checked_at) — one row per probe
- `monitored_urls` (url, domain, case_id, reason, added_by, next_check, check_count, last_status) — operator-managed watch list for cross-case monitoring
- `finding_classifications` (finding_id, category, confidence, evidence, summary, model, ai_call_id, triggered_by, classified_at) — one row per AI categorization
- `webhooks` (name, url, secret, event_filter, active, owner_id, last_status, last_fired_at, failures)
- `webhook_deliveries` (webhook_id, event, payload, status, response, attempt, delivered_at)
- `email_digest_subscriptions` (user_id, email, cadence, day_of_week, hour_utc, active, last_sent_at)
- `email_transport` (name, active, config_json) — pluggable transport config

## Files changed / added (since v1.4.0)

```
.env                                          ← (unchanged) operator sets SMTP_*/RESEND_*
app.js                                        ← request timeout + digest cron + webhook mount
config/db.js                                  ← 4 new tables (webhooks, deliveries, digest, transport)
services/webhook.js                           (NEW) pure-HTTP event delivery + HMAC + auto-disable
services/notifier.js                          (NEW) fan-out helper for code paths that emit events
services/emailTransport.js                    (NEW) pluggable transport (smtp/resend/sendgrid/mailgun/postmark/file)
services/digest.js                            (NEW) weekly/daily digest builder (HTML + text)
services/linkHealthChecker.js                 ← search proxy quota → blocked; body-shrink detection
services/contactDiscovery.js                  ← gen_mail() pattern extractor
jobs/weeklyDigest.js                          (NEW) cron — every 1h, sends due digests
jobs/linkHealthWorker.js                      ← 'blocked' status skips re-probe for 6h
routes/adminWebhooks.js                       (NEW) /admin/webhooks + /admin/digest-subscriptions
routes/takedownCheckNow.js                    (NEW) POST /takedown/:id/check-now + /bulk + /export.csv
middleware/timeout.js                         (NEW) global request timeout
public/views/dashboard.html                   ← bulk toolbar + check-all + Check buttons
public/js/dashboard.js                        ← bulk + Check now + CSV export wiring
test/webhook.test.js                          (NEW) 5 tests, all green
test/_setup.js                                ← truncate webhooks/deliveries/etc. between tests
test/linkHealth.test.js                       ← 18 link health tests
```

## Tests

```
$ node test/run.js
…
  64 passed · 0 failed
```

(59 in v1.4.0 + 5 new webhook tests.)

## Deploying to cPanel

1. Upload `ncii-shield-1.5.0.tar.gz` to the file manager.
2. Extract into the app root (e.g. `~/ncii-shield/`).
3. The tarball excludes `data/` and `.env` — your existing case
   data + admin user + API keys are preserved.
4. Run `npm ci --omit=dev --no-audit --no-fund` in the app
   directory (one new dep: nothing — the system uses native
   `fetch`, `crypto`, and `nodemailer` which is already a dep).
5. Click Restart in cPanel.
6. Visit `/health` → 200.
7. Visit `/admin` → **Webhooks & Digests** section to set up:
   - Webhook URLs for Slack/Discord/Teams (one-time setup)
   - Digest email subscriptions per user (click "Subscribe to
     weekly digest" and pick day/hour)

## Operator FAQ

**Q: Can I do this without transactional email?**
A: Yes. Set `SMTP_HOST` / `SMTP_PORT` / `SMTP_USER` / `SMTP_PASS`
in the env (or set them via the `email_transport` table), and
the system falls back to SMTP. Supported: transactional email, SMTP, SendGrid,
Mailgun, Postmark, file.

**Q: How do I get notified of takedown events?**
A: Add a webhook in `/admin → Webhooks`. Slack:
`https://hooks.slack.com/services/...`. Discord: `https://discord.com/api/webhooks/...`.
Every `takedown.sent` / `takedown.escalated` / `takedown.removed`
event fires a JSON POST to that URL. Optional HMAC signing.

**Q: My webhooks aren't firing — what do I check?**
A: `/admin → Webhooks → click the row → "Last 50 deliveries"`. Each
delivery shows HTTP status + body excerpt. 4xx/5xx means the
endpoint rejected; 0 means network failure.

**Q: My case owner (Nisali) doesn't have time to log in every day.**
A: Subscribe her email to the weekly digest:
`POST /admin/digest-subscriptions` with `{email, cadence: "weekly",
day_of_week: 1, hour_utc: 9}`. She'll get a Monday morning summary
email with: new cases, takedowns sent, confirmed removed, still
live, AI classifications.

**Q: How much does the "Check now" button cost?**
A: 1 search proxy credit per call. Rate-limited to 6 calls/minute
per user. The link-health worker uses the same 1-credit probe on
its 4-hour cycle, so manual checks don't add new overhead.

**Q: Why does the dashboard show a "still live" badge even when
I just talked to the platform rep?**
A: The platform may have removed the content from public view
but not from the URL itself (e.g. set to "private" instead of
deleting). Click "🔄 Check" to re-probe right now.

— shipped 2026-06-16