Outbound webhooks: endpoints, a versioned event catalog, signed delivery with retries - #306

Summary

Outbound webhooks: CoderHelm now tells a team's own systems what it did. Admins add webhook endpoints (Settings, Webhooks, or the API with the webhooks scope), pick events, and CoderHelm sends each event signed the Standard Webhooks way, retries for about a day, turns off an endpoint after 5 days of failures (and tells admins), and records every attempt in Deliveries.

Paired with CoderHelm/coderhelm#223 (settings page, Deliveries changes, and the rewritten https://docs.coderhelm.com/api/webhooks/).

Changes

• common::webhooks (new)
  • catalog: the versioned event catalog (api_version 2026-10-26), each event's data schema and example. Only events with a real emit point are listed; a test reads the sources and fails if a catalog event has no emit.
  • event: the envelope {id, type, created_at, team_id, api_version, data} and emit_event, which only queues on the new webhooks queue so the work never waits on endpoints. Events are kept 14 days for Resend and replay; event ids are idempotency keys (a duplicate queue message is sent once; SNS redeliveries map to the same id).
  • endpoint: endpoints on the settings table, signing secret sealed with KMS (bound to team, endpoint and purpose), rotation with an overlap window (default 24h, max 168h) during which both secrets sign; health kept with conditional updates.
  • sign: webhook-id, webhook-timestamp, webhook-signature: v1,<base64 HMAC-SHA256 of id.timestamp.body>, tested against the Standard Webhooks test vector.
  • deliver: https only; the host is resolved before every send and every address must be public (private, loopback, link-local, CGNAT, metadata refused), the connection is pinned to the checked address (no DNS rebinding), redirects not followed, 10s timeout. Retries on timeouts, connection failures, 408, 429 and 5xx with backoff 5s, 1m, 5m, 30m, 1h, 3h, 6h, 6h, 8h (about 24.6h); other 4xx and 3xx are final. 5 days of failures with no success turns the endpoint off.
• Deliveries reused, not duplicated: records gain endpoint_id and source webhook, a Webhook redeliver kind and plain reasons with fixes; the Deliveries page's Redeliver resends webhook deliveries.
• Emits (one emit helper per service): run.created, run.completed, run.failed, run.needs_input (every status write, read back from the run record), pull_request.opened, review.completed (verdict and how it was posted), ci_fix.started / ci_fix.completed (pushed, no_changes, failed), alert.received, log_finding.created / log_finding.resolved (the Python log analyzer queues them in the same envelope), release_notes.published, plan.approved, plugin.run.completed (behind the plugins flag). Plus webhook.test.
• Gateway /api/webhooks/*: event types; endpoints list, create (secret shown once), get, update, delete; secret reveal and rotate (audited); send test event; endpoint deliveries; resend; replay a time range (only-failed option, 500 cap). All in permissions.rs under a new admin capability webhooks, public for API keys, with OpenAPI schemas and examples. The address is checked after DNS at save time too.
• OpenAPI: the endpoint routes, plus every event as an OpenAPI 3.1 webhooks entry. The endpoint routes are tagged "Webhook endpoints"; the received webhooks keep the "Webhooks" tag, so their reference page URLs don't change.
• Worker: webhook_emit (keep, fan out, first attempt) and webhook_send (retries; waits longer than SQS allows are carried by due_at). Admins get an email (new webhook-disabled SES template) and the team's alert chat channel a message when an endpoint is turned off.
• Security
  • Secrets: 32 bytes from the OS-seeded CSPRNG per endpoint, KMS-sealed and bound to team, endpoint and purpose; shown in full only at create, rotate and an audited admin reveal; never logged (a test reads every log call in the webhook code).
  • Signatures compared in constant time; receivers are told to check the 5-minute timestamp tolerance and skip repeated webhook-ids.
  • Payloads never carry credentials: every event's text is scrubbed (tokens by prefix, Bearer values, passwords in addresses, key=value secrets, private key blocks) and cut, for the log analyzer's events too.
  • SSRF: re-resolved before every send, private/loopback/link-local/CGNAT/metadata and their IPv6 and IPv4-mapped forms refused, connection pinned to the checked address, no redirects, 10s timeout, at most 4 KB of the answer read and never parsed.
  • Team scope: every route is admin-only and reads only the caller's team (a test checks each handler); a delivery is resent only through its own endpoint.
  • No inbound path: outbound webhook routes need an API key or session and none sits on the unauthenticated /webhooks router (tested).
• Infra (CDK only): <prefix>-webhooks queue (DLQ, 16 min visibility), worker event source (batch 1, max concurrency 5), send grants and WEBHOOK_QUEUE_URL for gateway, worker, streaming and the log analyzer; the email template.

Verification

• cargo fmt --all --check: clean. cargo clippy --workspace --all-targets -- -D warnings: clean.
• cargo test --workspace (rebased on main): common 1025 passed, gateway 471 passed, worker 688 passed, 0 failed. New tests cover the Standard Webhooks vector, multi-signature rotation overlap, timestamp tolerance, retry classes and the backoff total, disable after 5 days, SSRF at save and send (a name that rebinds to 169.254.169.254, a mixed public and private answer, literal private IPs never reaching DNS), real HTTP sends against a loopback server (headers verify with both secrets, a timeout retries, a 503 is recorded), secret sealing bound to team and endpoint, replay range limits, and each emit's data against the catalog schema. The OpenAPI document is regenerated and its tests pass.
• python3 -m unittest discover -s lambda/tests: 113 tests OK (envelope, uppercase ULID ids, failures never raise).
• npm test in infra: 61 pass, 0 fail (new: queue, event source, grants, env, email template). Self-host cdk synth --all -c config=config/selfhost.example.json: exit 0, no hosted domain in any template.
• Example payloads use https://coderhelm.example.com/…; PR Self-host: every link points at the install, never at coderhelm.com #312's no_hosted_urls lint, run against this branch, flags nothing in the webhook code.
• No AWS changes were made; everything ships through CDK.