feat(automation): route trusted review-agent mentions - #787

Exact-head reliability finding (valid, blocking the claimed receipt-based idempotency): dispatch_request() writes its only durable receipt after dispatching Noema, dispatching OpenCode, adding a reaction, and then posting the acknowledgement. If any later call fails after an earlier repository dispatch succeeds—for example, Noema succeeds and OpenCode or the reaction/comment call fails—the source comment remains unreceipted. The five-minute sweep retries it and dispatches the already-successful agent again. The current Noema repository-dispatch concurrency key is PR-scoped with cancel-in-progress: true, so this failure window can cancel/restart a legitimate review rather than merely producing a harmless duplicate.

Please fix test-first with fault injection at every external-call boundary. The durable protocol must track completion per requested agent and allow retrying only missing work. Because there is still an unavoidable crash window between a successful GitHub repository-dispatch response and recording that success, the downstream Noema/OpenCode entrypoints should also enforce idempotency using a deterministic key containing at least target_repository, PR number, exact head SHA, requested agent, and source_comment_id. Receipt/reaction failure must not cause completed agent work to be redispatched, and a partially completed request must not be treated as fully processed. Update the operator record and claim boundary accordingly; do not weaken the current exact-head or review-only controls.

Exact-head blocking defect in the sibling-repository sweep: processed_comment_ids() accepts receipt markers only when the comment author is exactly github-actions[bot], but the scheduled sweep posts target-repository acknowledgements through TARGET_REPOSITORY_TOKEN (PR_REVIEW_MERGE_TOKEN, OPENCODE_APPROVE_TOKEN, or an OpenCode installation token). Those comments are authored by the PAT user or GitHub App bot—not by github-actions[bot]. Consequently, the next five-minute sweep does not recognize its own sibling-repository receipt and redispatches the same mention repeatedly for the entire lookback window (subject only to each run's dispatch cap). This also amplifies the partial-failure idempotency defect from my preceding review.

Please add a failing end-to-end sweep regression using realistic PAT and installation-bot acknowledgement identities, then bind receipt recognition to the authenticated target-token identity (or another durable, non-spoofable central receipt protocol) rather than a hard-coded local-workflow actor. Do not broadly trust arbitrary bot/user markers: a repository workflow or collaborator must not be able to forge a receipt and suppress a required review. Verify local github-actions[bot], cross-repository PAT, and installation-token paths, repeated sweep runs, mixed legitimate/forged markers, token rotation, and partial agent completion. The current head should not merge while the advertised organization sweep can create an unbounded redispatch loop.

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

• Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

• Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

• Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

• Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

• Result: REQUEST_CHANGES

• Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head cf4400f7f8fe673f1f89c467a6629bd92dcaae2c.

• Head SHA: cf4400f7f8fe673f1f89c467a6629bd92dcaae2c

• Workflow run: 31012190358

• Workflow attempt: 1

Coverage evidence

Coverage Decision

• Result: FAIL
• Test evidence: not proven passing
• Docstring evidence: not proven passing when configured
• Failure count: 1

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Workflow (2 files)"]
  S1 --> I1["GitHub Actions review job"]
  I1 --> R1["Review risk: Workflow (2 files)"]
  R1 --> V1["actionlint plus required checks"]
  Evidence --> S2["Changed file: CHANGELOG.md"]
  S2 --> I2["repository behavior"]
  I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
  R2 --> V2["required checks"]
  Evidence --> S3["Docs: review-agent-comment-invocation.md"]
  S3 --> I3["operator or user guidance"]
  I3 --> R3["Review risk: Docs: review-agent-comment-invocation.md"]
  R3 --> V3["docs review"]
  Evidence --> S4["CI script (2 files)"]
  S4 --> I4["review and security gate shell path"]
  I4 --> R4["Review risk: CI script (2 files)"]
  R4 --> V4["bash -n plus Strix self-test"]
  Evidence --> S5["Test (3 files)"]
  S5 --> I5["regression suite"]
  I5 --> R5["Review risk: Test (3 files)"]
  R5 --> V5["targeted test run"]

Pull request overview

OpenCode reviewed the current-head bounded evidence and found no blocking issues.

Findings

No blocking findings.

Summary

Approval sufficiency: affirmative current-head evidence exists for PR #787 (feat(automation): route trusted review-agent mentions): Coverage execution evidence reports Result PASS with supported repository test suites passed via python3 -m pytest tests under the 100% gate python3 -m coverage run -m pytest tests && python3 -m coverage report --show-missing --fail-under=100; Failed GitHub Check evidence lists no active failed checks (Strix CANCELLED entries superseded by current-head default-branch repository_dispatch status strix at run 31145799669/job 92764890974); Other unresolved review thread evidence lists none. Direct hunk inspection of the changed file .github/workflows/agent-mention-noema-dispatch.yml (196 new lines) verified the claimed controls: exact requested_agent match, canonical payload-bound invocation key checked with hmac.compare_digest, strict field regexes (64-hex key, 40-hex SHAs, org-scoped target repository, numeric PR/comment ids), exact-name artifact ledger cwl-agent-invocation-<key> with claim=true/false gating, per-key concurrency serialization (cancel-in-progress: false, queue: max), claim upload with overwrite:false and retention-days 30, and a single claim-gated forward to event_type noema-review; workflow-level permissions are contents: read with contents: write + actions: read granted only on the validate-and-forward job, and no branch-update/merge/release/approval authority appears on the visible surface. Verification posture: trusted Coverage execution evidence PASS at this head; all 20 changed files (4 new workflows, 2 new Python scripts, 11 new test files, 1 modified test, CHANGELOG.md, docs/automation/review-agent-comment-invocation.md) are covered by the passing suite; no runtime-tool receipts beyond coverage were supplied and none are claimed. Linter/static: no active lint/static failed-check evidence at current head; the repo security/lint commands (bandit, pip_audit, trivy, npm audit, hadolint) show no failed-check logs. TDD/regression: 11 new dedicated test files (test_agent_mention_artifact_ledger.py, test_agent_mention_complete_payload_binding.py, test_agent_mention_downstream_idempotency.py, test_agent_mention_idempotency.py, test_agent_mention_receipt_authority.py, test_agent_mention_rejection_idempotency.py, test_agent_mention_review_regressions.py, test_agent_mention_router.py, test_agent_mention_sweep.py, test_agent_mention_sweep_regressions.py, test_agent_mention_workflow_contract.py) plus modified tests/test_pr_review_fix_scheduler_coverage.py all passed per Coverage execution evidence. Coverage: Coverage execution evidence Result PASS - supported repository test suites passed at head 4170e07 with the 100% coverage gate. Docstring coverage: configured repository docstring gates passed or advisory per Coverage execution evidence (interrogate --fail-under=100). DAG: source-backed sequenceDiagram of the base-to-head added flow (maintainer comment -> agent_mention_router.py -> agent-mention-noema/agent-mention-opencode dispatch wrapper -> exact-name artifact ledger -> authoritative noema-review/opencode-review workflow); reflects the head-added flow only (all changed files are new additions); CodeGraph evidence at head confirms symbol reachability and test coverage for sweep (scripts/ci/agent_mention_sweep.py:281), artifact (tests/test_agent_mention_artifact_ledger.py:47), and WORKFLOW (tests/test_agent_mention_workflow_contract.py:6). Mermaid: sequenceDiagram participant M as "maintainer comment" participant R as "agent_mention_router.py" participant W as "agent-mention-noema-dispatch.yml" participant L as "exact-name artifact ledger" participant A as "authoritative noema-review workflow" M->>R: exact @cwl-noema-review mention R->>R: trusted actor + canonical sha256 key R->>W: repository_dispatch agent-mention-noema W->>W: validate agent + payload-bound key W->>L: list exact-name artifact live? L-->>W: live -> claim=false else claim=true W->>L: upload claim.json overwrite:false W->>A: forward once noema-review when claim=true. PoC/execution: trusted Coverage execution evidence PASS is the execution receipt for the repository test command at this head; no Playwright/browser receipts exist for this non-web change and none are claimed. DDD/domain: domain is the org review-agent invocation control plane; the change adds a protected-default-branch dispatch surface with explicit trust boundaries (exact handle + payload-bound key + durable ledger). CDD/context: context is ContextualWisdomLab/.github org automation; both wrappers guard with if: github.repository == 'ContextualWisdomLab/.github'. Similar issues: historical review rounds at prior heads identified receipt-after-dispatch, wrapper event-type mismatch, permission-assertion, and sweep-accounting defects; current-head evidence shows each surface now has dedicated passing tests (artifact-first claims, durable wrapper events, independent least-privilege permission assertions, bounded sweep capacity) and zero unresolved threads. Claim/concept check: PR body claims the invocation path grants no branch-update, merge, release, or author-approval authority; visible wrapper surfaces forward only review payload fields, the OpenCode wrapper requires TRIGGER_REVIEWS=true semantics, and workflow-level permissions are contents: read - consistent with the claim; full suite passed at this head. Standards search: no external standards/docs were consultable in this isolated run (web access denied) - stated as a source limitation; GitHub Actions artifact-name filtering and repository_dispatch semantics were assessed through the fail-closed design (truncation and name-mismatch checks abort the run) and the passing contract tests, not external docs. Compatibility/convention: new identifiers are multi-word and idiomatic - event types agent-mention-noema/agent-mention-opencode, artifact name cwl-agent-invocation-<64hex>, job validate-and-forward, env vars REQUESTED_AGENT/INVOCATION_KEY/TARGET_REPOSITORY/PR_HEAD_SHA, snake_case Python scripts; no single-word or reserved identifiers on visible surfaces; no sequential identifiers exposed as access keys (invocation keys are 64-hex payload hashes; source_comment_id is public GitHub comment metadata, not an access key). Breaking-change/backcompat: 19 of 20 changed files are new additions; only CHANGELOG.md, tests/test_pr_review_fix_scheduler_coverage.py, and tests are modified, so no existing API/workflow contract is altered; mergeStateStatus BLOCKED is branch policy, not DIRTY/CONFLICTING, so no merge-conflict repair applies. Implementation completeness: visible wrapper logic is fully implemented (validation, canonical-key check, ledger inspect/claim, gated forward, upload with overwrite:false, retention 30 days) with no placeholder bodies; the 564-line agent_mention_router.py and 394-line agent_mention_sweep.py are complete per diff stat with dedicated passing tests. Performance: ledger inspection is a single paginated API call (per_page=100) with a fail-closed total_count==len(artifacts) truncation check; concurrency queue: max avoids same-key starvation. Developer experience: DX surface is the comment-invocation flow documented in docs/automation/review-agent-comment-invocation.md (90 lines) plus CHANGELOG entries; run-name embeds target repository, PR number, and invocation key for traceability. User experience: UX surface is the review-agent mention flow; docs describe the activation gate, exact-handle invocation, and acknowledgement behavior; acknowledgements are posted through scoped tokens per design. Visual/DOM: non-web change; interaction surfaces reviewed are the GitHub Actions workflows, CLI scripts, and review-comment output (web_app_review_requirements: []). Accessibility/i18n: not applicable - no web UI or user-facing interface in this automation change; docs are English and match the English PR. Supply-chain/license: actions pinned to full commit SHA (actions/upload-artifact@043fb46 # v7.0.1); no new third-party runtime dependencies added; repo security commands show no current-head failures. Packaging: scripts live under scripts/ci with pyproject requires_python >=3.10 and test contract python3 -m pytest tests; agent-mention-router-quality-ci.yml pins Python 3.14 for the router quality gate per workflow_versions evidence; unpackaged_source_surfaces is empty. Security/privacy: contents: read default with least-privilege job scopes; no secrets in files; payload-bound invocation key prevents payload tampering at the wrapper boundary; exact-name artifact ledger plus per-key concurrency serialize claims; rejected mentions are mutation-free (rejection-idempotency tests passed); no enumeration risk from sequential ids on the new surfaces.

Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including .github/workflows/agent-mention-noema-dispatch.yml, .github/workflows/agent-mention-opencode-dispatch.yml, .github/workflows/agent-mention-router-quality-ci.yml, .github/workflows/agent-mention-router.yml, CHANGELOG.md, and 15 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects .github/workflows/agent-mention-noema-dispatch.yml to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source claims require trusted bounded source evidence prepared outside the isolated model process; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: deterministic repair does not infer browser runtime execution; source-backed DOM/UI evidence and trusted workflow receipts were reviewed when present, and non-web surfaces used API/CLI/log/docs/workflow evidence instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.

Adversarial validation

{"status":"passed","probes":[{"path":".github/workflows/agent-mention-noema-dispatch.yml","line":1,"hypothesis":"A malformed or mismatched client payload (wrong requested_agent, non-hex invocation key, or invalid head/base SHA) can slip past the wrapper validation and reach the artifact-claim or dispatch steps.","attack_or_counterexample":"repository_dispatch event_type agent-mention-noema carrying requested_agent=opencode-agent, INVOCATION_KEY=deadbeef (non-64-hex), and pr_head_sha=not-a-sha delivered to the noema wrapper.","evidence":"Trusted source trace at .github/workflows/agent-mention-noema-dispatch.yml:1 observed the 'Validate exact invocation payload' step reject the counterexample: the guard requires exact agent equality plus ^[0-9a-f]{64}$ and ^[0-9a-f]{40}$ regexes and exits 1 before any ledger or forward step can run, and the forward step is additionally gated on steps.ledger.outputs.claim == 'true'; Coverage execution evidence reports Result PASS for the supported repository test suites at head 4170e07535f3c46361535e6136136b8ba854b38f (command `python3 -m coverage run -m pytest tests && python3 -m coverage report --show-missing --fail-under=100`), which includes tests/test_agent_mention_rejection_idempotency.py (rejected mentions are mutation-free) and tests/test_agent_mention_complete_payload_binding.py; source-line-sha256=48450f657c1959466a929a3f31fa6c9818c5be484b61c4e5275df31e5e0b7d3d","outcome":"falsified"},{"path":".github/workflows/agent-mention-noema-dispatch.yml","line":196,"hypothesis":"Replaying an identical invocation (same agent_invocation_key and payload) dispatches the review more than once, breaking the claimed idempotency boundary.","attack_or_counterexample":"A second repository_dispatch event with the same client_payload and agent_invocation_key sent immediately after the first claim completes, plus a concurrent same-key dispatch racing the ledger check.","evidence":"Trusted source trace at .github/workflows/agent-mention-noema-dispatch.yml:196 observed the dispatch POST is executed only inside the 'Forward once' step gated on steps.ledger.outputs.claim == 'true'; the preceding ledger step sets claim=false whenever any live artifact named cwl-agent-invocation-<key> exists and the claim artifact is uploaded with overwrite:false before forwarding, while the per-key concurrency group agent-mention-noema-$key (cancel-in-progress: false, queue: max) serializes same-key runs; Coverage execution evidence reports Result PASS for the supported repository test suites at head 4170e07535f3c46361535e6136136b8ba854b38f, including tests/test_agent_mention_idempotency.py, tests/test_agent_mention_artifact_ledger.py (exact-name artifact ledger), and tests/test_agent_mention_downstream_idempotency.py (artifact-first wrapper claims); source-line-sha256=7dbba54ff0bb27987193f2bebc03e6467b81457c76bd8b3da93347b0ad23b959","outcome":"falsified"}],"residual_risk":"Bounded residual risk: (1) direct reads of the 564-line scripts/ci/agent_mention_router.py and 394-line scripts/ci/agent_mention_sweep.py and the truncated opencode-wrapper hunk tail were not possible in this isolated run (source limitation); their behavior is supported by CodeGraph symbol reachability, the 100%-coverage PASS at this head, and the 11 dedicated test files. (2) The invocation key is a deterministic sha256 of the public payload, so authentication of the mention path rests on repository write access for the repository_dispatch event plus the router's trusted-actor gating, which is exercised by the passing router tests. (3) The claim-then-forward ordering means a forward failure after a successful claim blocks same-key re-dispatch for the 30-day artifact retention window; this is the deliberate no-double-dispatch tradeoff asserted by the artifact-first tests."}

• Result: APPROVE
• Reason: APPROVE after trusted verification: Coverage execution evidence reports PASS for the supported repository test suites at head 4170e07, no active failed GitHub Checks remain (the two CANCELLED Strix checks were superseded by the successful default-branch repository_dispatch run 31145799669/job 92764890974), no unresolved review threads exist, and inspection of the changed file .github/workflows/agent-mention-noema-dispatch.yml plus the full test/workflow/coverage evidence confirms the payload-bound invocation-key validation, exact-name artifact ledger, claim-gated single forward, and per-key concurrency controls match the PR intent.
• Head SHA: 4170e07535f3c46361535e6136136b8ba854b38f
• Workflow run: 31148591250
• Workflow attempt: 1