Skip to content

fix(agent-org): address PR review correctness and performance findings - #419

Merged
Chloe-JY merged 2 commits into
developfrom
fix/issue-272-agent-org-recovery-invariants
Jul 18, 2026
Merged

fix(agent-org): address PR review correctness and performance findings#419
Chloe-JY merged 2 commits into
developfrom
fix/issue-272-agent-org-recovery-invariants

Conversation

@ShiboSheng

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #373 and the Agent Org work originally tracked in #272.

This PR addresses the correctness, performance, persistence, and test-fidelity findings from both review rounds on #373:

The product model remains unchanged:

  • Coordinators and workers reason about the work.
  • Rust validates and durably commits state transitions.
  • The scheduler performs budgeted Wake dispatch.
  • The frontend projects persisted facts.

This follow-up is intentionally limited to the two review rounds and the minimum integration dependencies required to make those fixes correct, independently testable, and production-compatible.

flowchart LR
    MODEL["Coordinator or worker proposes an action"]
    TOOL["Typed tool boundary"]
    TX["Short SQLite transaction"]
    WAKE["Budgeted Wake dispatch"]
    VIEW["Read-only compact Run View"]
    UI["Group Chat, Team Tasks, and Kanban"]

    MODEL --> TOOL --> TX --> WAKE
    TX --> VIEW --> UI
Loading

First review

Resolution matrix

Review finding Resolution Status
Run View reads performed writes. Frequent UI polling could take the global writer lock and mutate lifecycle state merely because a view was open. Run View now produces a deferred, read-only Snapshot. Reconciliation and intervention cleanup remain explicit write operations rather than polling side effects. ✅ Resolved
task_list used the global writer path. A read-only tool could serialize unrelated Task, Session, and Inbox work. task_list now uses a read transaction and returns bounded summaries. Full Task results are retrieved through task_get only when needed. ✅ Resolved
Polling payloads grew with Inbox, Plan, Task, and TaskOutput content. Every poll could repeatedly copy and parse increasingly large durable payloads. Run View now returns counts, previews, and Plan summaries. Full Plan, Task, Inbox, and Group Chat history use explicit detail or keyset-pagination APIs. ✅ Resolved
Task board loading repeated dependency graph work. Repeated lists and per-Task dependency scans created N+1 behavior. A transaction-local TaskGraphIndex is built once and reused by Task mutation, readiness evaluation, and transactional outbox decisions. ✅ Resolved
Finality checks disagreed across paths. A Run could repeatedly reconcile without reaching a stable terminal decision. Finality facts, assessment, blockers, and decisions are centralized. A durable work revision proves Coordinator observation, while org_run_complete handles legitimate empty-task Runs. ✅ Resolved
Idle Wake bypassed the recovery budget. Some Wake paths could repeatedly invoke the model while Watchdog recovery respected backoff. Production Wake sources now use durable reserve → enqueue → commit/refund accounting. Rejected and coalesced Wake requests do not consume an attempt. ✅ Resolved
Persisted Plan and text fields lacked direct bounds. One request could write unexpectedly large Plan content, feedback, Task descriptions, metadata, or messages. Review-requested fields are validated before entering files, SQLite, or Inbox. Existing history remains retained until explicit Run deletion. ✅ Resolved
Plan file I/O occurred while holding the shared writer lock. Slow filesystem work could block unrelated Agent Org updates. Plan content is prepared outside the lock. The lock protects only the short database transaction and atomic installation step. ✅ Resolved
Synchronous SQLite work ran on async execution threads. Database calls could stall unrelated async tasks. Synchronous persistence runs in blocking sections and reuses one connection per business operation. ✅ Resolved
Frontend rendering repeatedly reparsed completed outcomes. Large result wrappers were decoded during repeated renders. Explicit structured outcomes return early, and Task outcome resolution is memoized at the adapter boundary. ✅ Resolved
task_graph_create was not recognized consistently. Communication cards could display raw JSON and replay could erase existing Kanban state. Rust extraction, event routing, Task cards, Group Chat, and additive Kanban replay now recognize the structured Task Graph outcome. ✅ Resolved
One broken Run stopped Watchdog scanning. A database or analysis error in one Run could prevent later Runs from recovering. Watchdog now reports an inner per-Run error and continues scanning the remaining Runs. ✅ Resolved

Second review

Resolution matrix

Review finding Resolution Status
A failed or cancelled worker immediately abandoned the Run. Recoverable work could be declared terminal prematurely. Reviewed terminalization paths now use the shared finality model. Abandonment requires every relevant worker to be archived and no recovery path to remain. ✅ Resolved
Task Graph outcomes rendered differently across UI surfaces. Some views showed raw JSON while historical replay could clear the board. Group Chat, Team Tasks, and Workstation Kanban consume the same structured graph outcome and merge graph creation additively. ✅ Resolved
Rust E2E fixtures did not create a real Agent Org Run. Tests could pass while violating production Run invariants. Fixtures now initialize the canonical schema and create an actual Running Run before exercising Agent Org commands. ✅ Resolved
Watchdog errors were hidden and empty Recovery Plans could leave Runs stuck. Errors remain observable and isolated per Run. If reconciliation is rejected, recovery continues through valid Wake actions or a minimal Coordinator repair notice. ✅ Resolved
A legitimate empty-task Run could not complete. The Coordinator-only org_run_complete path records explicit completion intent and still passes through canonical finality validation. ✅ Resolved
Pause or restart cancelled pending Plan Approval. User approval state could disappear even though the Run remained resumable. Pending approval survives pause and restart. Only terminal or missing Runs clear it. ✅ Resolved
Crash windows could duplicate Inbox delivery. A durable Inbox row and the visible materialized turn could diverge across restart. Inbox materialization receipts and causation_inbox_id make acknowledgement conditional on successful durable materialization. ✅ Resolved
Legacy blocks dependencies were ignored by readiness checks. Old Tasks could unlock too early. Legacy blocks data is normalized into the same canonical dependency graph used by all readiness checks. ✅ Resolved
Workers could delete ownerless Tasks. A worker could remove coordinator-managed work it did not own. Ownerless Task deletion is now Coordinator-only. ✅ Resolved
Plan Approval could commit successfully but report failure when notification failed. Users might retry an already-committed decision. Approval state and Inbox notification commit together. Wake happens only after commit, and stale revisions return a readable structured error. ✅ Resolved
TypeScript treated args-only tool attempts as successful mutations. Failed attempts could alter historical Kanban state. New events require a structured persisted outcome. Legacy events require durable result evidence, and successful deletion removes the Task from replay. ✅ Resolved
Rust and TypeScript wire contracts drifted. executionMode could be missing and TaskOutput could be duplicated. executionMode is explicit, and TaskOutput now has one canonical wire location. ✅ Resolved
Tests initialized a weaker schema than production. Test-only behavior could conceal missing tables and invalid Run state. Production, unit, and HTTP E2E paths share schema initialization and real-Run invariants; test-only bypasses were removed. ✅ Resolved

Required integration dependency

The review fixes required one additional dispatch-time invariant:

Dependency Resolution Status
Direct and queued user messages must establish intervention only when actually dispatched. Establishing intervention while a message merely waits in the queue would suppress legitimate background recovery; failing to establish it before provider dispatch could race a Wake. Direct messages establish intervention after the durable user event and immediately before provider dispatch. Queued messages establish it only when dequeued. Intervention persistence failure prevents provider dispatch. ✅ Resolved

Resulting invariants

  1. Only a Running Run may mutate Tasks.
  2. Ownerless means unassigned, not available for arbitrary Worker self-claim.
  3. A normal Worker may mutate only its own assigned Task state.
  4. Task mutation and its TaskAssigned or TaskCompleted outbox commit together.
  5. Wake attempts are charged only after scheduler acceptance.
  6. Rejected or coalesced Wake requests refund their reservation.
  7. Wake rechecks Paused, Archived, missing, and terminal state before execution.
  8. Plan Approval remains durable independently of transient Wake delivery.
  9. Completion requires the Coordinator to observe the latest work revision.
  10. Historical Group Chat remains available through keyset pagination after reload.
  11. Task and Plan UI state comes from persisted structured outcomes, not attempted tool arguments.
  12. A queued user message does not suppress recovery before actual dispatch.
  13. One malformed or failing Run cannot stop recovery for unrelated Runs.

Verification

Gate Result
Agent Core application suite ✅ 3,028 / 3,028 passed
session_persistence suite ✅ 29 / 29 passed
Vitest ✅ 5,318 / 5,318 passed
TypeScript typecheck ✅ Passed
ESLint ✅ Passed
Circular-dependency check ✅ Passed
Changed-scope Rust formatting ✅ All 80 changed Rust files passed rustfmt --check
Isolated real Debug App Agent Org HTTP E2E ✅ 46 / 46 passed through production behavior paths
Rendered Group Chat WebDriver ⚠️ 5 / 6 passed; the remaining mention-menu failure reproduces identically on clean develop
Rendered Pause/Resume WebDriver ✅ 8 / 8 passed
Rendered Recovery WebDriver ✅ 2 / 2 passed
Husky ✅ Commit hooks and commitlint passed

Clean-develop baselines

  • Strict workspace Clippy remains blocked by seven orgtrack_core diagnostics reproduced on clean develop.
  • agent_core --no-deps reports the same 45 clean-develop diagnostics.
  • e2e-test --no-deps reports three diagnostics on this branch versus four on clean develop; this change introduces no additional diagnostic.
  • Twenty-three specialization::external_import tests are filtered because the same nested non-reentrant lock_home deadlock reproduces on clean develop.
  • The remaining rendered mention-menu scenario fails on this branch and clean develop with the same assertion. It is not counted as passing and is not attributed to this follow-up.

Scope boundary

This PR contains only:

  • findings explicitly raised in the first review;
  • findings explicitly raised in the second review;
  • minimum integration dependencies required for those fixes;
  • directly corresponding tests;
  • English backend and frontend audit reports.

It intentionally does not include the later red-team expansion, the full Revision Event architecture, or post-completion Follow-up Runs.


Follow-up: post-completion conversation and Follow-up Runs

A separate feature PR will address the distinction between a long-lived Group Chat and an individual Agent Org Run.

The planned behavior is:

  • The Root Coordinator Session remains the long-lived Group Chat.
  • One Group Chat may contain multiple sequential Runs.
  • Completed continues to mean that one round of work is finished; it does not permanently close the Group Chat.
  • After completion, ordinary questions are answered by the Coordinator without reopening the old Run, writing its Inbox, waking old Workers, or mutating completed Tasks.
  • Requests that require more work produce a structured Follow-up proposal.
  • The user confirms before a new team Run starts, preventing accidental Token use.
  • A Follow-up Run records continued_from_run_id, creates fresh Worker Sessions, Tasks, Inbox state, and Recovery Budget, and never changes the old terminal Run.
  • Coordinator and Worker Turns receive explicit Run scope so an old Worker cannot resolve to a newer Run through the shared Root Session.
  • Follow-up creation is atomic and idempotent; repeated clicks create only one Run.
  • The UI will show a completed banner, Follow-up proposal card, Run timeline, active Run board, and historical Run selector.

This will be implemented separately, rather than expanding this review follow-up.


Follow-up: Revision Event architecture

A separate architecture series will replace high-frequency full Snapshot polling with ordered incremental synchronization.

The planned model is:

  1. Canonical Run, Task, Inbox, Session, and Approval tables remain the source of truth.
  2. Every durable mutation appends one small sequential Revision Event in the same SQLite transaction.
  3. The frontend loads a compact Snapshot and then applies Events after that Snapshot revision.
  4. Missing revisions are detected and replayed in order.
  5. Duplicate delivery is harmless because reducers are idempotent.
  6. Large Plan, TaskOutput, and message content remains in detail APIs rather than Event payloads.
  7. Backpressure requests resynchronization instead of buffering without bounds.
  8. Low-frequency Snapshot verification remains until the Event path proves stable.

Expected user-visible benefits:

  • faster Task and approval updates;
  • fewer temporary disagreements between Group Chat, member views, and Kanban;
  • exact reconnect and restart recovery;
  • less repeated transfer and parsing of unchanged data.

This work will land through separate Event-contract, replay/transport, frontend-store, and polling-reduction PRs.


Follow-up: red-team hardening

Additional red-team findings were intentionally preserved outside this PR and will be reviewed in a separate dependent branch and PR.

The follow-up scope includes:

  • explicit Superseded/Cancelled disposition and repair tools for permanently undeliverable or orphaned Inbox rows;
  • additional Analyzer/Executor time-of-check/time-of-use validation;
  • stronger recovery-reason fingerprints and atomic Coordinator-notice accounting;
  • legacy Inbox deduplication when multiple members share one AgentDefinition;
  • repair and fail-closed handling for corrupt, cyclic, oversized, or ambiguous historical state;
  • external Plan path, symlink, and startup-artifact validation;
  • explicit org_run_id ownership for nested Turn Intents and multi-Run session chains;
  • additional Run View cache eviction, tombstone, bootstrap timeout, and detail-cache limits;
  • historical member-navigation, WebKit mention, and option-deduplication hardening;
  • additional Task, graph, artifact, fan-in, fan-out, and per-Run resource boundaries;
  • expanded production return-to-work and recovery E2E coverage.

These changes are not required to resolve the two #373 review rounds and are deliberately excluded here to keep this PR reviewable.


Audit reports

  • docs/architecture-audit-2026-07-16/AgentOrgReviewSafetyAudit.md
  • docs/frontend-ui-audit-2026-07-16/AgentOrgReviewSafetyAudit.md

Make Agent Org reads side-effect free and bounded so polling no longer
takes broad writer locks or repeatedly expands large Task, Inbox, and
Plan payloads. Preserve complete Group Chat history through keyset
pagination while loading full Task and Plan results only when requested.

Unify task graph, finality, wake-budget, Inbox materialization, and Plan
approval transitions around transactional persisted state. Prevent
duplicate delivery, stale dependency notifications, premature Run
abandonment, approval loss across pause or restart, and one broken Run
from blocking recovery for other Runs.

Align Rust and TypeScript outcomes for Task Graph, TaskOutput, Kanban,
Plan approval, pause and resume, and direct user intervention. Add
production-parity fixtures, focused regressions, rendered E2E coverage,
and architecture and frontend audit reports for both review rounds.

Verification:
- Agent Core application suite: 3,028 / 3,028 passed
- session_persistence suite: 29 / 29 passed
- Vitest suite: 5,318 / 5,318 passed
- TypeScript typecheck, ESLint, and circular-dependency checks passed
- rustfmt --check passed for all 80 changed Rust files
- Isolated Debug App Agent Org HTTP E2E: 46 / 46 passed
- Rendered Group Chat WebDriver: 5 / 6 passed; the remaining failure reproduces on clean develop
- Rendered Pause/Resume WebDriver: 8 / 8 passed
- Rendered Recovery WebDriver: 2 / 2 passed
- Husky staged-file TypeScript and Rust checks passed

Pre-commit hook ran. Total eslint: 0, total circular: 0
Bring the latest develop changes into PR #419 while preserving the recovery invariants introduced by the branch. Keep run-view reads side-effect free and retain Group Chat history pagination through the extracted history surface.

Verification:
- pnpm run typecheck
- pnpm run lint
- pnpm run check:circular
- cargo check -p e2e-test
- pnpm exec vitest run src/engines/ChatPanel/hooks/useAgentOrgGroupChatHistory.test.ts
- Focused Agent Org lifecycle tests passed

Pre-commit hook ran. Total eslint: 0, total circular: 0
@Chloe-JY
Chloe-JY merged commit e008615 into develop Jul 18, 2026
1 of 2 checks passed
Neonforge98 pushed a commit that referenced this pull request Jul 30, 2026
Bring the latest develop changes into PR #419 while preserving the recovery invariants introduced by the branch. Keep run-view reads side-effect free and retain Group Chat history pagination through the extracted history surface.

Verification:
- pnpm run typecheck
- pnpm run lint
- pnpm run check:circular
- cargo check -p e2e-test
- pnpm exec vitest run src/engines/ChatPanel/hooks/useAgentOrgGroupChatHistory.test.ts
- Focused Agent Org lifecycle tests passed

Pre-commit hook ran. Total eslint: 0, total circular: 0
Neonforge98 pushed a commit that referenced this pull request Jul 30, 2026
…invariants

fix(agent-org): address PR review correctness and performance findings
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants