fix(agent-invocations): End stranded agent invocations from the heartbeat - #2640

Follow-up to #2627. An independent review of #2627 found that it still left agent invocations open on other stop paths.

Problem

• fix(agent-invocations): Tell the parent when a child's work stops after its tries run out #2627 ends a child's open agent invocations from a hook inside the heartbeat's lease-expiry stop. Two other paths also stop work and drop the mailbox, and neither runs the hook:
  • recordConversationRetry, when an error escapes the worker's last try.
  • recordAttemptFailure, when it dead-letters messages.
    On those paths, the agent invocations stay open and the parent never gets a result.
• The hook fails agent invocations before the stop is written. If the 10s mutation lock expires or the heartbeat dies before that write, the work can start again, even though its agent invocations already failed.

Change

• Remove the hook. task-execution goes back to its shape before fix(agent-invocations): Tell the parent when a child's work stops after its tries run out #2627.
• Add a heartbeat phase, recoverStrandedAgentInvocations. It ends an open, appended agent invocation only when all of these are true:
  • its input left the child mailbox,
  • the child has no lease,
  • the child's status is failed or idle.
    Consumed ids stay in inboundMessageIds, so that input can never be appended again. Because the state cannot run again, the phase needs no lock, and it covers every stop path.
• The phase runs after mailbox-append recovery and before result recovery. The parent gets the result in the same heartbeat.

Tests
agent-invocation-worker-crash.test.ts:

• The killed-worker case still passes. It now also checks that the agent invocation stays running after each try except the last.
• New case: an error escapes the worker's last try (recordConversationRetry stop) while a follow-up waits in the mailbox. Both agent invocations fail, and the parent gets 2 results. This case fails on main.

Checks

• tsc, lint, test-architecture and file-length pass.
• The agent-invocation, agent-dispatch, task-execution and heartbeat suites pass, except for local timeouts that also happen on main (durable-queue 4, agent-dispatch-work 1).

via @dcramer.

--

View Session [Sentry]