fix(service-queue): reclaim a running job whose lease ran out, and renew a live handler's lease - #30277

Requested by Bill · project thread

domain:server#5 · os-bill · session_01Utp6CYVV6fcAQrNTDQRvPb

维护者速读

• 改了什么：数据库队列在处理器还没跑完时进程挂了（崩溃、OOM、重新部署），那条任务现在会在租约到期后被下一次轮询收回：记一次失败尝试，按退避重排，次数用尽进死信；活着的慢处理器会定期续租，不会被别的节点抢走重跑。
• 为什么改：leaseMs 文档写着"到期后可由其他 worker 回收"，但认领只读 pending，挂掉的任务永远停在 running，既不重试也不进死信，运维看不到。
• 风险与回滚：多节点（未开 soleWriter）时每个租约窗口（默认 30 秒）多一条带索引的读，空闲一小时从 125 条变 246 条；单写者（soleWriter，云端租户内核）只在开机多读一次，空闲一小时不再多读。处理器若同步阻塞事件循环超过大半个租约，会被当成死掉而重跑。回滚即撤本 PR，无数据迁移。
• 先看：packages/services/service-queue/src/db-queue-adapter.ts 的 reclaimRow（CAS 谓词）、renewLease（续租）、reclaimExpired（读的节流界）与 nextDueAt（并入最早活租约）。
• 两分钟验证：pnpm --filter @objectstack/service-queue exec vitest run src/db-queue-idle-backoff.test.ts -t 30228，预期 9 条全绿；其中"CRASHED-WORKER FIXTURE"一条在修复前的代码上是红的。
• 需要你做的：无，席位落地。

Fixes #30228

Clause-②: no — restores the documented leaseMs reclaim; no accept set or export widens
Metadata-protocol: none

What changes

The card's acceptance, quoted:

1. A running row whose locked_until has passed is claimable again (or moved to the retry path with its attempt counted), within one idle tick of the lease running out.
2. A live worker that is still inside its lease is never double-claimed.
3. Pins: a crashed-worker fixture (row left running with an expired lease) is reclaimed; a row inside its lease is not. The nextDueAt read core DispatchLoop: an idle loop re-runs its fast backoff ramp after every boot, so a fresh tenant kernel spends 30 statements in its first minute on empty delivery outboxes and job queues #29919 adds (PR fix(core): an idle DispatchLoop wakes at the next due time, and a sole writer skips the cold-start ramp #30225) also counts the earliest lease expiry, so a sole-writer loop wakes for it.

Reclaim (item 1). pollOnce() now starts with reclaimExpired(): one read of the table's status = 'running' rows ordered by locked_until, every queue's (reclaiming runs nothing, so a row of a queue this node does not subscribe to just goes back to pending for whoever does; the predicate is the claim read's plain status equality, served by the declared status index, with no operator a stand-in engine might not answer), and each row whose lease has passed (or that has no lease at all) goes to the retry path. Of the card's two options this takes the second, retry path with the attempt counted, because attempts is written only after a handler settles: a worker that died left the row one attempt short, so reclaiming without counting would retry forever a message that kills its worker every time (an OOM on a large payload). The reclaim writes that attempt down and takes the branch a thrown handler takes: pending on its backoff, or dlq once max_attempts is spent, with last_error naming the dead worker. A reclaimed row counts in the tick's result, so the loop snaps back to pollIntervalMs and claims it on its backoff rather than after the idle ceiling.

The reclaim write is a compare-and-set (item 2). Its predicate re-states the expiry: id, status: 'running', the locked_by it read, and locked_until null or at most now. A holder that renewed between the read and the write matches nothing and keeps its run. It is a multi: true write on purpose: the by-id path binds only the primary key and drops the rest of the predicate (the outbox-dispatcher-scope.ts precedent). Two workers reclaiming the same row write the same values from the same read and neither runs it, so that race costs nothing. sys_job_queue carries no tenant column (ADR-0131 D7), so the predicate writes need no tenant-audit warrant.

A live handler renews its lease (item 2). This is the risk the reclaim itself introduces. Before, nothing reclaimed, so a handler slower than leaseMs was safe; with a reclaim and no renewal it would be reclaimed mid-run and run twice. dispatch() now moves its row's locked_until a full window ahead every third of leaseMs while the handler runs: a multi: true compare-and-set on locked_by = workerId, logged when it matches nothing (the lease was lost underneath). A lease therefore passes only when its holder stopped renewing: the process died, or its event loop was blocked for most of a window. That is how a dead worker is told apart from a slow one, and the leaseMs doc now says the one thing an author must know: keep synchronous stretches well under the lease.

The read is bounded, not per tick. The read also returns the earliest lease still running. No lease can run out before the earlier of that lease and now + leaseMs (every running row was in the read, a row claimed after it holds at least leaseMs from its claim, and renewal only moves a lease later), so the next read waits until then. A full page means more may be hidden, so the next tick reads again. Cost without soleWriter: a busy 1 s poll pays one reclaim read per lease window; an idle loop at the 30 s ceiling pays one per tick, and an idle hour at the defaults goes from 125 to 246 reads (pinned).

A sole writer reads once per boot. With soleWriter no other process claims rows, so the window term is dropped. The first read after start finds what a crashed previous process left behind; after it, the only leases that can run out are the live ones it saw and this process's own claims whose settling write (completed, retry, dead letter, release) failed and left the row running. The adapter keeps those unsettled claims' leases (renewal moves them, a settling write drops them, the next read replaces them all), reads again only when one runs out, and folds them into nextDueAt. An idle sole writer's hour carries one reclaim read, the boot's, against 120 for the same hour without the flag (pinned); its fresh first minute goes from 2 to 3 reads.

nextDueAt counts the earliest live lease (item 3). Built on #30225's DbQueueAdapter.nextDueAt after it merged (771b2db). The reclaim read already returns the earliest lease still running; the adapter keeps it, and nextDueAt answers the earliest of that, a sole writer's unsettled claims and the earliest deferred scheduled_for, at no read of its own. That read ran on the empty tick the answer follows, or the bound it set has not passed. A soleWriter loop, which otherwise sleeps to its ceiling, now wakes when a row its own previous process left running across a restart runs out of lease. The window term (now + leaseMs) is deliberately not in the answer: it only bounds the next read, and folding it in would wake a sole-writer loop every lease window for nothing.

Docs. Read sentence by sentence, none made false by this PR, left unchanged: content/docs/plugins/packages.mdx:287 ("multi-node claim via a lease on the db adapter", note 6648413565) is now true as written; content/docs/automation/webhooks.mdx, the dispatchSoleWriter / db.soleWriter paragraph and the idle-cost sentences, describe the HTTP and notification dispatchers' reads and the flag's meaning, both still exact (the queue's sole-writer loop still sleeps to its next due time or the ceiling; a lease is one more due time); content/docs/kernel/runtime-services/audit-service.mdx:71's locked_until is the account lock's field on an auth audit event, not the queue's; the package README's leaseMs line is now true.

Tests

packages/services/service-queue/src/db-queue-idle-backoff.test.ts, new #30228 block, on that file's counting engine (its update already opens with assertEngineUpdateDispatch; it now applies a multi dispatch to the rows its where matches and reports the count):

• CRASHED-WORKER FIXTURE: a row written running, locked_by: 'worker_dead', lease 1 ms past: one tick returns it to pending, attempts: 1, scheduled_for = now + 1 s, last_error naming the worker; on its backoff it runs as attempt 2 and completes.
• A lease that runs out while the loop is deep in its idle backoff (another node's crash, nothing tells this process) is reclaimed within one idle interval of expiry and run.
• The lost run counts: a fixture at attempts: 2 of 3 goes to dlq, and listFailed returns it.
• A row inside its lease is not touched (no write at all).
• CAS: a renewal landing between the reclaim read and its write leaves the row running under its holder.
• A live handler three leases long, with a second adapter polling the same table: one run, completed, attempts: 1.
• In core DispatchLoop: an idle loop re-runs its fast backoff ramp after every boot, so a fresh tenant kernel spends 30 statements in its first minute on empty delivery outboxes and job queues #29919's block: a sole-writer adapter booting over a row its previous process left running, lease 40 s from expiry, ceiling 5 min: the row runs at exactly 41 s (reclaimed at the lease's expiry, then its 1 s backoff).
• A sole writer's idle hour, with one job run at 10 min, makes exactly 1 reclaim read; the same hour without soleWriter makes 120 (the engine counts the reclaim read on its own).
• A sole writer whose completed write fails once (the row stays running under itself) reclaims it when its 3 s lease runs out and runs it again: 2 runs, completed, attempts: 2, 2 reclaim reads (boot, then the expired lease).

Moved pins in the same file, each with its reason in the comment: the idle hour (125 → 125 + 121), the flat-poll negative control (3602 → 3602 + 121), the per-tick cost (n claim reads + 1 reclaim read, then n on a tick inside the same window), the sole-writer fresh minute (2 → 3) and its ramping control (6 → 6 + 2).

before  × CRASHED-WORKER FIXTURE: an expired lease goes back to the retry path …   AssertionError: expected +0 to be 1
        × a lease that runs out while the loop is deep in its idle backoff …       row still { status: 'running', attempts: 0 }
        × the lost run counts: … dead-lettered, not retried forever                AssertionError: expected +0 to be 1
        ✓ a row INSIDE its lease is never reclaimed                                (trivially: nothing reclaimed anything)
        ✓ the reclaim is a compare-and-set …                                       (trivially)
        ✓ a LIVE handler three leases long is never double-claimed …               (trivially)
after   ✓ all of them, and the whole package: Test Files 5 passed (5), Tests 91 passed (91)

Acceptance notes

• The pending → running claim is still the by-id write it was (the comment above it says so: concurrent claimers may both take a pending row, and delivery is at-least-once). This PR does not change that; it adds no new way to double-claim, and the reclaim never runs anything itself.
• A worker whose lease was reclaimed while it was stalled and then finishes still writes its terminal state by id. The renewal logs the lost lease; making the terminal writes a compare-and-set on locked_by too would close that last window and is a separate change.
• Deviation: census +2 write sites (reclaimRow, renewLease), surface widened by seat 03:38Z. check:tenant-audit-census counts every engine write call site, so content/docs/permissions/tenant-audit-census.mdx and docs/audits/2026-08-tenant-audit-write-call-sites.counts.md are regenerated (node scripts/tenant-audit-census.mjs --write) and the page's hand-written figures moved with them (259 → 261 sites, 182 → 184 decidable, 107 → 109 undecidable, 66 → 68 const-named, 56% → 55%).
• Leases assume every node runs the same leaseMs, as they do when the nodes share one QueueServicePlugin configuration. A node with a shorter lease than this one's is still reclaimed, at most one lease window late.

Verification

All readings on head 8d430e0c (code identical to 87fd3a2e; the last commit edits only the changeset).

• pnpm --filter @objectstack/service-queue exec vitest run --maxWorkers=2: Test Files 5 passed (5), Tests 91 passed (91).
• pnpm --filter @objectstack/plugin-email exec vitest run --maxWorkers=2 (after building its closure): Test Files 48 passed (48), Tests 785 passed (785). On 9a2d7712 its suites were red in CI (Test Core 7/8, "fakeEngine: unsupported operator $in"): the reclaim read then filtered queue: { $in: … }, which that package's stand-in engine refuses; the read now carries only status equality.
• pnpm --filter @objectstack/service-queue typecheck (tsc --noEmit, whose program lists db-queue-idle-backoff.test.ts): exit 0.
• eslint --no-inline-config on the two changed source files: exit 0. pnpm lint: exit 0.

Generated by Claude Code