Skip to content

perf(service-messaging): an idle dispatcher tick reads the database twice, not 18 and 10 times - #28403

Merged
objectstack-fleet[bot] merged 5 commits into
mainfrom
claude/project-thread-9dajyu
Oct 31, 2026
Merged

objectstack-fleet[bot] merged 5 commits into
mainfrom
claude/project-thread-9dajyu

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Requested by Sam · project thread
Fixes #28383

Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:通知调度器和 HTTP 调度器在 outbox 空闲时,每次轮询只读两次库(回收过期 claim 前先读一条、再用一条跨所有分区的"有没有到期行"读决定要不要逐分区探测),不再是 18 条和 10 条。
  • 为什么改:cloud 实测一个空闲租户内核每分钟读库 56 次,全部是空转;这些读计入 Turso 的行读配额。
  • 风险与回滚:有活时仍走原来的逐分区 claim/digest 路径,只多一次读;回收 UPDATE 的谓词不变,只在读到过期行时才执行。outbox 接口新增的 hasDueWork 是可选方法,自定义 outbox 不实现就保持旧行为。回滚即 revert 本 PR。
  • 先看:packages/services/service-messaging/src/dispatcher.ts(runTick 里的门)、src/sql-outbox.ts(reapExpired 与 hasDueWork)、src/dispatcher-idle-cost.integration.test.ts(语句数钉子)。
  • 两分钟验证:pnpm --filter @objectstack/service-messaging exec vitest run --maxWorkers=2 src/dispatcher-idle-cost.integration.test.ts src/http-dispatcher-idle-cost.integration.test.ts,预期 13 个用例全绿,空闲 tick 钉在 2 条 SQL。
  • 需要你做的:无,席位落地。

What changes

An idle tick of each delivery dispatcher now costs two SQL statements, whatever partitionCount is. Measured at the client (knex query events) on a harness with one global afterUpdate hook, the kernel shape cloud measured:

Dispatcher main ac09e7b85 this PR
NotificationDispatcher 18 2
HttpDispatcher 10 2

The card's bar is ≤ 6 per dispatcher.

  • Candidate 1: the reap writes only when there is something to reap. reapExpired in sql-outbox.ts and sql-http-outbox.ts first reads one expired in_flight id (fields: ['id'], limit: 1). If it finds none, it returns. If it finds one, it runs the predicate UPDATE, which is unchanged, so every claim expired by then still goes back to pending, not only the one the read saw. On a kernel the old unconditional UPDATE was two statements (the engine reads the matched rows for the per-row afterUpdate dispatch, then writes); it is now one read.
  • Candidates 2 and 3: one due-work read gates every partition probe, digest probes included. INotificationOutbox and IHttpOutbox gain an optional hasDueWork({ now }). It asks, in one SELECT id … LIMIT 1 across every partition and organization, whether any pending row is due. The predicate is the claim's own with the partition term dropped, and for notifications with the digest_key arm dropped, so it covers immediate and batched rows alike. runTick asks it after the reap. If nothing is due, the tick ends there: no partition lock is taken and no claim or digest probe runs. If something is due, the tick runs the full per-partition pass exactly as before, for one extra read. Both built-in stores (SQL and memory) implement it. A store without it keeps probing every partition. DueWorkOptions and HttpDueWorkOptions are exported.
  • partitionCount and the stored partition_key are unchanged.

Why

An idle hosted tenant kernel read its database 56 times a minute with nothing queued: 2 ticks × (18 + 10) at the framework's 30 s idle ceiling, plus a 140-statement backoff ramp in the first minute. Those reads count against Turso's rows-read quota, and cloud#2177 once had a staging account suspended for reads. With this PR an idle minute costs 8 statements at the 30 s ceiling.

Tests

  • dispatcher-idle-cost.integration.test.ts and http-dispatcher-idle-cost.integration.test.ts now count SQL statements on the delivery table at the client as well as engine calls, with a global no-op afterUpdate hook registered so a predicate UPDATE is the read-then-write pair it is on a kernel. On this harness, main measures 18 and 10 statements per tick, the same figures cloud measured.
    • The idle pin: { update: 0, find: 2 × TICKS } engine calls, and exactly 2 × TICKS SQL statements, all SELECT. This replaces the old ≤ TICKS × (1 + 2 × partitionCount) and ≤ TICKS × (1 + partitionCount) ceilings.
    • New: a row deferred into the future (notification notBefore; an HTTP retry acked with nextRetryAt) keeps the tick at two reads and goes out on the first tick after it comes due.
    • New (notification): a due digest-only outbox is found by the due-work read and collapsed on that tick.
    • The crash-recovery legs now also assert the reap's UPDATE ran exactly once when a claim had expired. This guards against a reap that never writes.
    • The legacy-outbox legs (no reap, and so no hasDueWork) are unchanged and still see 2 × partitionCount and partitionCount claims a tick.
  • Pins that assumed the old shape, reconciled:
    • sql-outbox-audit-columns.test.ts: its fake engine's scripted find queue now opens with a reap read that finds a stale row, so the reap UPDATE payload it checks for updated_at is still produced.
    • system-context.pin.test.ts: the population counts include the reap reads and one hasDueWork call, so the new reads are also held to the explicit system opt-in.
    • http-dispatcher-idle-backoff.test.ts: it timestamped a tick at claim(), which an idle tick no longer calls. It now timestamps at reap(), which every tick calls once first, as the notification side's file already did.

Acceptance notes

  • Deviation from the dispatch's mechanism assumption (candidate 2). The dispatch described "one probe across the partitions this process holds". The due-work read here spans all partitions and takes no lock, and it gates the per-partition pass instead of replacing it. Lock state is unknown until each acquire, and on the common single-node or memory lock every partition is held anyway. A read that moves nothing needs no lock, as with the reap. Keeping the per-partition claims under their locks leaves the busy path, the batch size per partition and the digest-under-lock rule exactly as they were. On a multi-node cluster, a node whose partitions are all held elsewhere pays the same lock attempts as before whenever something is due anywhere, and two reads when nothing is.
  • Candidate 3 is folded into the one read, which is the card's first option ("fold into the claim probe"). A busy tick still runs both probes in every held partition. Narrowing the busy path is not what this card measured.
  • Out of scope, as dispatched:
    • Candidate 4: wake at the next due time instead of a blind ceiling (packages/core/src/dispatch-loop.ts).
    • Candidate 5: a persisted watermark for the objectql lifecycle sweep's first run 60 s after every boot.
    • The backoff ramp's first-minute price now falls with the per-tick price (5 ramp ticks × 4 statements instead of × 28). cloud re-measures under cloud#3413.
  • The new reads add ORDER BY id (the engine's default when a limit is set) over a limit 1. That is one row from the primary key ordering, not a scan cost this card measured.

Verification

All readings at head 95f4da30a unless stated.

Statement counts (before / after)

A throwaway measurement on main ac09e7b85 (not committed), with and without a global afterUpdate hook, one tick each over an empty outbox:

  • no hook: notification 17, HTTP 9; with the hook: notification 18, HTTP 10. The hook shape is select * … where status = ? and claimed_at < ? then the update, followed by one or two select id … partition_key = ? per partition.
  • After the change, both shapes: notification 2, HTTP 2. Each is select id … where status = ? and claimed_at < ? … limit ?, then select id … where status = ? and ((next_… is null) or (next_… <= ?)) … limit ?.
Package tests and typecheck
  • pnpm --filter @objectstack/service-messaging test: Test Files 62 passed (62), Tests 759 passed (759). Lock VERDICT command-exit 0 at 18e1c546b. The later commit only adds the changeset and one as HttpClaimCredential cast in a test.
  • pnpm --filter @objectstack/service-messaging typecheck: exit 0 at 95f4da30a. At 18e1c546b the first run caught one TS2345 in the new HTTP retry leg (an HttpDelivery passed as the claim credential), fixed with the cast the dispatcher itself uses.
  • The two idle-cost files at 95f4da30a: 13 passed (13).
Ablation (scripts/ablation-replace.mjs, restored, git diff HEAD empty, marker grep 0)

Tests import src/ relatively, so no dist rebuild sits between mutation and run.

  • A: the hasDueWork gate in NotificationDispatcher.runTick replaced by a marker. Notification idle-cost file: 2 failed / 5 passed (the idle pin: find: 170 vs 20, 17 statements vs 2; the deferred-row pin).
  • B: SqlHttpOutbox.reapExpired's early return replaced by a marker. HTTP idle-cost file: 2 failed / 4 passed (the idle pin: update: 10 vs 0; the retry pin: 4 statements vs 2).
Dispatch gates and lint
  • node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack: 62 families derived from 15 paths vs merge base ac09e7b85 (473 changed lines, under the 3000 threshold). Every command was run with stdin closed, and each exit code was captured to disk before any pipe.
  • --ran reconciliation: 62 derived, 62 run, 0 NOT-MEASURED, 0 UNRUN; it exits 0.
  • Every derived command exited 0. Three first exited 3 (PREREQUISITE NOT MET) and were re-run after their prerequisite:
    • pnpm check:dual-build-cjs-loads and pnpm check:i18n after a full pnpm build (lock VERDICT command-exit 0).
    • node scripts/check-plugin-teardown-shape.mjs --self-test after git fetch --unshallow origin main (48 cases pass).
  • pnpm lint (repo-wide eslint . --no-inline-config): lock VERDICT command-exit 0.

Human Minimum

  • 1. Schemas start as Zod; packages/spec holds no runtime logic. (Not touched.)
  • 2. Every name I added uses the casing for its kind.
  • 3. No formatter ran; pnpm lint passes. (See Verification.)
  • 4. One issue, one branch off an up-to-date main, one pull request.
  • 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset.
  • 6. packages/spec touched: n/a, not touched.
  • 7. No hand edits to content/docs/references/, packages/*/CHANGELOG.md or content/docs/releases/.
  • 8. Title, body and comments are in English.
  • The Before Submitting list in CONTRIBUTING.md is done.

Generated by Claude Code

…of probing every partition

The reap reads before it writes, and one environment-wide due-work read
gates the per-partition claim and digest probes.

Claude-Session: https://claude.ai/code/session_01QUs3Bvkfh86Jgvi7gewhBD
Co-authored-by: Claude <noreply@anthropic.com>
…ments

Counts at the client (knex query events) with a global afterUpdate hook,
the kernel shape cloud measured 18 and 10 at.

Claude-Session: https://claude.ai/code/session_01QUs3Bvkfh86Jgvi7gewhBD
Co-authored-by: Claude <noreply@anthropic.com>
…onal reap or a per-partition probe

Claude-Session: https://claude.ai/code/session_01QUs3Bvkfh86Jgvi7gewhBD
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/m documentation Improvements or additions to documentation tests tooling Fix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface labels Oct 31, 2026
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/service-messaging, touching 12 documentable anchor(s). ⚠️ 1 changed file(s) yielded no anchor (packages/services/service-messaging/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

1 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/automation/webhooks.mdx (via in_flight (literal, a string literal in reapExpired))

⛔ 2 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17/17-2.mdx (via IHttpOutbox (symbol, a top-level interface), SqlHttpOutbox (symbol, a top-level class))
  • content/docs/releases/v17/17-3.mdx (via INotificationOutbox (symbol, a top-level interface))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/services/service-messaging/src/index.ts) — pages documenting those are invisible to this run
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 60 of 212 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 6 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 63a9bcb773045ccffe336d0ee4876c748f5adbfb → packageMentionDocs.

Which tree this was computed on

This run read content/docs from dbc31404398cbcc414248981e4a70877b8331e72 — the merge of head 570a250c1096c8565086de20b28b0792de780773 into base 63a9bcb773045ccffe336d0ee4876c748f5adbfb, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin dbc31404398cbcc414248981e4a70877b8331e72 && git checkout dbc31404398cbcc414248981e4a70877b8331e72
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 63a9bcb773045ccffe336d0ee4876c748f5adbfb 570a250c1096c8565086de20b28b0792de780773 && git checkout -B drift-repro 63a9bcb773045ccffe336d0ee4876c748f5adbfb && git merge --no-ff 570a250c1096c8565086de20b28b0792de780773

node scripts/docs-audit/affected-docs.mjs --json 63a9bcb773045ccffe336d0ee4876c748f5adbfb

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 63a9bcb773045ccffe336d0ee4876c748f5adbfb → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

domain:server#4 · os-zhuang · session_01QUs3BtyH9zgxKzYrJiH18J

Contract review

Served-tier: TIER_DEFAULT
Head-sha: 570a250c1096c8565086de20b28b0792de780773
Local-runs: none

Face: the declaration face alone (Clause-②: yes (widening) in the changeset at this head). There is no packages/spec/src/** path and no governed path, so per contract-review.md the record is the lane seat's at TIER_DEFAULT. This corrects the claim amendment 6543085330, which named CONTRACT_REVIEW_TIER by analogy to #28281's triage answer; the rule sets the tier by face, and this face maps to TIER_DEFAULT. Inputs: card #28383 and claim 6542566200 plus amendment, the os-dev-report 6543067755, the PR's 15 files read whole at 95f4da30a, and the one-file delta to this head (the changeset only).

① Derived judgments

  • INotificationOutbox.hasDueWork? and IHttpOutbox.hasDueWork? (new optional members on two exported interfaces): right. They are optional, so every existing outbox implementation compiles and behaves as before; the dispatchers fall back to probing every partition when a store omits them (outbox.hasDueWork && …). Both built-in stores (SQL and memory, each family) implement them.
  • The predicate cannot hide due work: right. The SQL hasDueWork is status: 'pending' with next_attempt_at / next_retry_at null or ≤ now, which is claim's and claimDigest's predicate with the partition term and the digest_key arms removed: a superset. So false implies every partition's probe would have found nothing. The memory stores mirror it. Each dispatcher passes the same clock its claims use: the notification side passes neither, and both default to the store clock; the HTTP side passes this.opts.now?.() to both.
  • DueWorkOptions and HttpDueWorkOptions (two new exported types, { now?: number }): right. Additive, no existing export changes.
  • Reap read-before-write: not a surface change. The predicate UPDATE is unchanged and runs only when a one-row id read finds an expired claim. Expiry timing is the once-per-tick reap window service-messaging: NotificationDispatcher issues 32 statements per tick on an EMPTY outbox (reap runs per partition, twice) — idle cost scales with partitions × table size and never backs off #17610 already had.
  • Nothing else moves: no route, error code, stored column, metadata key or packages/spec path. Metadata-protocol: none holds.

② Semver level

  • @objectstack/service-messaging patch: accepted. A non-protocol widening ships patch under AGENTS.md's changeset rule. The changeset at this head names the widening in one sentence.

③ Boundary flags

  • PR body line Clause-②: no: stale. The changeset at this head and the claim amendment are the declaration of record. The relay cannot patch the body (its first line is the attribution, not a signature), and the dev's contract leaves the body to the seat; the ACCEPT records the correction.
  • Out of scope, as dispatched: candidates 4 (wake at the next due time) and 5 (sweep watermark), and cloud's re-measure under cloud#3413, are listed in the report. Not owed here.

Implemented-by: claude/project-thread-9dajyu
Reviewed-by: session_01QUs3BtyH9zgxKzYrJiH18J

VERDICT: PASS

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

domain:server#4 · os-zhuang · session_01QUs3BtyH9zgxKzYrJiH18J

ACCEPT · PR #28403 · Head-sha 570a250c1096c8565086de20b28b0792de780773 · 2026-10-31T12:19Z

维护者速读

  • 改了什么: 通知调度器和 HTTP 调度器在 outbox 空闲时,每次轮询只读两次库:回收过期 claim 前先读一条,再用一条跨所有分区的"有没有到期行"读决定要不要逐分区探测。以前是 18 条和 10 条。
  • 为什么改: cloud 实测一个空闲租户内核每分钟读库 56 次,全是空转,这些读计入 Turso 的行读配额。
  • 风险与回滚: 有活时仍走原来的逐分区 claim/digest 路径,只多一次读;回收 UPDATE 的谓词不变,只在读到过期行时才执行。outbox 接口新增可选方法 hasDueWork,另导出两个选项类型(公开面加宽,Clause-② yes);自定义 outbox 不实现它就保持旧行为。回滚即 revert 本 PR。
  • 先看: packages/services/service-messaging/src/dispatcher.ts(runTick 里的门);src/sql-outbox.ts(reapExpired 与 hasDueWork)。
  • 两分钟验证: pnpm --filter @objectstack/service-messaging exec vitest run --maxWorkers=2 src/dispatcher-idle-cost.integration.test.ts src/http-dispatcher-idle-cost.integration.test.ts,空闲 tick 钉在 2 条 SQL。
  • 需要你做的: 无,席位落地。

席位意见:空闲开销从 28 条降到 4 条,hasDueWork 的谓词是 claim 谓词的超集,不会漏掉到期行;可合。

席位核对

  • Form: 15 files, +407/−68. check-governed-merges.mjs --pr 28403: NOT governed, 475 lines ≤ 3000, Tier S. Declaration of record: Clause-②: yes (widening), carried by the changeset at this head and claim amendment 6543085330. The PR body's line 5 still reads Clause-②: no; the relay cannot patch the body, and this record corrects it. Metadata-protocol: none.
  • Premise measured: the dev re-measured on main ac09e7b85 at the SQL client with a global afterUpdate hook: 18 (notification) and 10 (HTTP) statements per idle tick, matching cloud's figures; after the change, 2 and 2, against the card's bar of 6. The idle-cost integration tests pin exactly 2 statements and zero reap UPDATEs; two ablations turn the pins red. Contract review at TIER_DEFAULT (declaration face): 6543106545, same head, PASS.
  • Gates (check-runs at the pinned head): 48 runs: 35 success, 13 skipped, 0 failed, 0 pending. The Lint & Repo Gates and TypeScript Type Check reds on 95f4da30a were lanes cancelled when the new head was pushed, not failures; this head passes both.
  • Accepted deviations: the due-work read spans all partitions and takes no lock, and it gates the per-partition pass instead of replacing it. Busy-path batching and digest-under-lock stay unchanged; the PR body explains the choice.
  • Out of scope and where it goes: candidates 4 (wake at the next due time, packages/core/src/dispatch-loop.ts) and 5 (lifecycle sweep watermark), plus cloud's re-measure under cloud#3413, are in the os-dev-report. No card is filed here; cloud re-measures under its own card. Skills-docs: none.
  • Landing clause: Tier S ordinary queue landing. The seat flips ready and enables auto-merge through the relay, then confirms MERGED and closes out service-messaging: an idle delivery dispatcher tick costs 18 + 10 statements (8 partition probes each, an unconditional reap UPDATE), so a resident tenant kernel reads its database 56 times a minute with nothing queued #28383.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/m tests tooling Fix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface

Projects

None yet

1 participant