Skip to content

feat(cli): port the deterministic scenario scorer — shape score, artifact graph reader and Run grader — with a discrimination self-test - #30332

Merged
objectstack-fleet[bot] merged 6 commits into
mainfrom
claude/issue-30290-scenario-scorer-port
Nov 5, 2026
Merged

feat(cli): port the deterministic scenario scorer — shape score, artifact graph reader and Run grader — with a discrimination self-test#30332
objectstack-fleet[bot] merged 6 commits into
mainfrom
claude/issue-30290-scenario-scorer-port

Conversation

@objectstack-fleet

@objectstack-fleet objectstack-fleet Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Fixes #30290

Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:把云端评测里的确定性打分器搬进 packages/cli/src/lint(三个新文件:结构打分 scenario-score.ts、构建产物转打分图 scenario-graph.ts、对话回合判分 scenario-run-grade.ts),各带测试;不调用任何模型,不加命令行参数,不加包入口导出,目前没有调用方。
  • 为什么改:cloud#3600 D2 裁决「打分器可以迁移到开源项目」,场景库需要一个开源、纯代码的打分器;成本裁决「目前阶段尽量不要用云端的token，api的更贵」排除了模型评审。搬运时用 5 个黄金样例做「区分度自测」,实测原代码有两处会给坏构建打满分(导航指向不存在的对象、仪表盘组件绑定不存在的数据集),已在本 PR 修掉。
  • 风险与回滚:风险低。只新增文件,没有任何现有代码引用它们,发布面只多出 dist/lint/scenario-* 三个模块(包入口不可达);回滚即删这三个文件和它们的测试。
  • 先看:packages/cli/src/lint/scenario-score.ts(看顶部「与云端源码的差异」一段和 scoreGraph 的导航检查、measureChecks);packages/cli/src/lint/scenario-graph.ts(只按 spec 的唯一形状读产物,读不懂就抛错);packages/cli/src/lint/scenario-score.test.ts(区分度自测)。
  • 两分钟验证:pnpm --filter @objectstack/cli exec vitest run src/lint/scenario-score.test.ts src/lint/scenario-graph.test.ts src/lint/scenario-run-grade.test.ts,预期 3 个文件、33 个用例全绿。
  • 需要你做的:无,席位落地。

What changes

Ported under the cloud#3600 D2 ruling from the cloud eval harness (structural-score.ts, the graph reader in agent-e2e.ts, and ask-accuracy/scorer.ts), each file with the Apache-2.0 header and one provenance line. No model call anywhere; no new CLI flag; no new package-entry export.

  • packages/cli/src/lint/scenario-score.ts: ScenarioExpectation (field-for-field the card's expect block, mirrored from the sibling contract and not imported), EvalGraph, scoreGraph (exact names), ShapeExpectation, deriveShape and scoreGraphShape (name-agnostic). The reference type set is the spec's REFERENCE_VALUE_TYPES.
  • packages/cli/src/lint/scenario-graph.ts: artifactToEvalGraph(artifact) reads a built dist/objectstack.json into the graph. Collections come through resolveStackCollection (@objectstack/core, already re-exported by utils/stack-collections.ts), the rule the CLI and @objectstack/verify share, so a multi-package artifact with its collections only under packages[] (ADR-0130 option B) reads the same as a single-package one. ⛔ No local walk of packages[]. Each key is read in its one spec shape (fields record, reference via referenceTargetOf, navigation + nested children, widget dataset + values); the cloud reader's alias chains (navigation ?? nav ?? tabs ?? …, reference_to ?? referenceTo ?? reference) are deliberately not carried over. An object with no name or a fields that is not a record throws a TypeError instead of being read as an empty (degraded) object. A fixed-target type (user) gets no reference in the graph; which types fix their target is asked of the spec (referenceTargetOf({ type })), not listed here.
  • packages/cli/src/lint/scenario-run-grade.ts: gradeRun(truth, hidden, { replyText, toolCalls: [{ name, input, output }] }) returns { correct, cited, leaks, problems }. The dataset and the hosted-chat transcript type are cut out: the caller passes the truth figures and the hidden scope (objects, fields, hidden values, hidden aggregates). A tool call answered over a hidden field is detected structurally (every key and string in the input), not through a list of tool parameter names, so a raw sort / group_by spelling is still seen. Records-in-order and refusal-wording grading belonged to the cloud dataset's question kinds and are not ported.

Two deviations from the cloud source, both measured

The discrimination self-test degrades each golden build one defect at a time and requires a strictly lower score. With the cloud semantics restored, two degradations score exactly as high as the intact build:

  1. nav:TARGET->exists asked whether an object/dashboard/app of that name exists, not whether any app navigates to it. Now it passes only when an app nav entry targets the name and resolves, and every app nav entry is scored for resolving (nav-resolves:TARGET), as the shape scorer already did.
  2. A widget bound to a dataset the graph does not carry passed its measure check (!ds ||). A local artifact is the whole build, so that binding dangles and now fails, in both scorers.

Smaller: the field target key is reference (the spec's) rather than reference_to, and the platform-object prefix is sys_ / ai_ (no cloud_).

Tests

All readings at c0a6cf98 (git rev-parse --short HEAD), the final commit, under scripts/pm/os-verify-lock.sh (shared-box seconds).

  • New suites: pnpm --filter @objectstack/cli exec vitest run --maxWorkers=2 src/lint/scenario-score.test.ts src/lint/scenario-graph.test.ts src/lint/scenario-run-grade.test.ts gave Test Files 3 passed (3) · Tests 33 passed (33).
  • Package unit tier: pnpm --filter @objectstack/cli exec vitest run --project unit --maxWorkers=2 gave Test Files 387 passed (387) · Tests 5460 passed (5460). The integration tier is left to CI: the diff touches no integration file, spawn entry, or driver/kernel boot path.
  • pnpm --filter @objectstack/cli typecheck exited 0 (tsc --noEmit, then check:test-typecheck: OK). eslint --no-inline-config packages/cli/src/lint/scenario-*.ts exited 0.
Discrimination self-test and the two ablations (one-time proof, not kept as tests)

Every intact golden build scores 1 on both scorers. Each variant (missing lookup, optionless select, dangling nav target, dangling dashboard measure, dangling dashboard dataset) scores strictly lower on both scoreGraph and scoreGraphShape, over 5 builds (optionless select over 4).

The cloud semantics were put back with node scripts/ablation-replace.mjs (the anchor must hit, and it restores to blob == HEAD with an empty git diff HEAD):

  1. Nav check reverted to targets.has(norm(navName)) with the per-entry nav-resolves loop removed. Result: 3 failed / 13 passed in scenario-score.test.ts. That includes dangling nav target with invoice_with_line_items exact: expected 1 to be less than 1, which means the degraded build scored the same 1.00 as the intact one. Restored: blob eed52bc0d3ca == HEAD.
  2. Measure check reverted to !ds ||. Result: 1 failed / 15 passed, dangling dashboard dataset with invoice_with_line_items exact: expected 1 to be less than 1. Restored: blob eed52bc0d3ca == HEAD.
Clause-② measurement

After pnpm --filter @objectstack/cli build, I walked the declaration closure of the three package entries (dist/index.d.ts, dist/console.d.ts, dist/hook-body.d.ts). It holds 39 declaration files, and 0 of them are under dist/lint/. Positive control: the same walker started at dist/lint/scenario-graph.d.ts finds 2 files under dist/lint/. Reading: Clause-②: no.

Acceptance notes

  • PM mechanism assumption on the artifact reader: the CLI already has the reader, resolveStackCollection (re-exported by packages/cli/src/utils/stack-collections.ts), and it is reused. packages/verify/src/artifact-collections.ts's declaredCollection was not used: it answers a different question (a truthy top level wins, so a declared-empty array behaves differently) and the CLI does not reach into @objectstack/verify for a read it already owns.
  • PM mechanism assumption on the corpus: the 5 fixtures in corpus.ts carry objects only (no app, dataset or dashboard), so the self-test completes each with an app (nav over every object and the dashboard), one dataset and one dashboard, then parses the result with ObjectStackDefinitionSchema (the parse os build runs before writing the artifact). Every intact build scores 1 on both scorers; the reproduction of the "always 1.00" blind spot is the two ablations below, not the intact scores.
  • blog_post_with_comments declares no select, so the optionless-select variant runs on 4 of 5 builds; the test pins that count.
  • The two scorer blind spots also live in the cloud source this was ported from (carrier: cloud, reported to the seat, not filed here).

Verification

Gate list from node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands (no paths: the script takes the change set from the merge-base). Each command ran at c0a6cf98, with its exit code captured before any pipe. --ran reconciliation: 59 derived famil(ies) accounted for — 57 run, 2 NOT-MEASURED (2 DERIVED from a recorded exit 3).

  • 57 exited 0. node scripts/check-plugin-teardown-shape.mjs --self-test first exited 3 because its pinned positive-control commit was missing on a shallow checkout. After git fetch --depth=1 origin 621a4876 it exited 0.
  • NOT MEASURED: check:dual-build-cjs-loads, reason: PREREQUISITE NOT MET (it reads dist/ of 10 packages outside this diff's build closure).
  • NOT MEASURED: check:i18n-coverage, reason: PREREQUISITE NOT MET (the showcase config loads packages without build output, e.g. @objectstack/service-print).
  • Both are declared narrowings, and CI runs them on its full build. The diff touches neither dual-build output nor i18n.

Generated by Claude Code

@github-actions github-actions Bot added size/xl documentation Improvements or additions to documentation tests tooling labels Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/cli, touching 50 documentable anchor(s).

4 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/frontend/layout.mdx (via pageName (literal, a string literal in page))
  • content/docs/getting-started/common-patterns.mdx (via dashboardName (literal, a string literal in dashboard))
  • content/docs/ui/apps.mdx (via dashboardName (literal, a string literal in dashboard), pageName (literal, a string literal in page), reportName (literal, a string literal in report))
  • content/docs/ui/reports.mdx (via reportName (literal, a string literal in report))

⛔ 2 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17/17-5.mdx (via pageName (literal, a string literal in page))
  • content/docs/releases/v18/18-3.mdx (via reportName (literal, a string literal in report))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 anchor(s) matched too much of the corpus to be a work list: objectName (literal, 56 pages)
  • 29 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 61 of 219 client-bound route-ledger rows — the other 158 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 158: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 103 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 35 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json bfca0efb79ac5df2049f392cdecd3bbeeea9ec61 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 8749c7f16c2c0162a18014e8019ed8497ac74b00 — the merge of head 2c7d2600eb3e847383cf9e567fbdf89d81e23241 into base bfca0efb79ac5df2049f392cdecd3bbeeea9ec61, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 8749c7f16c2c0162a18014e8019ed8497ac74b00 && git checkout 8749c7f16c2c0162a18014e8019ed8497ac74b00
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin bfca0efb79ac5df2049f392cdecd3bbeeea9ec61 2c7d2600eb3e847383cf9e567fbdf89d81e23241 && git checkout -B drift-repro bfca0efb79ac5df2049f392cdecd3bbeeea9ec61 && git merge --no-ff 2c7d2600eb3e847383cf9e567fbdf89d81e23241

node scripts/docs-audit/affected-docs.mjs --json bfca0efb79ac5df2049f392cdecd3bbeeea9ec61

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs bfca0efb79ac5df2049f392cdecd3bbeeea9ec61 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@objectstack-fleet

objectstack-fleet Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

席位核对

Verdict: ACCEPT at head c0a6cf98d05340f4d3219d59d3e755d6abb3cd18. Landing waits for every check-run on that head to finish green.

  1. Shape.
    • Draft PR → main. Line 1 is Fixes #30290, which is right: the whole card is delivered.
    • Clause-②: no and Metadata-protocol: none sit at line start in both the body and the changeset.
  2. Premise and Clause-② measured. The report states it measured the declaration closure of dist/index.d.ts, console.d.ts and hook-body.d.ts after a build: 39 files, 0 under dist/lint. The positive control (walking from dist/lint/scenario-graph.d.ts) finds 2. I read that as no. The seat did not rebuild.
  3. Diff and scope (read via the PR files endpoint): 7 added files, +1306/−0. They are three modules plus three tests under packages/cli/src/lint/, and .changeset/30290-scenario-scorer-port.md (a patch for @objectstack/cli, which is a published package, so the changeset is owed). That is exactly the claimed surface.
    • Spot-checked the diff: provenance headers cite only issue numbers (cloud#3600, cloud#2146, cloud#2149) and carry no private cloud issue text.
    • No model or network call.
    • No CLI flag and no package-entry export.
  4. Gates.
    • Dev side, each with its command and output: 3 files / 33 tests; the CLI unit tier 387 files / 5460 tests; typecheck; eslint. dispatch-gates: 59 derived, 57 run with exit 0, 2 NOT MEASURED with the prerequisite stated (they go to CI).
    • Check-runs on c0a6cf98 at review time: 10 success, 13 skipped, the rest in_progress, 0 failure. Lint & Repo Gates and TypeScript Type Check are still running, so this is not yet queue-eligible.
  5. Accepted deviation. The port is not byte-faithful: it closes two blind spots of the cloud source, which the discrimination self-test exposed.
    • Nav existence is now checked through app nav.
    • A widget bound to a missing dataset now fails its measure check.
    • Both are proven by ablation: with each check reverted, the degraded build scored 1 == intact 1.
    • This is the card's own acceptance ("degraded scores strictly lower"), so it is accepted.
  6. Out-of-scope findings and Skills-docs.
    • The cloud source still carries both blind spots. Carrier: objectstack-ai/cloud#3604, which replaces structural-score.ts with an import of this module (pointer posted there).
    • Skills-docs: is not owed (Metadata-protocol: none).
    • Landing: once every check is green, the seat flips this PR to ready and enables auto-merge through the relay.

@objectstack-fleet objectstack-fleet Bot mentioned this pull request Nov 5, 2026
Closed
@objectstack-fleet

objectstack-fleet Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

席位核对 (patch round 1)

Verdict: ACCEPT at head 2c7d2600eb3e847383cf9e567fbdf89d81e23241. This replaces the ACCEPT on c0a6cf98, which the head move spent. Landing waits for every check-run on this head to finish green.

  1. Delta. The round's only change is a merge of origin/main plus one new row in the CALLERS table of packages/core/src/security/second-object-read-exposure.pin.test.ts (+7/−0), read via the PR files endpoint and git diff origin/main.... It is inside the amended file surface (card comment 6651380112).
  2. Class checked. The row is not-a-served-read, the same arm as the existing packages/lint/src/object-graph.ts precedent ("authoring-time static analysis of metadata; reads no record"). The reason matches the code: artifactToEvalGraph reads a declared reference target out of a built artifact, and no record is read or served.
  3. Reproduction. The pin is red at c0a6cf98 ("unclassified - packages/cli/src/lint/scenario-graph.ts", 1 failed / 3 passed) and green at 2c7d2600 (4/4). The 3 scorer suites stay 33/33. dispatch-gates re-derived 61 families; the 2 new ones exited 0.
  4. Check-runs on 2c7d2600 at review time: 14 success, 8 skipped, 18 in_progress, 0 failure. Not yet queue-eligible.
  5. PR body note. The body does not list the pin row. This record is where it is stated: packages/core/src/security/second-object-read-exposure.pin.test.ts gains one CALLERS row classifying scenario-graph.ts as not-a-served-read.
  6. Landing. Once every check is green, the seat flips this PR to ready and enables auto-merge through the relay. Items 2–6 of the round-0 record stand unchanged.

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review November 5, 2026 06:53
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge November 5, 2026 06:53
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Nov 5, 2026
Merged via the queue into main with commit 31d23c3 Nov 5, 2026
64 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-30290-scenario-scorer-port branch November 5, 2026 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation size/xl tests tooling

Projects

None yet

Milestone

No milestone

1 participant

@claude