Skip to content

Scenario library T1a: scenario-card schema + loader (private packages/qa package) and 9 open seed cards under docs/qa/scenarios/ #30289

Closed
1 of 6 issues completed

Description

@objectstack-fleet

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

Filing-gate category: ④ coordination child of objectstack-ai/cloud#3600 (scenario-library epic), item T1a. Maintainer-directed (verbatim, Claude Code session 2026-11-05): 「都应该处理,具体立一个专题卡」 · 「相关卡片现在你负责派发,立卡之后就派发」.

Business rationale: inherited from the parent. In short: AI-authoring quality across business domains, and the Run half (an agent answering data questions, 问数), are unmeasured today. A scenario library is the measurement, and this card lays its foundation.

Ruling carried

D1 on cloud#3600 (maintainer, verbatim): 「同意」. It accepts this layout:

  • the open main library lives in this repo under docs/qa/scenarios/;
  • the schema and loader live in a private packages/qa/* package (devx, not packages/spec, so the format carries no protocol obligation);
  • a private cloud overlay holds the held-out and customer-derived cards. That overlay is out of scope here.

⛔ No customer-derived scenario ever lands in this repo as-is.

Scope

  1. New private workspace package under packages/qa/ for the scenario-card schema (zod) and loader.
    • Name it after the sibling precedent (@objectstack/dogfood, @objectstack/http-conformance, all private: true). Measure that precedent before naming.
  2. Card format, six layers:
    1. prompt: { short, detailed }, each { zh?, en? } with at least one language.
    2. expect: the structural expectation. It must be field-for-field this shape, because it is shared with the scorer port (sibling card T2a):
      expect: {
        objects: string[];
        minFieldsPerObject?: number;
        selectFields?: { object: string; field: string; minOptions: number }[];
        lookups?: { object: string; field: string; to: string }[];
        dashboards?: string[];
        appNavTargets?: string[];
      }
    3. capabilities and category.
      • capabilities is a list of tags <area>.<name>. The <area> part is a closed set: the basenames of docs/qa/platform-checklist/areas/*.json, read from the tree. An unknown area is refused loudly, with the file and the path.
      • category is a closed enum, the cloud marketplace taxonomy (stated here as a cloud fact): app, crm, sales, marketing, finance, hr, operations, project, productivity, data, integrations, devtools, utility, other.
    4. run.questions (the 问数 half).
      • Seed rows per object.
      • Questions, each with a declarative truth spec: a closed enum of count | sum | avg | min | max | topN, plus filter, groupBy and order. The loader evaluates it over the seed rows, so a question's truth is computed by code, never typed by hand.
      • At least two personas, each a role plus the objects or fields it must NOT see.
    5. run.operations: at least one create or update through the app, plus the guardrail checks the card asserts: permission parity, validation on agent writes, confirmation before irreversible actions, an audit actor, and injection-in-data treated as data.
    6. iterate: a one-sentence change request plus the expected expect delta.
  3. Loader.
    • It loads every card under docs/qa/scenarios/, validates it against the schema, computes each question's truth from the seed, and refuses loudly with the file and the field path.
    • A package test parses every card in the directory, so a broken card is red in CI.
  4. Nine seed cards, each with a full Run half (seed rows, at least 3 questions, 2 personas):
    • The 5 golden cases in packages/cli/src/lint/corpus.ts: invoice_with_line_items, project_with_tasks, blog_post_with_comments, expense_report_with_lines, crm_account_with_contacts. Derive expect from each fixture.
    • The 4 generic build cases of cloud's private eval dataset. Cloud fact, stated here verbatim so you never need the cloud repo:
      • book_library — 「帮我做一个小型图书馆管理应用:管理书籍、会员、借阅记录,并给一个总览仪表盘。」
        • objects book, member, loan; minFieldsPerObject 4;
        • selects book.status ≥2 and loan.status ≥2;
        • lookups loan.book→book and loan.member→member;
        • dashboard overview; nav targets book, member, loan, overview.
      • tickets_select_dense — "Build a support-ticket tracker with priority and status."
        • object ticket; minFieldsPerObject 4;
        • selects ticket.priority ≥3 and ticket.status ≥3.
      • leave_system — 「做一个请假申请系统:员工档案 + 请假申请(含请假天数、状态、审批人),再加一个按状态统计的仪表盘。」
        • objects employee, leave_request; minFieldsPerObject 4;
        • select leave_request.status ≥2;
        • lookup leave_request.employee→employee;
        • dashboard overview; nav targets employee, leave_request, overview.
      • purchase_master_detail — "Create a purchase order app: orders with multiple line items."
        • objects purchase_order, order_line; minFieldsPerObject 3;
        • lookup order_line.purchase_order→purchase_order.
  5. docs/qa/scenarios/README.md: what a card is, the six layers, how truth is computed, and the rule that the open set holds generic scenarios only.

Out of scope

Each of these is a sibling card under cloud#3600:

  • the scorer (T2a);
  • capability-tag checks and CLI wiring;
  • the local runner;
  • the cloud overlay;
  • customer-derived cards.

Acceptance

  • The package exists, private: true, and its tests pass.
  • All 9 seed cards load, and their computed truths are asserted in a test.
  • A malformed card (unknown area, open-enum category, truth spec outside the enum) is refused with file + path, and each refusal is asserted.
  • No packages/spec change and no public-surface change.

Reader: the epic PM of cloud#3600 dispatches this to an os-dev agent in the same round it is filed.

Dedup: see the parent's dedup table. Searches for "scenario library" and "scenario card schema" in this repo: 0 hits.


Generated by Claude Code

Activity

  1. added
    domain:devxDomain lane — seat card indexed by label:pm:seat
    pm:epicReserved by a dedicated epic PM, parent or sub-issue; other PMs never take it; never with pm:queue
    area:devpathThe road — create, dev, verify, publish/install, connect an agent, iterate
    on Nov 5, 2026
  2. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Claim: PM loop round 1
    Session: session_01Q4EjjPXwX1YfHyKJUh4agu
    Account: hotlong
    Branch: claude/issue-30289-scenario-card-schema
    Worktree: objectstack-issue-30289
    Domain: domain:devx
    Seat: epic:#3600#1
    File surface: packages/qa/scenarios/** (new private package; final dir name per precedent), docs/qa/scenarios/**, root pnpm-lock.yaml/workspace entries the new package needs, .changeset/30289-scenario-card-schema.md (stop on breach; explain in the report)
    Container & model: M, mode:subagent, model: TIER_DEFAULT (dispatch-gates --tier: "no path-derived mandate"), effort: high
    Clause-②: no
    Metadata-protocol: none
    Responsibility: n/a — not a defect card
    Thread-read: none
    Premise: holds
    Serial constraints cleared: none — the 33 open pm:dispatched cards' claimed file surfaces were read and none touches packages/cli/src/lint, packages/qa or docs/qa/scenarios; sibling #30289/#30290 surfaces are disjoint

    Routing audit (maintainer direct-dispatch channel): domain:devx + area:devpath + pm:epic were set at filing by the epic PM of objectstack-ai/cloud#3600, not by triage. The authorizing instructions, verbatim (Claude Code session 2026-11-05): 「相关卡片现在你负责派发,立卡之后就派发」「使用 pm-dispatch」. Domain read from the landing paths: packages/qa/** and docs/qa/** are devx. Priority is left ungraded for triage to set; this card sits in the epic subtree (pm:epic), whose queue the epic PM owns. Seat-account override: see cloud#3600 comment 6650372672.

  3. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

    Path: none

    Triage: first-touch audit · p2 · epic child · domain:devx kept · 2026-11-05T05:59Z

    ⛔ Not a claim.


    Generated by Claude Code

  4. added
    pm:blockingDerived cache from the Blocked-by reverse index: open card with open dependents — never hand-set
    on Nov 5, 2026
  5. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
    os-dev-report

    {
      "issue": 30289,
      "status": "done",
      "branch": "claude/issue-30289-scenario-card-schema",
      "pr": "https://github.com/objectstack-ai/objectstack/pull/30340",
      "session": "session_01Q4EjjPXwX1YfHyKJUh4agu",
      "premise_still_valid": true,
      "summary": "New private package @objectstack/scenarios (packages/qa/scenarios; private, exports at src, no build, zod only) holds the scenario-card zod schema (six strict layers; the expect shape is exactly the one the card fixes, with nothing renamed and no optional key added), a truth evaluator and loadScenarios(dir, {areas}). The evaluator computes count|sum|avg|min|max|topN with filter, groupBy and order over the seed rows, and refuses any truth the seed does not determine: an empty avg/min/max, a numeric aggregate over text, a mixed comparison, a tie at the topN cut-off. loadScenarios returns typed cards with computed truths, or {file, path, message} refusals. Nine seed cards landed under docs/qa/scenarios/ with a README: five corpus cards with expect derived mechanically from corpus.ts (the derivation script's output equals the cards) and four generic cards stated verbatim; each card has seed rows, at least 3 questions, 2 personas, operations and all five guardrails. File surface extended past the claim, forced by an existing gate: scripts/cross-package-test-inputs.mjs + turbo.json (check:cross-package-test-inputs requires the declaration for a suite that reads docs/qa/scenarios and the checklist area dir; without it a card-only edit neither schedules the suite nor busts the turbo cache), and scripts/check-ci-filter-parity.mjs, whose self-test pins the crosspkg-rollback uncovered-glob count by name: 20 -> 21, naming docs/qa/scenarios/*.json. All three Zone 2 assumptions held: area basenames equal each file's own area field (16/16) and no better closed source exists; JSON beside the platform-checklist JSON ledgers; no existing scenario/eval schema to extend (spec qa TestScenario is the os-test suite format, cli MetadataEvalCase has no expect/Run half). Resumed after a rate-limit stop; no work was redone.",
      "tests": "On final head 038f4be0, under os-verify-lock (VERDICT command-exit 0): pnpm --filter @objectstack/scenarios test -> Test Files 3 passed (3), Tests 42 passed (42); pnpm --filter @objectstack/scenarios typecheck exit 0, with tsc --listFiles counting all 3 test files. library.test.ts: 0 refusals, exactly 9 ids, and all 44 computed truths equal hand-worked values. refusals.test.ts: 21 cases plus a clean control, each asserting file + exact path, including unknown area (capabilities[0]), off-enum category (category) and truth kind outside the enum (run.questions[0].truth.kind). truth.test.ts: evaluator semantics. One ablation through scripts/ablation-replace.mjs (no dist involved; tests import ../src): avg / rows.length instead of / values.length, anchor 1 -> 0, blob 0afa46f9 -> a6a44d3e; truth.test.ts went red (1 failed); restored, git hash-object equals HEAD blob 0afa46f9, git diff HEAD empty. test:repo is not run locally (no test:repo file touched; declared to CI).",
      "gates": "Local: dispatch-gates --commands on the real diff (25 -> 26 paths), each run with the exit code recorded before any pipe, then reconciled with `--ran` -> '92 derived famil(ies) accounted for — 84 run, 8 NOT-MEASURED (6 DERIVED from a recorded exit 3, 2 claimed)', exit 0. All 84 run exit 0. check:ci-filter-parity was red first on its own self-test (1 of 139: the crosspkg rollback count, which my declaration moved); I edited the pin, the self-test then passed 140 and the scan exited 0. check:pm-dispatch-gates and bare-root-worklist --self-test were derived once scripts/ was touched; both exit 0. NOT MEASURED, exit 3 PREREQUISITE NOT MET (no dist in this worktree; the diff builds no package): check:dts-closure, check:dual-build-cjs-loads, check:lean-entry-closure, check:sourcemap-no-sources-content, @objectstack/lint check:doc-formula-expressions. check-plugin-teardown-shape --self-test needs an unshallowed clone and exits 3 on the shared checkout too. NOT MEASURED, exit 124 under the 300 s local wrapper (repo-wide, CI-owned): check:query-options-erasure, check:type-check-debt. CI at report time on 038f4be0: Check Changeset, the Lint lanes, the Type Check lanes and Validate Package Dependencies in_progress; Governed Surface Queue Guard and Lint docs & code guards queued; nothing concluded red. Changed lines 2823 (under the 3000 human-merge line).",
      "dogfood_battery": "n/a — no `!` in the PR title and no (narrowing) arm",
      "mcp_calls": "0",
      "api_writes": "2 — relay pr_create (repository_dispatch -> POST /repos/objectstack-ai/objectstack/pulls, draft, no labels, no assignees; read-back identical 11597 bytes), and this os-dev-report comment via post-stamped.mjs (POST /repos/objectstack-ai/objectstack/issues/30289/comments). git push is not a REST write.",
      "open_questions": [],
      "security_findings": "none",
      "out_of_scope_findings": [
        "carrier: CLI-wiring sibling card under cloud#3600 · noted, not filed: the five corpus cards' expect is derived once and is not pinned equal to packages/cli/src/lint/corpus.ts (a pin would make a private qa package import CLI internals; corpus.ts is not exported). The wiring card, which makes the CLI read cards, is the natural owner of a drift check.",
        "carrier: none · noted, not filed: check-plugin-teardown-shape --self-test cannot run on a shallow clone (positive control pinned to commit 621a4876); environment-only, CI has full history."
      ]
    }
  6. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    File-surface amendment for the claim above, recorded by the seat at review of PR #30340:

    • Added: scripts/cross-package-test-inputs.mjs (one @objectstack/scenarios row), turbo.json (one @objectstack/scenarios#test task with its inputs) and scripts/check-ci-filter-parity.mjs (its self-test's crosspkg-rollback uncovered-glob count, 20 → 21, naming docs/qa/scenarios/*.json).
    • Why: an existing gate forced them. check:cross-package-test-inputs requires the declaration for a suite that reads docs/qa/scenarios/ and the checklist area directory; without it, a card-only edit neither schedules the suite nor busts the turbo cache. The parity self-test pins that count by name.
    • None of the three is a governed surface. The dev reported the breach rather than staying silent (report on this card, comment 6651475028).
  7. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    ACCEPT → PR #30340 (head 038f4be0), landing held. Waits-on: objectstack-ai/cloud#3600 decision Q1 (comment 6651166095). Since: this comment.

  8. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Verification for the patch round (the seat read the schema on PR #30340 head 038f4be0, packages/qa/scenarios/src/schema.ts). The maintainer's 8–10 问数 directive on objectstack-ai/cloud#3600 (comment 6651865232) needs four things the current truth spec cannot express:

    1. Cross-object questions. A question names one object, and filter / groupBy read that object's own fields only (FILTER_OPS = eq ne gt gte lt lte in). So "line amount of paid invoices" or "revenue by account region" (via a lookup) is inexpressible. Needs a relation path on filter and groupBy (e.g. invoice.status) resolved through the seed rows' lookup ids.
    2. Time windows. Only absolute string comparison is possible. "This month" or "last 30 days" needs a card-level fixed asOf date plus relative-window filters, so a truth stays deterministic.
    3. Permission-boundary truths. A persona declares hidden objects and fields only. Row scope (a rep sees only their own records) is not modeled, so a question's answer cannot differ per persona. Needs a per-persona row scope (e.g. an owner field) and truths computed per persona.
    4. "Should answer: I don't know." There is no truth kind for an unanswerable question. Needs one (e.g. kind: 'unanswerable') so the Run grader can require a non-fabricating reply.

    These join the PR #30340 patch round once decision Q1 v2 is ruled. They extend closed enums and shapes; they do not loosen them.

  9. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Patch round 1 dispatched to the same claim and branch (claude/issue-30289-scenario-card-schema), with the original dev resumed. The ACCEPT on PR #30340 (head 038f4be0) is spent by this round; the seat re-reviews the new head, and the landing hold is lifted (Q1 fully ruled).

    Rulings carried (objectstack-ai/cloud#3600):

    • Q1 = v3 list: comment 6658194654, 「同意这9个场景」.
    • Execution parameter and declarative floor: comment 6658499884.
    • 8–10 问数 directive: comment 6651865232.
    • Truth-spec gaps: this card, comment 6652011807.

    Size re-graded from M to L: 6 mid-tier cards and 5 schema extensions. Effort xhigh; model TIER_DEFAULT (dispatch-gates --tier: no path-derived mandate). File surface unchanged: docs/qa/scenarios/**, packages/qa/scenarios/**, plus the three gate-forced files already amended in comment 6651510630. objectstack-ai/templates is read as a reference only.

  10. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
    os-dev-report

    Addendum, patch round 1: stopped before push on the 3000-line threshold; measurement and the decision needed below.

    {
      "issue": 30289,
      "status": "blocked",
      "branch": "claude/issue-30289-scenario-card-schema",
      "pr": "https://github.com/objectstack-ai/objectstack/pull/30340",
      "session": "session_01Q4EjjPXwX1YfHyKJUh4agu",
      "premise_still_valid": true,
      "summary": "Patch round 1 is STOPPED before any push, as the dispatch orders when the diff will exceed the 3000-line human-merge threshold. PR #30340 head is unchanged at 038f4be0. Measurement (objectstack-ai/objectstack; worktree from origin/claude/issue-30289-scenario-card-schema 038f4be0, merged with origin/main f506b79d into local ca2bee0a, never pushed): the branch is now 2838 changed lines vs merge base (2835+/3-). With the six retired cards git-rm'd it is 2070 (2067+/3-), which leaves 930 lines of headroom. The ruled scope's floor, at one line per seed row/question/operation and with ZERO schema code, tests, README or control upgrades, is about 906 lines of card JSON: 6 x >=100 main-object rows = 600; >=4 related objects x 6 with key/bracket lines ~84; 6 x 8 questions = 48; 6 x 4 operations = 24; 6 x 3 agent actions = 18; personas 12; guardrails 30; card headers ~90. That alone reaches ~2976. The five schema arms plus agentActions plus their refusal tests and truth pins add at least ~350 more, so the floor is ~3330 and the PR is over 3000 before a single real card is written. Realistically the line/child objects each main object implies (claim lines, order lines, invoices, payments, receipts...) put the total at roughly 5000-6500. No work was split or pushed silently. Decision needed from the seat (options and recommendation in open_questions). The measurement worktree and its local branch tmp-30289-measure were removed with --force on purpose: it held only the throwaway staged deletions and the unpushed merge commit.",
      "tests": "None run this round: no code was changed. Last measured state is unchanged, PR head 038f4be0: Test Files 3 passed, Tests 42 passed (see the original report).",
      "gates": "NOT MEASURED: patch round stopped before implementation; no new diff to derive gates for. PR #30340 CI on 038f4be0 is unaffected by this round.",
      "dogfood_battery": "n/a — no `!` in the PR title and no (narrowing) arm",
      "mcp_calls": "0",
      "api_writes": "1 — this addendum comment on #30289 via post-stamped.mjs (relay, POST /repos/objectstack-ai/objectstack/issues/30289/comments). No push, no PR body update (nothing to describe yet).",
      "open_questions": [
        {
          "question": "How should the v3 library land, given that the measured floor (~3330 changed lines) and the realistic size (~5000-6500) exceed the 3000-line human-merge threshold?",
          "options": [
            "A. Split by landing order, every PR branched off main (not stacked). PR #30340 becomes `Part of #30289` and carries the five schema extensions + agentActions + their refusal tests + the 3 upgraded controls + deletion of the 6 retired cards (estimated ~2600-2900, re-measured before push; if it still crosses, the controls' upgrade moves out). Each mid-tier card then lands as its own PR (~500-900 lines each: card JSON + its own per-card truth-pin test file, so the six do not conflict), each `Blocked-by` #30340 and dispatchable in parallel once it merges; the last one closes #30289. Cost: 7 PRs and 7 reviews, but each is under the line and reviewable. Work: same total.",
            "B. One PR over 3000 via the AGENTS.md §7(c) path (an authorized APPROVED review, then the owning seat lands it). Cost: one ~5000-6500 line review for the maintainer-authorized approver, and nothing lands until all 6 mid-tier cards are done. Work: same total.",
            "C. Add a declarative seed generator to the card format (row templates x cycles over dates and owners), so 100+ rows cost a few lines and the whole round might fit one PR under 3000. Cost: a new format arm no ruling asked for, seeds no longer readable row by row, hand-verifying truths gets much harder (an oracle independent of the evaluator becomes impractical), and it expands the declared surface. Work: more than A."
          ],
          "recommendation": "A, on the four axes. Real business need: every piece serves a ruled card, and the schema arms are needed under every option, so landing them first unblocks parallel card authoring. Long-term soundness: small, reviewable, contract-first PRs; the per-card test file is the sustainable shape for a growing library. Hard-to-get-wrong for AI authors: the schema's strict refusals are in force before any mid-tier card is authored, so each card PR is judged by the hardened loader. No scope creep for the startup stage: A adds no format arm, while C adds a speculative one; B keeps one PR but piles everything onto one approver. The seat chooses; if A, the next patch round on this PR is the schema + controls slice and the six mid-tier cards are filed as sub-issues."
        }
      ],
      "security_findings": "none",
      "out_of_scope_findings": []
    }
  11. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Seat decision on the dev's fork (report comment 6658595859): split, option A

    The dev stopped before pushing, as ordered: the v3 scope measured at a floor of ~3330 changed lines and a realistic ~5000–6500, over the 3000-line human-merge threshold. How work is sliced into PRs is task ordering, which the protocol does not escalate. Product and contract choices are not made here: option C would add a seed-generator format arm that no ruling asked for, and it is not taken.

    Decision (A):

  12. added
    priority:p1High: required for production / M2
    and removed on Nov 5, 2026
  13. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Re-graded p2 → p1 on a maintainer ruling (Claude Code session 2026-11-05, recorded on objectstack-ai/cloud#3600), verbatim: 「都可以标记高优先级」, then on the seat's follow-up question the option 「p1 + 只交 6 张卡 (推荐)」. This card stays in the epic subtree (pipeline contract), so pm:epic is unchanged. Note for triage: this is a direct maintainer re-grade. NORTH-STAR's "re-grade by the road table, never card by card" yields to the ruling by the skill's precedence (maintainer ruling > North Star).

  14. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
    os-dev-report

    Addendum, patch round 1 (option A, foundation): done on head c1affd3. The PR body is not patched by the dev (os-dev.md); the replacement body for the seat to write is in the details block below.

    {
      "issue": 30289,
      "status": "done",
      "branch": "claude/issue-30289-scenario-card-schema",
      "pr": "https://github.com/objectstack-ai/objectstack/pull/30340",
      "session": "session_01Q4EjjPXwX1YfHyKJUh4agu",
      "premise_still_valid": true,
      "summary": "Patch round 1 (option A, foundation only) is delivered on head c1affd3f (merged with origin/main f506b79d as e5a49c33). It adds five schema extensions. (1) Relation paths on filter/groupBy, resolved through lookups declared in expect.lookups and the seed's lookup ids. (2) A card-level asOf plus a new in_window filter op (last_days N / calendar this|previous month|year), inclusive and deterministic. (3) A persona row scope ({object, field-or-path, value}); a question that lists personas also gets per-persona truths, returned as personaTruths, and the truth is `withheld` when the question reads an object or field that persona hides. (4) A truth kind `unanswerable` with a reason. (5) Every operation names actingAs and declares an outcome: applied post-state by id/where with values or absent, plus auditActor; refused with a code from the spec's closed ErrorCode catalogue; or pending_confirmation. The guardrail checks are held to those outcomes. Also: optional run.agentActions (>=3 actions, >=1 confirm-gated), and a counts band (>=5 questions and >=2 operations; below 8/4, countsReason is required). The expect shape is untouched. The 3 controls are upgraded: invoice_with_line_items 9 questions / 5 operations, project_with_tasks 8/5, crm_account_with_contacts 8/5. Each control has cross-object, per-persona and unanswerable questions (invoice also a this_month window), 5 operations (applied create, audited update, confirm-gated delete, refused VALIDATION_FAILED, refused PERMISSION_DENIED) and 3 agent actions. The 6 retired cards and their pins are git rm'd. The README documents the v3 library shape and the mid-tier floor, including the declarative floor. No mid-tier card is written. CI fix from the patch-round message: the refusal-code field is now `ErrorCode` from @objectstack/spec/api, read from the export and not restated, so the package depends on @objectstack/spec (workspace) with a vitest source alias. The controls expect VALIDATION_FAILED, the engine record validator's per-field code (packages/types validation-failure.ts, objectql VALIDATION_FAILED_CODE), instead of the generic VALIDATION_ERROR. CONFLICT noted, not resolved silently: the dispatch orders the dev to update the PR body via the relay, but .claude/agents/os-dev.md says the dev writes the body once at pr_create and never PATCHes it, with the seat writing later changes, and that file wins on conflict. So the body is not patched. The full replacement body is attached under the addendum comment's details block (four-backtick fence) for the seat to write. Size measured before every push: 2532 changed lines (+2529/-3, 20 files), under 3000.",
      "tests": "On head c1affd3f under os-verify-lock (VERDICT command-exit 0): pnpm --filter @objectstack/scenarios test -> Test Files 3 passed (3), Tests 59 passed (59). pnpm --filter @objectstack/spec build && pnpm --filter @objectstack/scenarios typecheck -> exit 0. library.test.ts: 25 truths + 10 persona truths, hand-worked. refusals.test.ts: 38 tabled refusal cases, each asserting exactly one refusal with file + exact path + message class, plus 2 controls and the id/basename and invalid-JSON cases. New-arm cases: relation paths 2 (run.questions[6].truth.groupBy, run.questions[5].truth.filter[0].field); windows 4 (run.questions[0].truth.filter[0].value x2, run.questions[0].truth.filter[0].field, asOf); persona scope/personas 2 (run.personas[0].scope[1].field, run.questions[4].personas[0]); unanswerable 1 (run.questions[7].truth.reason); outcomes 5 (run.operations[0].outcome, run.operations[0].actingAs, run.operations[3].outcome.code, run.operations[0].outcome.state[0].id, run.operations[1].outcome.state[0].id); guardrail-outcome 2 (run.guardrails[2].operation, run.guardrails[3].operation); agentActions 2 (run.agentActions, run.agentActions[0].object); countsReason 1 (run.countsReason). Ablations, one per new arm, via scripts/ablation-replace.mjs (tests import ../src; spec aliased to source, so no dist leg), each restored byte-identical (git hash-object == HEAD blob), git diff HEAD empty after: relation_paths | exit 1 | landed True | blob 73dcb24d->3fb8af58 | 32 ⎯⎯⎯⎯⎯⎯⎯ | restored True | red: ['loads every card with no refusal', 'holds exactly the three control cards (the mid-tier cards land on their own)'] ;; time_windows | exit 1 | landed True | blob d562f0d3->9254d982 | 2 ⎯⎯⎯⎯⎯⎯⎯ | restored True | red: [\"invoice_with_line_items: every question's computed truth is the hand-worked one\", 'computes windows against asOf, inclusive, across month and year edges'] ;; persona_row_scope | exit 1 | landed True | blob 73dcb24d->c801f0c3 | 3 ⎯⎯⎯⎯⎯⎯⎯ | restored True | red: [\"invoice_with_line_items: every question's computed truth is the hand-worked one\", \"project_with_tasks: every question's computed truth is the hand-worked one\"] ;; persona_withheld | exit 1 | landed True | blob 73dcb24d->39a838a3 | 3 ⎯⎯⎯⎯⎯⎯⎯ | restored True | red: [\"invoice_with_line_items: every question's computed truth is the hand-worked one\", \"project_with_tasks: every question's computed truth is the hand-worked one\"] ;; unanswerable | exit 1 | landed True | blob d562f0d3->f9a664ff | 4 ⎯⎯⎯⎯⎯⎯⎯ | restored True | red: [\"invoice_with_line_items: every question's computed truth is the hand-worked one\", \"project_with_tasks: every question's computed truth is the hand-worked one\"] ;; outcome_guardrail_match | exit 1 | landed True | blob 73dcb24d->d93bd217 | 1 ⎯⎯⎯⎯⎯⎯⎯ | restored True | red: ['refuses a confirmation check on an operation that does not wait'] ;; outcome_post_state_id | exit 1 | landed True | blob 73dcb24d->f1d67cba | 1 ⎯⎯⎯⎯⎯⎯⎯ | restored True | red: ['refuses a post-state naming a created row by an id the seed lacks'] ;; agent_actions_confirm | exit 1 | landed True | blob 73dcb24d->58d94865 | 1 ⎯⎯⎯⎯⎯⎯⎯ | restored True | red: ['refuses agent actions with none confirm-gated'] ;; counts_reason | exit 1 | landed True | blob 73dcb24d->313cb397 | 1 ⎯⎯⎯⎯⎯⎯⎯ | restored True | red: ['refuses fewer than 8 questions without a stated reason'] ;; refusal_code_catalogue | exit 1 | landed True | blob 47f63c7d->163daac4 | 1 ⎯⎯⎯⎯⎯⎯⎯ | restored True | red: ['refuses a refusal code outside the platform error-code catalogue'] ;; git diff HEAD empty: True",
      "gates": "dispatch-gates --commands on c1affd3f (20 paths; changed lines 2532 vs threshold 3000: under), each run with the exit recorded before any pipe; `--ran` -> '92 derived famil(ies) accounted for — 88 run, 4 NOT-MEASURED (3 DERIVED from a recorded exit 3, 1 claimed)', exit 0. All 88 run exit 0, including check:pm-dispatch-gates (rerun without the wrapper timeout, exit 0), check:cross-package-test-inputs, check:ci-filter-parity, check:test-source-alias and check:undeclared-dep-imports. check:error-code-casing, the CI red on c3a257d5: reproduced locally exit 1 ('packages/qa/scenarios/test/refusals.test.ts:83 validation_error (assignment)'); exit 0 on c1affd3f ('no unlisted lowercase error codes in 11258 scanned file(s)'); no EXEMPT_FILES entry. NOT MEASURED: exit 3 PREREQUISITE NOT MET (no dist for those packages) check:dual-build-cjs-loads, check:lean-entry-closure, @objectstack/lint check:doc-formula-expressions; exit 124 under the 420 s wrapper check:type-check-debt (CI-owned). CI on c1affd3f at report time: 58 check-runs success, the rest skipped, none red or pending.",
      "dogfood_battery": "n/a — no `!` in the PR title and no (narrowing) arm",
      "mcp_calls": "0",
      "api_writes": "1 — this addendum comment on #30289 via post-stamped.mjs (relay, POST /repos/objectstack-ai/objectstack/issues/30289/comments). The PR body was NOT patched (os-dev.md rule: the dev writes the body once; the seat writes later changes). The replacement body is attached in this comment.",
      "open_questions": [],
      "security_findings": "none",
      "out_of_scope_findings": [
        "carrier: mid-tier sub-issues #30476-#30481 · noted, not filed: the existing rule that every persona hides at least one object or field means a card has no all-seeing admin persona, and operations act as scoped personas. If a mid-tier card needs an admin role, relaxing that rule is a card-author decision for those sub-issues; it is not made here.",
        "carrier: none · noted, not filed: write_validation outcomes assume the runtime answers VALIDATION_FAILED (record validator) for a missing required field or a malformed email. This is grounded in the engine's ValidationError code, not in a booted run. The local runner card measures it against a live app."
      ]
    }
    Replacement PR body for #30340 (seat to write)
    Fixes #30289
    
    Clause-②: no
    Metadata-protocol: none
    
    ## 维护者速读
    
    - **改了什么:** 新增私有包 `@objectstack/scenarios`(`packages/qa/scenarios`,不发布、无构建步骤、不调用任何模型),内含场景卡的 zod schema 与加载器。按 v3 场景库的地基改造:五项 schema 扩展(跨对象关系路径、卡级 `asOf` 加相对时间窗、按角色的行范围与逐角色真值、"种子答不了"的真值类型、写操作声明预期结果),外加可选的 `run.agentActions`。三张简单对照卡(发票、项目任务、CRM)升级到新格式,每张 8–9 个问数、5 个写操作。删除 6 张退役卡。README 写明 v3 场景库形态和中档卡的能力底线。
    - **为什么改:** cloud#3600 Q1 = v3(「同意这9个场景」)与执行参数裁决;席位选定方案 A:本 PR 只做地基,6 张中档卡各自成单(#30476–#30481,均 Blocked-by 本 PR,合并后并行派发),以免单个 PR 超过 3000 行人工合并线。
    - **风险与回滚:** 不触 `packages/spec`、不改公开面、包为 `private: true`、`expect` 形状未动(与已合并的评分器共用)。仓级改动只有已登记的三处(cross-package 声明、turbo 输入、ci-filter-parity 自测按名记录)。回滚 = revert 本 PR。
    - **先看:** ① `packages/qa/scenarios/src/loader.ts` 的 `checkPath` / `resolverFor`(关系路径)和逐角色真值那一段;② `packages/qa/scenarios/src/schema.ts` 的 `OutcomeSchema` 与 guardrail 对 outcome 的要求(在 `loader.ts`);③ `docs/qa/scenarios/invoice_with_line_items.json`(新格式的完整样子)。
    - **两分钟验证:** `pnpm --filter @objectstack/scenarios test`,预期 `Test Files 3 passed`、`Tests 59 passed`;把 `invoice_with_line_items.json` 里 sales_rep 的 `scope` 删掉再跑,library 用例转红,`amount_per_invoice` 的逐角色真值对不上。
    - **需要你做的:** 无,席位落地。
    
    ## What changes
    
    **Rescope (seat decision, option A).** This PR is now the v3 library's foundation. The six mid-tier cards (travel expense, order-to-cash, service desk, leave and balances, requisition-to-receipt, asset lifecycle) are sub-issues #30476, #30477, #30478, #30479, #30480 and #30481. Each is Blocked-by this PR and each lands as its own PR. No mid-tier card content is written here.
    
    **New private package `@objectstack/scenarios`** (`packages/qa/scenarios`), named after its siblings (`private: true`, `exports` at `src`, no build step, `zod` its only runtime dependency). It has three files:
    - `src/schema.ts`: the card shape, with strict objects throughout.
    - `src/truth.ts`: the declarative truth evaluator.
    - `src/loader.ts`: `loadScenarios(dir, { areas })` returns typed cards with `truths` and `personaTruths`, or `{ file, path, message }` refusals. A card with any refusal is not returned.
    
    **The five schema extensions.** Each one extends a closed enum or shape; nothing is loosened.
    
    1. **Relation paths.** `filter[].field` and `groupBy` accept `lookup.field` and deeper paths, such as `invoice.status` or `account.industry`. Every hop except the last must be a lookup declared in `expect.lookups`, and the path resolves through the seed rows' lookup ids. A hop that is not a declared lookup is refused at the exact path, and so is a final field that no seed row of the reached object carries.
    2. **Time windows.** `asOf` is a card-level fixed date, validated as an ISO date. The new filter op `in_window` takes a value of `{ kind: last_days, days }` or `{ kind: calendar, period: this_month | previous_month | this_year | previous_year }`, and the window is inclusive and computed against `asOf`. The loader refuses three things: a window when the card has no `asOf`, a window over a field that holds no date, and a window value given to any op other than `in_window`.
    3. **Persona row scope.** A persona can now declare `scope: [{ object, field, value }]`, where `field` may be a path (`invoice.owner` scopes invoice lines through their invoice). A question can list `personas`, and the loader then also computes that question's truth as each listed persona. Before evaluating, it applies the persona's scope rules for the question's object. If the question reads an object or field the persona hides, the persona's truth is `{ kind: withheld }`, meaning a correct reply declines.
    4. **Unanswerable.** A new truth kind `{ kind: unanswerable, reason }` says the seed cannot answer, so a correct reply says so instead of giving a figure. It joins the closed enum.
    5. **Declared write outcomes.** Every operation now names the persona it acts for (`actingAs`) and declares an `outcome`, so it can be graded by code alone. The outcome is one of:
       - `applied`, with the post-state rows: each named by a seeded `id`, or by `where` for a row the write creates, and carrying either the `values` the data API must read back or `absent`. It can also name an `auditActor`.
       - `refused`, with a code from the spec's closed `ErrorCode` catalogue (`@objectstack/spec/api`: StandardErrorCode plus the registered ledger, ADR-0112). The catalogue is read from the spec export and never restated here, so the package now depends on `@objectstack/spec` and its vitest config aliases that import to spec source.
       - `pending_confirmation`.
    
       The guardrail checks are now held to the outcome: `write_validation` needs a refused create or update, `confirm_irreversible` needs `pending_confirmation`, and `audit_actor` needs `applied` with an `auditActor`.
    
    Two smaller additions:
    - The optional `run.agentActions` lists at least 3 agent-callable business actions, `{ name, object, summary, confirm }`, with at least one confirm-gated.
    - The counts band: a card needs at least 5 questions and 2 operations. Below the default of 8 questions or 4 operations it must state why in `run.countsReason`, or it is refused.
    
    **The three controls, upgraded.** The `expect` blocks are untouched: the corpus derivation still equals the cards.
    
    | Card | Questions | Operations | New-arm questions |
    |:---|:---|:---|:---|
    | `invoice_with_line_items` | 9 | 5 | cross-object `invoice.status`; `issued_on` this month (`asOf` 2026-10-31); 2 per-persona (sales_rep row scope, warehouse_staff withheld); 1 unanswerable |
    | `project_with_tasks` | 8 | 5 | cross-object filter and groupBy through `project.status`; 2 per-persona (contractor scope by assignee, client_viewer and contractor withheld); 1 unanswerable |
    | `crm_account_with_contacts` | 8 | 5 | cross-object filter and groupBy through `account.industry`; 2 per-persona (sdr scope via `account.owner`, partner_user withheld); 1 unanswerable |
    
    Each control's five operations are the same set of outcomes: one applied create, one applied update with an audit actor, one confirm-gated delete, one refused validation violation (`VALIDATION_FAILED`, the code the engine's record validator answers for a per-field refusal), and one refused overstep by a persona without the right (`PERMISSION_DENIED`). Each control also declares 3 agent actions, one of them confirm-gated.
    
    **Retired:** the six earlier generic cards are removed with their test pins. `docs/qa/scenarios/README.md` now describes the v3 shape, the new arms, and the mid-tier capability floor, including the declarative floor: flows, formulas and validation rules are preferred over code hooks, so a graduated app can install unmanaged.
    
    **Repo wiring forced by existing gates**, unchanged from the first round:
    - `scripts/cross-package-test-inputs.mjs` and `turbo.json` declare that this suite reads `docs/qa/scenarios/*.json` and the checklist area files.
    - `scripts/check-ci-filter-parity.mjs` names that glob in its rollback pin, which moves from 20 to 21.
    
    ## Tests
    
    - `test/library.test.ts`: the three controls load with zero refusals. All 25 truths are asserted against hand-worked values, and so are the 10 persona truths.
    - `test/refusals.test.ts`: 38 refusal cases in a table, each asserting the single refusal's `file`, exact `path` and message class. There are also 2 controls (the unmutated card loads clean, and a below-band card loads once it states `countsReason`), plus id/basename and invalid-JSON cases. The new arms are covered by:
      - relation paths: 2 cases;
      - time windows: 4 cases;
      - persona scope and personas: 2 cases;
      - unanswerable: 1 case;
      - outcomes and actingAs: 5 cases;
      - guardrail-to-outcome matching: 2 cases;
      - agent actions: 2 cases;
      - countsReason: 1 case.
    - `test/truth.test.ts`: the evaluator, now including unanswerable, the resolver for paths, and window ranges across month and year edges (including a leap February).
    
    ## Acceptance notes
    
    - **Size:** 2532 changed lines vs the merge base (+2529 / -3, 20 files), under the 3000 human-merge line. This was measured before every push.
    - **Persona rows:** a persona must still hide at least one object or field (the earlier rule). So no persona is all-seeing, and an operation acts as a scoped persona. That fits the per-persona grading, but it means a card has no admin persona. If the mid-tier cards need one, relaxing that rule is a follow-up decision, not made here.
    - **Withheld semantics:** a question reading an object or field a persona hides answers `withheld` for that persona. That includes a hidden field reached through a relation path, and the lookup field used as a hop counts as read.
    - Not wired to the CLI or the scorer, which is sibling work. The `expect` shape is untouched.
    
    ## Verification
    
    <details><summary>Package suite and typecheck (head c1affd3f)</summary>
    
    ```
    pnpm --filter @objectstack/scenarios test       Test Files  3 passed (3) · Tests  59 passed (59)
    pnpm --filter @objectstack/scenarios typecheck  exit 0
    os-verify-lock: VERDICT command-exit 0
    ```
    </details>
    
    <details><summary>Ablations: one per new arm, through scripts/ablation-replace.mjs, each restored byte-identical (git hash-object == HEAD blob), git diff HEAD empty after</summary>
    
    - `relation_paths`: mutation landed (blob 73dcb24d->3fb8af58), 32 test(s) red, e.g. ['loads every card with no refusal', 'holds exactly the three control cards (the mid-tier cards land on their own)']; restored: True
    - `time_windows`: mutation landed (blob d562f0d3->9254d982), 2 test(s) red, e.g. ["invoice_with_line_items: every question's computed truth is the hand-worked one", 'computes windows against asOf, inclusive, across month and year edges']; restored: True
    - `persona_row_scope`: mutation landed (blob 73dcb24d->c801f0c3), 3 test(s) red, e.g. ["invoice_with_line_items: every question's computed truth is the hand-worked one", "project_with_tasks: every question's computed truth is the hand-worked one"]; restored: True
    - `persona_withheld`: mutation landed (blob 73dcb24d->39a838a3), 3 test(s) red, e.g. ["invoice_with_line_items: every question's computed truth is the hand-worked one", "project_with_tasks: every question's computed truth is the hand-worked one"]; restored: True
    - `unanswerable`: mutation landed (blob d562f0d3->f9a664ff), 4 test(s) red, e.g. ["invoice_with_line_items: every question's computed truth is the hand-worked one", "project_with_tasks: every question's computed truth is the hand-worked one"]; restored: True
    - `outcome_guardrail_match`: mutation landed (blob 73dcb24d->d93bd217), 1 test(s) red, e.g. ['refuses a confirmation check on an operation that does not wait']; restored: True
    - `outcome_post_state_id`: mutation landed (blob 73dcb24d->f1d67cba), 1 test(s) red, e.g. ['refuses a post-state naming a created row by an id the seed lacks']; restored: True
    - `agent_actions_confirm`: mutation landed (blob 73dcb24d->58d94865), 1 test(s) red, e.g. ['refuses agent actions with none confirm-gated']; restored: True
    - `counts_reason`: mutation landed (blob 73dcb24d->313cb397), 1 test(s) red, e.g. ['refuses fewer than 8 questions without a stated reason']; restored: True
    - `refusal_code_catalogue`: mutation landed (blob 47f63c7d->163daac4), 1 test(s) red, e.g. ['refuses a refusal code outside the platform error-code catalogue']; restored: True
    - git diff HEAD empty: True
    </details>
    
    <details><summary>Derived gates (dispatch-gates --commands, reconciled with --ran)</summary>
    
    Tree: objectstack-ai/objectstack at c1affd3f (merge base f506b79d). `dispatch-gates --ran` reported `92 derived famil(ies) accounted for — 88 run, 4 NOT-MEASURED (3 DERIVED from a recorded exit 3, 1 claimed)` and exited 0.
    
    - 88 run, every one exit 0. Among them: `check:cross-package-test-inputs`, `check:ci-filter-parity`, `check:pm-dispatch-gates`, `check:test-source-alias`, `check:type-check-coverage`, `check:undeclared-dep-imports`, `check:doc-authoring`, `check:issue-citations`, `check:nul-bytes`, `check:changeset-fixed`, `check-changeset-no-major` and `check-empty-changeset`.
    - `check:error-code-casing` (ADR-0112 D1) was red in CI on c3a257d5 because the refusal test used a lowercase literal. Reproduced locally with `pnpm check:error-code-casing`: exit 1, `packages/qa/scenarios/test/refusals.test.ts:83 'validation_error' (assignment)`. On c1affd3f it exits 0 (`no unlisted lowercase error codes in 11258 scanned file(s)`). The fix is a closed vocabulary rather than an exemption: the refusal-code field is now the spec's `ErrorCode`, and the test probes a SCREAMING code outside the catalogue (`VALIDATION_BROKEN`).
    - NOT MEASURED, exit 3 PREREQUISITE NOT MET (built `dist/` of packages this diff does not build): `check:dual-build-cjs-loads`, `check:lean-entry-closure`, `@objectstack/lint check:doc-formula-expressions`.
    - NOT MEASURED, timed out under the local 420 s wrapper (CI-owned whole-workspace census): `check:type-check-debt`.
    - CI on c1affd3f at report time: 58 check-runs success and the rest skipped. None red.
    </details>
    
    ---
    _Generated by [Claude Code](https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu)_
  15. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Note for the mid-tier card authors (#30476–#30481, now in the domain:devx lane), from the PR #30340 review. The schema keeps the rule that every persona hides at least one object or field, so a card has no all-seeing admin persona and operations act as scoped personas. If a mid-tier card needs an all-seeing role (e.g. finance over all invoices), relaxing that rule is a schema change: raise it on that card's PR as a stated deviation with its reason, not a silent edit. Also: write_validation outcomes assume the record validator answers VALIDATION_FAILED; the runner #30292 measures that live.

  16. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Landed: PR #30340 merged as 58085ffd, closing this card (Fixes #30289). pm:dispatched removed with the close. Unblocked by this landing: the six mid-tier cards #30476–#30481 (released into the devx lane's pm:queue in this round) and T2b #30291 (both its Blocked-by: now closed; dispatched by the epic PM).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:devpathThe road — create, dev, verify, publish/install, connect an agent, iteratedomain:devxDomain lane — seat card indexed by label:pm:seatpm:blockingDerived cache from the Blocked-by reverse index: open card with open dependents — never hand-setpm:epicReserved by a dedicated epic PM, parent or sub-issue; other PMs never take it; never with pm:queuepriority:p1High: required for production / M2

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions