fix(service-analytics)!: the analytics where door refuses a non-boolean $null / $exists flag, which it read as IS NOT NULL (#20040) - #20115

Fixes #20040

Clause-②: no (narrowing)

What this changes

The analytics where door (lowerAnalyticsWhere in packages/services/service-analytics/src/strategies/filter-normalizer.ts) now refuses a $null or $exists flag whose value is not a boolean. It refuses before any lowering, in the door's INVALID_FILTER / 400 envelope, with the message kept.

• FieldOperatorsSchema declares both flags z.boolean(). The $null 的比较值不是布尔时,driver-sql 与 driver-memory 给出**完全相反**的答案(一个 IS NULL,一个 IS NOT NULL)—— 实测 #5347 / $exists 的比较值不是布尔时,三个后端面给出三个答案,driver-memory 自己的两个面在 'yes' 上就已分叉 —— 实测,$null(#5347)的同族另一轴 #5369 rulings refuse a non-boolean flag in every position and on every backend, because the backends read one in opposite directions. driver-sql refuses it (nonBooleanNullComparandError / nonBooleanExistsComparandError), and so does the read-scope compiler (read-scope-sql 的 $null / $exists 按真值性读比较数 —— {$null: "false"} 编成 IS NULL，#5347 / #5369 的先例没推到 RLS 编译器 #6387, assertBooleanFlagComparands).
• This door read the flag by identity (=== true for $null, === false for $exists), so every other value lowered to set, IS NOT NULL.

How:

• normalizeWhereComparands gains a third pass, assertBooleanNullFlags, after the shared comparand-shape face and the shared comparand-type face.
  • It walks their traversal (forEachWhereFieldEntry), so it finds a flag in every position they judge: $and / $or members, $not, field entries, and a nested relation's entries.
  • It reports a flag at the path those faces give it (where.acct.stage.$null).
• The message names the operator, the field and the path: [analytics] Operator "$null" on field "stage" requires a boolean comparand (true or false). Received a string ("x") at where.stage.$null. …
  • Its main clause is driver-sql's, word for word through "(true or false)".
  • The rest says what this door used to do and how to repair it. It carries no tracker number.
• Reuse. @objectstack/spec, core and types publish no predicate or sentence for this refusal (searched; none exists).
  • So the one check lives in this file, and its docblock cites $null 的比较值不是布尔时,driver-sql 与 driver-memory 给出**完全相反**的答案(一个 IS NULL,一个 IS NOT NULL)—— 实测 #5347 / $exists 的比较值不是布尔时,三个后端面给出三个答案,driver-memory 自己的两个面在 'yes' 上就已分叉 —— 实测,$null(#5347)的同族另一轴 #5369 and the sibling faces.
  • It imports no driver and adds no spec export.
  • The received value is rendered with this package's own shapePreview (comparand-shape.ts).
• true / false lower exactly as before.
• The FilterArray spelling needs no pass (measured, below):
  • parseFilterAST writes a hard-coded boolean for is_null / is_not_null, and refuses $null / exists as array operators;
  • isFilterAST refuses an embedded object.

The order chosen, and why

• After the shape and type faces. A flag the type face refuses keeps the face's sentence: undefined, a plain object, a Map, a bigint beyond 2^53. That is the sentence the FilterArray spelling and the engine seam give. A shape or type defect elsewhere in the same where is still answered first. Ablation C pins this order.

• Before the undefined gate (service-analytics 的 where 门把 undefined 值的键整个丢掉 —— 单键 where 退化成「无过滤器」，方向是加宽（#6125 五面表漏记的第六、七种读法） #6386) and everything else buildNode reaches. Two readers see the flag before the identity read in fieldLeaves:

  • the $not 的语义在 driver-sql 与 driver-memory / formula 之间分叉：NULL 行的去留相反，$not: {} 一个是 TRUE 一个是 FALSE #5146 $not rewrite, which classifies every field spec through nullValueSatisfiesOperator / operatorIsNullTotal (the flag readers at the old filter-normalizer.ts:1411 / :1464);
  • the draft preview, which evaluates what normalizeWhereComparands returns and never reaches fieldLeaves.

  The undefined gate skips both flags by name, so the two gates never judge one value.

• One precedence change, same envelope. A where carrying a non-boolean flag and a defect only the lowering diagnoses is now answered with the flag. Those defects are a mixed $ / non-$ wrapper (service-analytics 的 where 门：字段约束里 $ 算子与非 $ 键混写时，非 $ 兄弟键被静默丢掉（方向是加宽；与值无关，不是 #6386 的 undefined） #6444), an operator outside the vocabulary, and a zero-operator constraint. The code and status stay INVALID_FILTER / 400. service-analytics: the object-form analytics where skips the shared comparand-TYPE face, so a plain-object / Map / oversized-bigint comparand is bound as JSON text on the native path while the FilterArray spelling and the engine refuse 400 #20035 took the same precedence for the type face over service-analytics 的 where 门：字段约束里 $ 算子与非 $ 键混写时，非 $ 兄弟键被静默丢掉（方向是加宽；与值无关，不是 #6386 的 undefined） #6444. It is pinned, with boolean controls that keep the old sentences.

• The preview. Before the fix it refused every flag ("not evaluated", 400) while the published faces served the cell, so the two answered it differently. It reaches the new pass through normalizeWhereComparands and now refuses this cell in the door's words, byte for byte. preview-evaluator.ts is not edited. A boolean flag still gets the preview's own "not evaluated" refusal.

Measured first (recorded on this branch as 12e3081867, test-only, before any source change)

Base origin/main 8d76c2d38c.

• Data. A real sql.js engine (driver-sqlite-wasm) over rows r1 'won', r2 NULL and r3 'lost' (field stage), with a real ObjectQL behind the ObjectQL face. The echo's SQL was run on the same database.
• HTTP legs. Read and measured only, through throwaway harnesses (not committed) over a real AnalyticsService:
  • packages/runtime's dispatcher route POST /api/v1/analytics/query, native and ObjectQL-only;
  • packages/rest's POST /analytics/dataset/query, with the flag in selection.runtimeFilter and in an inline dataset.filter.

• Positions. The top level, $and, $or, $not, and $not beside $ne; every coerced cell answered the same in each. { $not: { stage: { $null: 'true' } } } served the NULL row the author excluded.
• $null: null / $exists: null are reachable, in process and over HTTP (JSON null). Both read as IS NOT NULL and answered 200:
  • the undefined gate (service-analytics 的 where 门把 undefined 值的键整个丢掉 —— 单键 where 退化成「无过滤器」，方向是加宽（#6125 五面表漏记的第六、七种读法） #6386) skips both flags by name;
  • the type face accepts null, which is in its accepted set;
  • the shape face's null carve-outs cover $in / $nin members, $between endpoints and ordering comparands, not the flags.
• The card's HTTP 200 is confirmed on main. FilterConditionSchema types a field entry as z.unknown(), so the body parse admitted every non-boolean flag on both routes.
• The FilterArray spelling carries no flag value.
  • ['stage', 'is_null', 'x'] gives notSet and ['stage', 'isnotnull', 0] gives set; the third member is filler.
  • ['stage', '$null', 'x'], 'null', 'exists' and '$exists' are refused as not a filter.
  • isFilterAST answers false for every array embedding an object node (5 shapes probed).
• Other analytics paths through the same lowering (the card's check):
  • dataset scope filters and measure filters reach it through the strategies' normalizeAnalyticsFilterTree calls;
  • the preview reaches it through normalizeWhereComparands;
  • the read-scope door does not use it.

After (head 66266c2a3b)

• In process. Every non-boolean row above is refused INVALID_FILTER / 400 on all seven in-process columns, the preview included, in the door's sentence (44 of 44 rows). Nothing reaches the database or the engine: 0 statements and 0 engine.aggregate calls.

• HTTP re-measure. Both routes answer 400 INVALID_FILTER for every non-boolean cell, with the message kept and 0 statements. The controls still answer 200 with the same rows.

• Controls, byte for byte. For $null / $exists × true / false, at the top level, under $not, and under $not beside $ne, I captured:

  • the compiled tree;
  • the native SQL and its params;
  • the echo SQL and its params;
  • the engine where.

  The capture taken on 8d76c2d38c and the one taken after the fix are byte-identical (10,328 bytes, the same sha256). The test pins them as literals.

• The read scope keeps its own answer, untouched:

  • on native, the echo and AnalyticsService.query (native): READ_SCOPE_COMPILE_FAILED / 500, withheld, in read-scope-sql's sentence;
  • on the ObjectQL face, driver-sql answers it: INVALID_FILTER / 400, with the policy content withheld.

Tests

New file packages/services/service-analytics/src/__tests__/where-boolean-flag-refusal.test.ts, 63 cases. Every refusal asserts code + status.

• the door rows: 2 flags × 10 non-boolean values;
• every position, the nested-relation path, and the received value named in the message;
• the FilterArray control;
• existing refusals keeping their order and sentence;
• the measured precedence against the lowering's own refusals, with boolean controls;
• every face over a real engine, with statement and engine-call counters;
• the preview, whose message is byte-equal to the door's;
• the byte-for-byte controls;
• the read-scope face on every sub-face, plus compileScopedFilterToSql;
• stored dataset and measure filters through queryDataset.

Pins re-judged, none deleted.

• The census was taken before the change, over service-analytics, packages/rest and packages/runtime, plus a repo-wide test scan.
• rest and runtime had 0 hits asserting this door's admission. rest-4xx-message-truncation.test.ts copies a driver-sql message and is not this door.
• One pin in filter-normalizer-undefined-comparand.test.ts asserted the admission: the case "$null / $exists: an undefined flag is refused by the type face; an accepted flag value reads as before".
  • It asserted { $null: 'false' } and { $exists: 'yes' } lower to set.
  • Flipped to refusals, with a note. The undefined half is unchanged, and the case title now says what it asserts.

Runs.

• Test-only commit 12e3081867, before the fix: 53 failed | 10 passed (63).
  • Every door row was red.
  • Green: the four controls, the type-face and shape-face order pins, the FilterArray control and the three read-scope pins.
• Head 66266c2a3b:
  • pnpm --filter @objectstack/service-analytics suite: 125 files, 2949 tests passed;
  • typecheck: exit 0, and tsc --listFiles includes both touched test files;
  • consumer sanity run against the rebuilt dist, not a pin: packages/rest's 10 analytics route suites (148 passed) and packages/runtime's 3 analytics suites (36 passed).

Ablations

Each ran through scripts/ablation-replace.mjs from committed state, with the anchor hit ×1 and the restore proven (blob == HEAD, git diff HEAD empty). service-analytics tests import the subject by relative src paths, so no dist is in the path, except for leg F.

Gates (head 66266c2a3b)

• Derivation. node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands derived 60 families from the real diff. That is 6 more than the dispatch-time list: check:engine-double-contract, check:objectql-double-limit, check:query-options-erasure, check:type-check-coverage, check:type-check-debt and check:where-matcher.
• 60 of 60 exit 0. Among them:
  • check-adr-0087-registration --base origin/main, with the disposition not-required (no-migration-prescription);
  • check-changeset-no-major, check-empty-changeset and check:nul-bytes;
  • check:issue-citations, which runs only its self-test. The real run (GITHUB_TOKEN="$GH_TOKEN" node scripts/check-issue-citations.mjs) judged 29 citations, and all resolve.
• The two built-output gates. check:dual-build-cjs-loads and check:type-check-debt read built output. On the first head, before any full build, both answered PREREQUISITE NOT MET, which is not a failure. They ran green after the lint.yml build command (turbo run build --filter='./packages/*' --filter='./packages/*/*').
• Reconciliation. --ran: 60 derived, 60 run, 0 NOT MEASURED, 0 UNRUN. Every entry carries its exit code.
• Merges. main moved twice while this ran: aa04ea2964 (objectql) and fa00ebf447 (runtime), both disjoint from this diff. Both were merged, never rebased, and the whole union was re-run on the final head.
• Lint, narrowed and proven.
  • eslint --no-inline-config --format json over the 3 changed .ts files: 3 files, 0 errors, 0 warnings.
  • isPathIgnored is false for all three. The changeset .md is ignored and has no config, so it is outside the population.
  • eslint.config.mjs never enables type-aware linting (no parserOptions.project and no projectService, as its line 327 states). So this diff cannot move the verdict of any untouched file.

Changeset

.changeset/20040-analytics-null-flag.md: minor with ! and a Clause-②: no (narrowing) line, following the #20035 / #20068 precedents.

• ADR-0087 marker. It is an HTML-comment line, described here in words so this body keeps it. It declares not-required (no-migration-prescription), and the gate accepts it.
  • No ledger entry names the flags' boolean domain: the $null 的比较值不是布尔时,driver-sql 与 driver-memory 给出**完全相反**的答案(一个 IS NULL,一个 IS NOT NULL)—— 实测 #5347 / $exists 的比较值不是布尔时,三个后端面给出三个答案,driver-memory 自己的两个面在 'yes' 上就已分叉 —— 实测,$null(#5347)的同族另一轴 #5369 refusals in driver-sql and the read-scope-sql 的 $null / $exists 按真值性读比较数 —— {$null: "false"} 编成 IS NULL，#5347 / #5369 的先例没推到 RLS 编译器 #6387 refusal were never registered.
  • A non-boolean flag has no boolean it can be mechanically rewritten to.
• Who is affected. A text scan over 4598 tracked non-test files found 22 matches for a non-boolean flag. 21 are code comments; the 22nd is an operator-name lookup table in driver-memory, not a filter. Stored datasets, widget filters and report runtime filters in a deployment were NOT measured.

Acceptance notes

• The preview still refuses a BOOLEAN flag ("not evaluated"). It evaluates no $null / $exists at all (the [finding] the analytics draft-preview matcher answers every row for $icontains, $startsWith, $endsWith and other operators it has no case for — a drafted chart ignores those filters, then changes at publish #19810 table), and that is unchanged and not this card's to change. Widening that table is ruled to land together with the conformance kits. A drafted chart carrying { $null: 'x' } now gets the flag refusal first. Once repaired to true, it gets the preview's own refusal, while publish serves it.

Out-of-scope findings (reported to the seat, not filed here)

1. Class (c). The save door admits a non-boolean flag that every query face refuses.
   • Measured at 66266c2a3b:
     • FilterConditionSchema.safeParse({ stage: { $null: 'x' } }) succeeds;
     • DatasetSchema.safeParse with filter: { stage: { $null: 'x' } } (also with $exists: 'false' and with $null: null) succeeds;
     • the same save door refuses $icontains: '' (finding(spec): ViewFilterRuleSchema accepts two shapes every consumer refuses, and ObjectGridProps.defaultFilters is z.unknown() so nothing gates it at all — the protocol half of objectui#9050's ruling C′ #19514).
   • So a stored dataset or widget filter carrying the shape saves, and then answers 400 on every query face: driver-sql since $null 的比较值不是布尔时,driver-sql 与 driver-memory 给出**完全相反**的答案(一个 IS NULL,一个 IS NOT NULL)—— 实测 #5347, the analytics door since this PR.
   • The named producer is whoever authors a dataset filter: defineStack metadata or the Studio.
   • Seam: spec:FilterConditionSchema (checkFilterConditionComparands) → runtime:driver-sql reduceFilterKey | service-analytics normalizeWhereComparands
   • Dedupe words: FilterConditionSchema $null non-boolean admitted at save · dataset filter $exists string saves then 400 · checkFilterConditionComparands boolean flag

Generated by Claude Code