diff --git a/.changeset/22183-import-kept-option-values.md b/.changeset/22183-import-kept-option-values.md new file mode 100644 index 00000000000..4f85d7cab63 --- /dev/null +++ b/.changeset/22183-import-kept-option-values.md @@ -0,0 +1,16 @@ +--- +"@objectstack/spec": minor +"@objectstack/core": minor +"@objectstack/objectql": minor +--- + +fix(import): `createMissingOptions` now keeps an unmatched option value through the engine's option check + +Clause-②: yes (widening) + +- **What was wrong.** With `createMissingOptions: true` on an import (objectui's Import Wizard sends it for *Keep unknown option values*), the import coercion kept a `select` / `radio` / `multiselect` cell that matched none of the field's options. The engine's write-path option check then refused the same value ("Priority must be one of: high, low"), so on every writable field the row still failed. Only the stage that refused it changed. +- **What happens now.** The import stores the kept value: a single-value cell trimmed, and each unmatched item of a multi-value cell. The dry run admits the same rows the write does. The engine admits exactly the values the import kept, on that import's writes. Every other value outside the options is still refused, and a field bound to a shared picklist that did not resolve is still refused. +- **The field's options do not change.** No option is added to the field. A later write that sends the field is judged against the options again, so sending the kept value back is refused and picking an option is accepted. A write that leaves the field out, such as an edit form saving only the fields it changed, is not judged on that field. +- **`@objectstack/spec`.** `ExecutionContext` declares `keptOptionValues`: per field name, the option values an import kept. It is server-constructed only, like `skipStateMachine` and `preserveAudit`, and is never client-supplied. The `ImportRequest.createMissingOptions` description now states exactly the behaviour above. +- **`@objectstack/core`.** `coerceRow` returns `keptOptionValues` beside `data` and `errors`, and `coerceFieldValue` reports the values it kept. `runImport` puts exactly those values on each write's context. A per-row write carries its own row's values, and a batched create carries the union of its rows' values. A row that kept nothing is written with the import's context unchanged. +- **`@objectstack/objectql`.** `ValidateRecordOptions.keptOptionValues` lets the single-value and multi-value option arms admit a listed value. The engine fills it from the write's `ExecutionContext.keptOptionValues` on insert, update and `validate`. diff --git a/content/docs/references/api/export.mdx b/content/docs/references/api/export.mdx index 44b5aebccfb..e10a91a8e00 100644 --- a/content/docs/references/api/export.mdx +++ b/content/docs/references/api/export.mdx @@ -58,7 +58,7 @@ const result = CreateImportJobRequestSchema.parse(data); | **treatAsHistorical** | `boolean` | optional (default: `false`) | Import as established historical facts. Two effects, both off by default so a normal import is unchanged: (1) skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates; and (2) preserve the original audit timeline — keep the supplied created_at / updated_at / updated_by and author-declared business readonly fields (e.g. closed_at, resolved_by) instead of stamping-now / stripping them. Undoing a historical import mirrors (2): the captured pre-import values are restored verbatim rather than re-stamped. | | **trimWhitespace** | `boolean` | optional (default: `true`) | Trim leading/trailing whitespace from string cells | | **nullValues** | `string[]` | optional | Strings treated as null/blank (besides empty string) | -| **createMissingOptions** | `boolean` | optional (default: `false`) | Keep unmatched select values instead of failing the row | +| **createMissingOptions** | `boolean` | optional (default: `false`) | Keep a select / radio / multiselect cell that matches none of the field's options instead of failing the row. The cell is stored as written (trimmed; for a multi-value cell, each unmatched item), and the engine's option check admits exactly the values this import kept, on this import's writes (dry run included). The field's option list is not changed: a later write that sends the field is judged against the options again, while one that leaves the field out is not affected. A field bound to a shared picklist that did not resolve is still refused. | | **skipBlankMatchKey** | `boolean` | optional (default: `false`) | Skip rows whose matchFields are blank (default: upsert creates them, update skips them) | ### Nested Shape: `CreateImportJobRequest.mapping[number]` @@ -312,7 +312,7 @@ Type: `{ sourceField: string; targetField: string; targetLabel?: string; transfo | **treatAsHistorical** | `boolean` | optional (default: `false`) | Import as established historical facts. Two effects, both off by default so a normal import is unchanged: (1) skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates; and (2) preserve the original audit timeline — keep the supplied created_at / updated_at / updated_by and author-declared business readonly fields (e.g. closed_at, resolved_by) instead of stamping-now / stripping them. Undoing a historical import mirrors (2): the captured pre-import values are restored verbatim rather than re-stamped. | | **trimWhitespace** | `boolean` | optional (default: `true`) | Trim leading/trailing whitespace from string cells | | **nullValues** | `string[]` | optional | Strings treated as null/blank (besides empty string) | -| **createMissingOptions** | `boolean` | optional (default: `false`) | Keep unmatched select values instead of failing the row | +| **createMissingOptions** | `boolean` | optional (default: `false`) | Keep a select / radio / multiselect cell that matches none of the field's options instead of failing the row. The cell is stored as written (trimmed; for a multi-value cell, each unmatched item), and the engine's option check admits exactly the values this import kept, on this import's writes (dry run included). The field's option list is not changed: a later write that sends the field is judged against the options again, while one that leaves the field out is not affected. A field bound to a shared picklist that did not resolve is still refused. | | **skipBlankMatchKey** | `boolean` | optional (default: `false`) | Skip rows whose matchFields are blank (default: upsert creates them, update skips them) | ### Nested Shape: `ImportRequest.mapping[number]` diff --git a/content/docs/references/data/data-engine.mdx b/content/docs/references/data/data-engine.mdx index c170512b420..f438ccfa8d8 100644 --- a/content/docs/references/data/data-engine.mdx +++ b/content/docs/references/data/data-engine.mdx @@ -71,6 +71,7 @@ const result = BaseEngineOptionsSchema.parse(data); | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | @@ -124,6 +125,7 @@ Options for DataEngine.aggregate operations | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | @@ -202,6 +204,7 @@ Options for DataEngine.count operations | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | @@ -275,6 +278,7 @@ Options for DataEngine.delete operations | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | @@ -455,6 +459,7 @@ Options for DataEngine.insert operations | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | @@ -526,6 +531,7 @@ Query options for IDataEngine.find() operations | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | @@ -822,6 +828,7 @@ Options for DataEngine.update operations | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | @@ -937,6 +944,7 @@ QueryAST-aligned options for DataEngine.aggregate operations | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | @@ -1019,6 +1027,7 @@ QueryAST-aligned options for DataEngine.count operations | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | @@ -1071,6 +1080,7 @@ QueryAST-aligned options for DataEngine.delete operations | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | @@ -1132,6 +1142,7 @@ QueryAST-aligned query options for IDataEngine.find() operations | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | @@ -1221,6 +1232,7 @@ QueryAST-aligned options for DataEngine.update operations | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | diff --git a/content/docs/references/kernel/execution-context.mdx b/content/docs/references/kernel/execution-context.mdx index 898b44e3313..e62bde6e6a4 100644 --- a/content/docs/references/kernel/execution-context.mdx +++ b/content/docs/references/kernel/execution-context.mdx @@ -56,6 +56,7 @@ const result = ExecutionContextSchema.parse(data); | **seedReplay** | `boolean` | optional | | | **skipStateMachine** | `boolean` | optional | | | **preserveAudit** | `boolean` | optional | Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected. | +| **keptOptionValues** | `Record` | optional | Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field's options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field's options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied. | | **oauthScopes** | `string[]` | optional | | | **accessToken** | `string` | optional | | | **transaction** | `any` | optional | | diff --git a/packages/core/src/security/assemble-execution-context.test.ts b/packages/core/src/security/assemble-execution-context.test.ts index b4da636e83f..df5b90b104d 100644 --- a/packages/core/src/security/assemble-execution-context.test.ts +++ b/packages/core/src/security/assemble-execution-context.test.ts @@ -462,6 +462,7 @@ describe('#6216 — the field set is CLOSED', () => { 'seedReplay', 'skipStateMachine', 'preserveAudit', + 'keptOptionValues', ]; it('every ExecutionContext field is either assembled at the entry or declared non-entry', () => { diff --git a/packages/core/src/security/assemble-execution-context.ts b/packages/core/src/security/assemble-execution-context.ts index c779a5a9f8e..6bfd88409bf 100644 --- a/packages/core/src/security/assemble-execution-context.ts +++ b/packages/core/src/security/assemble-execution-context.ts @@ -73,8 +73,9 @@ import type { ResolvedAuthzContext } from './resolve-authz-context.js'; * - `rlsMembership` — engine-side RLS scoping cache. * - `transaction` / `traceId` — per-operation handles. * - `flowRunId`, `skipTriggers`, `skipAutomations`, `seedReplay`, - * `skipStateMachine`, `preserveAudit` — per-write behaviour flags, - * server-constructed at the call site. + * `skipStateMachine`, `preserveAudit`, `keptOptionValues` — per-write + * behaviour flags, server-constructed at the call site (`keptOptionValues` + * by the import runner, from the cells one import kept). */ type NonEntryExecutionContextField = | 'actor' @@ -87,7 +88,8 @@ type NonEntryExecutionContextField = | 'skipAutomations' | 'seedReplay' | 'skipStateMachine' - | 'preserveAudit'; + | 'preserveAudit' + | 'keptOptionValues'; /** * The CLOSED field set every transport entry point must decide. Derived from diff --git a/packages/core/src/utils/import-coerce.test.ts b/packages/core/src/utils/import-coerce.test.ts index 2b5dbe4f1f0..5dbc64bffa9 100644 --- a/packages/core/src/utils/import-coerce.test.ts +++ b/packages/core/src/utils/import-coerce.test.ts @@ -410,6 +410,34 @@ describe('coerceRow', () => { expect(lax.data).toEqual({ s: 'zzz' }); }); + // #22183 — the engine's option check admits a kept value only when the write + // names it, so coercion reports exactly what it KEPT, never what it matched. + it('reports the kept option values per field, and only the unmatched ones', async () => { + const metaMap = meta({ + s: { type: 'select', options: [{ label: 'A', value: 'a' }] }, + tags: { type: 'multiselect', options: [{ label: 'A', value: 'a' }, { label: 'B', value: 'b' }] }, + plain: { type: 'text' }, + }); + const lax = await coerceRow( + { s: ' zzz ', tags: 'B, qq; a, rr', plain: 'x' }, + metaMap, + { createMissingOptions: true }, + ); + expect(lax.errors).toEqual([]); + expect(lax.data).toEqual({ s: 'zzz', tags: ['b', 'qq', 'a', 'rr'], plain: 'x' }); + expect(lax.keptOptionValues).toEqual({ s: ['zzz'], tags: ['qq', 'rr'] }); + + // Every cell matched: nothing kept, so the write carries no admission. + const matched = await coerceRow({ s: 'A', tags: 'a; b' }, metaMap, { createMissingOptions: true }); + expect(matched.data).toEqual({ s: 'a', tags: ['a', 'b'] }); + expect(matched.keptOptionValues).toEqual({}); + + // The option off: the cell is refused at coercion and nothing is kept. + const strict = await coerceRow({ s: 'zzz' }, metaMap, {}); + expect(strict.errors.map((e) => e.code)).toEqual(['invalid_option']); + expect(strict.keptOptionValues).toEqual({}); + }); + it('passes unknown columns through untouched', async () => { const { data } = await coerceRow({ mystery: 'raw' }, new Map(), {}); expect(data).toEqual({ mystery: 'raw' }); diff --git a/packages/core/src/utils/import-coerce.ts b/packages/core/src/utils/import-coerce.ts index 91744bc7b28..0ea300c6a7e 100644 --- a/packages/core/src/utils/import-coerce.ts +++ b/packages/core/src/utils/import-coerce.ts @@ -119,8 +119,11 @@ export interface CoerceContext { nullValues?: string[]; /** * When a select/multiselect cell matches no known option, keep the raw value - * instead of failing. Note: the engine still validates option membership, so - * this only helps when the option is (or will be) present in the schema. + * (trimmed; each unmatched item of a multi-value cell) instead of failing. + * The kept values are reported on the result (`keptOptionValues`), and the + * import runner hands exactly those to the engine write as + * `ExecutionContext.keptOptionValues`, whose option check admits them on that + * write only (#22183). The field's option list is never changed. */ createMissingOptions?: boolean; /** Async reference resolver (name/email/id → record id). Optional. */ @@ -646,7 +649,7 @@ export async function coerceFieldValue( raw: unknown, meta: ExportFieldMeta | undefined, ctx: CoerceContext, -): Promise<{ value?: unknown } | { error: FieldCoerceError }> { +): Promise<{ value?: unknown; keptOptionValues?: string[] } | { error: FieldCoerceError }> { const trim = ctx.trimWhitespace !== false; const field = meta?.name ?? ''; @@ -691,22 +694,30 @@ export async function coerceFieldValue( // (multiselect/…) OR a select/radio is flagged `multiple: true`; split then // and match each token, else match the whole cell as one option. if (OPTION_TYPES.has(t) || MULTI_OPTION_TYPES.has(t)) { + // [#22183] A kept cell is reported as well as stored: the engine's option + // check refuses a value outside the options unless the write names it in + // `ExecutionContext.keptOptionValues`, and only coercion knows which values + // it kept rather than matched. if (isMultiValueField(meta)) { const parts = splitMulti(raw); const out: unknown[] = []; + const kept: string[] = []; for (const part of parts) { const v = matchOption(part, meta?.options); if (v === undefined) { - if (ctx.createMissingOptions) { out.push(part); continue; } + if (ctx.createMissingOptions) { out.push(part); kept.push(part); continue; } return coerceError(meta, field, 'invalid_option', 'import_unknown_option', part, ctx); } out.push(v); } - return { value: out }; + return kept.length > 0 ? { value: out, keptOptionValues: kept } : { value: out }; } const v = matchOption(raw, meta?.options); if (v === undefined) { - if (ctx.createMissingOptions) return { value: String(raw).trim() }; + if (ctx.createMissingOptions) { + const keptValue = String(raw).trim(); + return { value: keptValue, keptOptionValues: [keptValue] }; + } return coerceError(meta, field, 'invalid_option', 'import_unknown_option', raw, ctx); } return { value: v }; @@ -797,9 +808,20 @@ export async function coerceRow( rawRow: Record, metaMap: Map, ctx: CoerceContext, -): Promise<{ data: Record; errors: FieldCoerceError[] }> { +): Promise<{ + data: Record; + errors: FieldCoerceError[]; + /** + * [#22183] Per field, the option values `createMissingOptions` kept on this + * row — empty unless that option is on and a cell matched no option. The + * import runner hands it to the row's write as + * `ExecutionContext.keptOptionValues`. + */ + keptOptionValues: Record; +}> { const data: Record = {}; const errors: FieldCoerceError[] = []; + const keptOptionValues: Record = {}; for (const [key, raw] of Object.entries(rawRow)) { const meta = metaMap.get(key); const res = await coerceFieldValue(raw, meta ? meta : undefined, ctx); @@ -809,6 +831,7 @@ export async function coerceRow( continue; } if (res.value !== undefined) data[key] = res.value; + if (res.keptOptionValues && res.keptOptionValues.length > 0) keptOptionValues[key] = res.keptOptionValues; } - return { data, errors }; + return { data, errors, keptOptionValues }; } diff --git a/packages/core/src/utils/import-runner-kept-options.test.ts b/packages/core/src/utils/import-runner-kept-options.test.ts new file mode 100644 index 00000000000..80e5df0b19a --- /dev/null +++ b/packages/core/src/utils/import-runner-kept-options.test.ts @@ -0,0 +1,156 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #22183 — `createMissingOptions` reaches the engine write. + * + * The coercion keeps a select / multiselect cell that matches no option, and + * the engine's option check refuses a value outside the options unless the + * write names it in `ExecutionContext.keptOptionValues`. runImport owns the + * hand-over: every write call (and the dry run's `validateData`) carries + * exactly the values the rows it writes kept, and a call whose rows kept + * nothing carries the import's write context unchanged. + * + * The engine's half (the option arms admit a listed value, and only it) is + * pinned in @objectstack/objectql (`import-kept-option-values.test.ts`, which + * also drives this runner through the real engine); this file pins the wiring. + */ + +import { describe, it, expect, vi } from 'vitest'; +import { runImport, type ImportProtocolLike } from './import-runner'; +import type { ExportFieldMeta } from './import-field-meta.js'; + +type CreateArgs = Parameters[0]; +type UpdateArgs = Parameters[0]; +type CreateManyArgs = Parameters>[0]; +type InsertManyArgs = Parameters>[0]; +type ValidateArgs = Parameters>[0]; + +const metaMap = new Map([ + ['code', { name: 'code', type: 'text' }], + ['priority', { name: 'priority', type: 'select', options: [{ label: 'High', value: 'high' }, { label: 'Low', value: 'low' }] }], + ['tags', { name: 'tags', type: 'multiselect', options: [{ label: 'Important', value: 'important' }] }], +]); + +const baseOpts = { + objectName: 'ticket', + metaMap, + writeMode: 'insert' as const, + matchFields: [] as string[], + dryRun: false, + runAutomations: false, + trimWhitespace: true, + createMissingOptions: true, + skipBlankMatchKey: false, +}; + +/** Records `{ kind, rows, context }` for every call that writes or previews. */ +function makeProvider(shape: 'inline' | 'createMany' | 'insertMany' | 'update' | 'validate') { + const calls: Array<{ kind: string; rows: Array>; context: any }> = []; + let idc = 0; + const p: ImportProtocolLike = { + findData: vi.fn(async () => (shape === 'update' ? [{ id: 'existing', code: 'k' }] : [])), + createData: vi.fn(async (args: CreateArgs) => { + calls.push({ kind: 'createData', rows: [args.data as Record], context: args.context }); + return { id: `d${++idc}`, ...(args.data as object) }; + }), + updateData: vi.fn(async (args: UpdateArgs) => { + calls.push({ kind: 'updateData', rows: [args.data as Record], context: args.context }); + return { id: args.id, ...(args.data as object) }; + }), + }; + if (shape === 'createMany') { + p.createManyData = vi.fn(async (args: CreateManyArgs) => { + calls.push({ kind: 'createManyData', rows: args.records as Array>, context: args.context }); + return { records: args.records.map((r: any) => ({ id: `d${++idc}`, ...r })) }; + }); + } + if (shape === 'insertMany') { + p.insertManyData = vi.fn(async (args: InsertManyArgs) => { + calls.push({ kind: 'insertManyData', rows: args.records as Array>, context: args.context }); + return { outcomes: args.records.map((r: any) => ({ ok: true, record: { id: `d${++idc}`, ...r } })) } as any; + }); + // `bulkWrite` needs the all-or-nothing primitive beside the partial one. + p.createManyData = vi.fn(async (args: CreateManyArgs) => { + calls.push({ kind: 'createManyData', rows: args.records as Array>, context: args.context }); + return { records: args.records.map((r: any) => ({ id: `d${++idc}`, ...r })) }; + }); + } + if (shape === 'validate') { + p.validateData = vi.fn(async (args: ValidateArgs) => { + calls.push({ kind: 'validateData', rows: [args.data as Record], context: args.context }); + return { results: [{ valid: true, errors: [] }] } as any; + }); + } + return { p, calls }; +} + +const ROWS = [ + { code: 'a', priority: 'Bogus' }, // kept: priority + { code: 'b', priority: 'High' }, // matched: nothing kept + { code: 'c', tags: 'Important, Other' }, // kept: tags +]; + +describe('runImport — createMissingOptions hands the kept values to the write (#22183)', () => { + it('a per-row create carries exactly that row\'s kept values, and a row that kept nothing carries none', async () => { + const { p, calls } = makeProvider('inline'); + const summary = await runImport({ ...baseOpts, p, rows: ROWS, context: { userId: 'u1' } }); + expect(summary.created).toBe(3); + expect(calls.map((c) => c.kind)).toEqual(['createData', 'createData', 'createData']); + expect(calls[0].context.keptOptionValues).toEqual({ priority: ['Bogus'] }); + expect(calls[1].context).not.toHaveProperty('keptOptionValues'); + expect(calls[2].context.keptOptionValues).toEqual({ tags: ['Other'] }); + // The rest of the write context is the import's own, untouched. + for (const c of calls) { + expect(c.context.userId).toBe('u1'); + expect(c.context.skipAutomations).toBe(true); + } + expect(calls[0].rows[0]).toMatchObject({ priority: 'Bogus' }); + expect(calls[2].rows[0]).toMatchObject({ tags: ['important', 'Other'] }); + }); + + it('a batched create (createManyData) carries the union of its rows\' kept values', async () => { + const { p, calls } = makeProvider('createMany'); + const summary = await runImport({ ...baseOpts, p, rows: [...ROWS, { code: 'd', priority: 'Bogus' }] }); + expect(summary.created).toBe(4); + expect(calls.map((c) => c.kind)).toEqual(['createManyData']); + expect(calls[0].rows).toHaveLength(4); + expect(calls[0].context.keptOptionValues).toEqual({ priority: ['Bogus'], tags: ['Other'] }); + }); + + it('a partial-success batch (insertManyData) carries the union of its rows\' kept values', async () => { + const { p, calls } = makeProvider('insertMany'); + const summary = await runImport({ ...baseOpts, p, rows: ROWS }); + expect(summary.created).toBe(3); + expect(calls.map((c) => c.kind)).toEqual(['insertManyData']); + expect(calls[0].context.keptOptionValues).toEqual({ priority: ['Bogus'], tags: ['Other'] }); + }); + + it('an update carries the row\'s kept values', async () => { + const { p, calls } = makeProvider('update'); + const summary = await runImport({ + ...baseOpts, p, writeMode: 'update', matchFields: ['code'], rows: [{ code: 'k', priority: 'Bogus' }], + }); + expect(summary.updated).toBe(1); + expect(calls.map((c) => c.kind)).toEqual(['updateData']); + expect(calls[0].context.keptOptionValues).toEqual({ priority: ['Bogus'] }); + }); + + it('the dry run asks validateData with the same per-row context the write would carry', async () => { + const { p, calls } = makeProvider('validate'); + const summary = await runImport({ ...baseOpts, p, dryRun: true, rows: ROWS }); + expect(summary.created).toBe(3); + expect(calls.map((c) => c.kind)).toEqual(['validateData', 'validateData', 'validateData']); + expect(calls[0].context.keptOptionValues).toEqual({ priority: ['Bogus'] }); + expect(calls[1].context).not.toHaveProperty('keptOptionValues'); + expect(calls[2].context.keptOptionValues).toEqual({ tags: ['Other'] }); + }); + + it('with the option off, an unmatched cell fails at coercion and no write carries the key', async () => { + const { p, calls } = makeProvider('inline'); + const summary = await runImport({ ...baseOpts, p, createMissingOptions: false, rows: ROWS }); + expect(summary.created).toBe(1); + expect(summary.results.filter((r) => !r.ok).map((r) => r.code)).toEqual(['invalid_option', 'invalid_option']); + expect(calls).toHaveLength(1); + expect(calls[0].context).not.toHaveProperty('keptOptionValues'); + }); +}); diff --git a/packages/core/src/utils/import-runner.ts b/packages/core/src/utils/import-runner.ts index 10ea4ddf19e..fc8e26c964d 100644 --- a/packages/core/src/utils/import-runner.ts +++ b/packages/core/src/utils/import-runner.ts @@ -5,6 +5,7 @@ import { coerceRow, type RefResolver, type RefMatch } from './import-coerce.js'; import type { ExportFieldMeta } from './import-field-meta.js'; import type { ValidationMessageTranslator } from '@objectstack/spec/system'; import type { DroppedFieldsEvent } from '@objectstack/spec/data'; +import type { ExecutionContext } from '@objectstack/spec/kernel'; import type { CreateDataRequest, FindDataRequest, UpdateDataRequest, ValidateDataIssue, ValidateDataRequest, ValidateDataResponse } from '@objectstack/spec/api'; import { bulkWrite, withTransientRetry, defaultIsTransientError, type BulkWriteRowResult } from './bulk-write.js'; import { isUniqueViolationError, uniqueViolationColumn, isEngineDuplicateRecordEnvelope, mapDataError } from '@objectstack/types'; @@ -232,6 +233,9 @@ export interface RunImportOptions { treatAsHistorical?: boolean; trimWhitespace: boolean; nullValues?: string[]; + /** #22183 — keep a select / multiselect cell that matches no option. Each + * write carries the values its rows kept as `keptOptionValues` on the write + * context, the one signal the engine's option check admits them by. */ createMissingOptions: boolean; skipBlankMatchKey: boolean; /** @@ -666,6 +670,27 @@ export function runImport(opts: RunImportOptions): Promise { ...(treatAsHistorical ? { skipStateMachine: true, preserveAudit: true } : {}), }; + /** + * [#22183] The write context for rows whose coercion KEPT option values + * (`createMissingOptions`): {@link writeCtx} plus exactly those values, per + * field, as `keptOptionValues` — the declared signal the engine's option + * check admits them by, on that write only. A batch write carries the union + * of its rows' values, because one call carries one context. Rows that kept + * nothing (every row, when the option is off) get `writeCtx` itself, so a + * normal import's writes are unchanged. + */ + const writeCtxFor = (kept: ReadonlyArray | undefined>) => { + const union: NonNullable = {}; + for (const rowKept of kept) { + if (!rowKept) continue; + for (const [field, values] of Object.entries(rowKept)) { + const into = Object.prototype.hasOwnProperty.call(union, field) ? union[field] : (union[field] = []); + for (const v of values) if (!into.includes(v)) into.push(v); + } + } + return Object.keys(union).length > 0 ? { ...writeCtx, keptOptionValues: union } : writeCtx; + }; + /** * The dry run's verdict for one coerced row — asked of the engine, never * derived here (#4633 ruling D). `null` means this protocol offers no @@ -686,14 +711,16 @@ export function runImport(opts: RunImportOptions): Promise { const previewVerdict = async ( data: Record, mode: 'insert' | 'update', + rowCtx: typeof writeCtx, ): Promise[number] | null> => { if (typeof p.validateData !== 'function') return null; const res = await p.validateData({ object: objectName, data, mode, // The SAME context the write would carry, so a historical import's - // `skipStateMachine` and the caller's locale reach validation here - // exactly as they reach it on the write path. - context: writeCtx, + // `skipStateMachine`, the caller's locale and the row's kept option + // values (#22183) reach validation here exactly as they reach it on the + // write path. + context: rowCtx, ...(environmentId ? { environmentId } : {}), }); return res?.results?.[0] ?? null; @@ -732,7 +759,7 @@ export function runImport(opts: RunImportOptions): Promise { // batch call, so a bad row never forces the whole-batch degradation that // re-runs beforeInsert hooks on its siblings. const canPartialCreate = typeof p.insertManyData === 'function'; - const pendingCreates: Array<{ index: number; rowNo: number; data: Record }> = []; + const pendingCreates: Array<{ index: number; rowNo: number; data: Record; kept: Record }> = []; // bulkWrite is at-least-once: a retry (or a mismatch-driven degradation) may // re-run a create whose prior attempt already committed. Every buffered // CREATE row is therefore pre-assigned a client-generated id at flush time @@ -782,6 +809,11 @@ export function runImport(opts: RunImportOptions): Promise { for (const b of batch) { if (b.data.id == null || b.data.id === '') b.data.id = randomUUID(); } + // [#22183] Each row's kept option values, keyed by the row object itself + // (the same objects `bulkWrite` hands the closures), so every write call + // below carries the values of exactly the rows it writes. + const keptByRow = new Map, Record>(batch.map((b) => [b.data, b.kept])); + const ctxForRows = (chunkRows: ReadonlyArray>) => writeCtxFor(chunkRows.map((r) => keptByRow.get(r))); // [#20701] Each row's drop report, recorded by the write call that wrote // it and keyed by the row object itself (`bulkWrite` hands the closures // the same objects it was given), because `bulkWrite`'s per-row result @@ -824,7 +856,7 @@ export function runImport(opts: RunImportOptions): Promise { } else { try { freshOutcomes = (await p.insertManyData!({ - object: objectName, records: toCreate, context: writeCtx, + object: objectName, records: toCreate, context: ctxForRows(toCreate), ...(environmentId ? { environmentId } : {}), })).outcomes; } catch (e) { @@ -869,7 +901,7 @@ export function runImport(opts: RunImportOptions): Promise { } else { try { createdRecords = (await p.createManyData!({ - object: objectName, records: toCreate, context: writeCtx, + object: objectName, records: toCreate, context: ctxForRows(toCreate), ...(environmentId ? { environmentId } : {}), })).records; } catch (e) { @@ -906,7 +938,7 @@ export function runImport(opts: RunImportOptions): Promise { } try { const res = await p.createData({ - object: objectName, data: row, context: writeCtx, + object: objectName, data: row, context: ctxForRows([row]), ...(environmentId ? { environmentId } : {}), }); dropsByRow.set(row, responseDrops(res)); // [#20701] one row's response, so the row's report @@ -940,7 +972,7 @@ export function runImport(opts: RunImportOptions): Promise { const rowNo = i + 1; try { // 1. Coerce every cell to its storage value (+ resolve lookups). - const { data, errors } = await coerceRow(rows[i], metaMap, { + const { data, errors, keptOptionValues } = await coerceRow(rows[i], metaMap, { trimWhitespace, nullValues, createMissingOptions, resolveRef, // Cell-coercion failures land in the same row report as the engine's // validation errors, so they speak the same language (#3957). @@ -958,6 +990,8 @@ export function runImport(opts: RunImportOptions): Promise { errCount++; results[i] = { row: rowNo, ok: false, action: 'failed', field: first.field, code: first.code, error: first.message }; } else { + // [#22183] This row's write context: the kept option values ride along. + const rowCtx = writeCtxFor([keptOptionValues]); // 2. Decide create vs update vs skip. let existing: Record | 'blank' | 'none' | 'ambiguous' = 'none'; let handled = false; @@ -988,7 +1022,7 @@ export function runImport(opts: RunImportOptions): Promise { // The write path needs no counterpart: `validateRecord` runs // there for real, after the hooks, and the row report is built // from the very same findings by `toFailedResult`. - const verdict = await previewVerdict(data, willUpdate ? 'update' : 'insert'); + const verdict = await previewVerdict(data, willUpdate ? 'update' : 'insert', rowCtx); if (verdict && !verdict.valid) { errCount++; const first = verdict.errors[0]; @@ -1015,7 +1049,7 @@ export function runImport(opts: RunImportOptions): Promise { let updateSummaryStale = false; let updateDrops: readonly DroppedFieldsEvent[] | undefined; try { - res2 = await withTransientRetry(() => p.updateData({ object: objectName, id: target.id, data, context: writeCtx, ...(environmentId ? { environmentId } : {}) })); + res2 = await withTransientRetry(() => p.updateData({ object: objectName, id: target.id, data, context: rowCtx, ...(environmentId ? { environmentId } : {}) })); updateDrops = responseDrops(res2); // [#20701] read off the response, never off a recovered record } catch (e) { // Record updated but summary recompute failed (framework#3147): @@ -1034,14 +1068,14 @@ export function runImport(opts: RunImportOptions): Promise { ...rowDrops(updateDrops) }; } else if (canBulkCreate) { // Buffer — the actual write happens in a batched flush below. - pendingCreates.push({ index: i, rowNo, data }); + pendingCreates.push({ index: i, rowNo, data, kept: keptOptionValues }); } else { // No bulk-create primitive on this protocol: original inline path. // Wrap in transient retry to match the update path above (L352) // and the batched create path (bulkWrite's internal retry) — a // single `fetch failed` blip must not silently drop the row // (framework#3150). - const res2 = await withTransientRetry(() => p.createData({ object: objectName, data, context: writeCtx, ...(environmentId ? { environmentId } : {}) })); + const res2 = await withTransientRetry(() => p.createData({ object: objectName, data, context: rowCtx, ...(environmentId ? { environmentId } : {}) })); const id = extractRecordId(res2); okCount++; created++; if (collectUndo && id != null) undoLog.created.push(id); diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index b0ab6aac69d..10152d86085 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -12916,6 +12916,9 @@ export class ObjectQL implements IObjectQLEngine { // in a second pass after its own strip). validateRecordInScope(schemaForValidation, row, mode, 'include', { mediaValueShapeStrict, valueShapeStrict, messages, onAdmittedValueShapeViolation, + // [#22183] The option values an import kept for this write — the + // preview admits exactly what the write would. + keptOptionValues: options?.context?.keptOptionValues, }); evaluateValidationRules(schemaForValidation as any, row, mode, { logger: this.logger, currentUser, skipStateMachine, messages, @@ -13730,7 +13733,8 @@ export class ObjectQL implements IObjectQLEngine { // its SHAPE is judged here like any other field's. See // `ReadonlyValueScope` (record-validator.ts). normalizeMultiValueFields(schemaForValidation, rows[i], 'include'); - validateRecordInScope(schemaForValidation, rows[i], 'insert', 'include', { mediaValueShapeStrict, valueShapeStrict, messages: msgCtx, onAdmittedValueShapeViolation }); + // [#22183] `keptOptionValues`: the option values an import kept for this write. + validateRecordInScope(schemaForValidation, rows[i], 'insert', 'include', { mediaValueShapeStrict, valueShapeStrict, messages: msgCtx, onAdmittedValueShapeViolation, keptOptionValues: opCtx.context?.keptOptionValues }); evaluateValidationRules(schemaForValidation as any, rows[i], 'insert', { logger: this.logger, currentUser: this.buildEvalUser(opCtx.context), skipStateMachine: shouldSkipStateMachine(opCtx.context), messages: msgCtx, parent: insertParentForRow?.(rows[i]), related: insertRelatedForRow(rows[i]), permissions: insertPermissionsFor(rows[i]) }); await this.assertReferencesResolve( schemaForValidation, rows[i], suppliedPerRow[i], opCtx.context, msgCtx, @@ -15088,7 +15092,8 @@ export class ObjectQL implements IObjectQLEngine { // become a refusal. Readonly values are judged after the strip // (`validateRecordInScope(…, 'only')`, below). normalizeMultiValueFields(updateSchema, hookContext.input.data as Record, 'skip'); - validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, valueShapeStrict, messages: updateMsgCtx, onAdmittedValueShapeViolation }); + // [#22183] `keptOptionValues`: the option values an import kept for this write. + validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, valueShapeStrict, messages: updateMsgCtx, onAdmittedValueShapeViolation, keptOptionValues: opCtx.context?.keptOptionValues }); // [#5284] Demand-driven, and the demand is asked PER OBJECT. // // This gate used to ask `this.hooks.get('afterUpdate').length > 0` @@ -15423,7 +15428,8 @@ export class ObjectQL implements IObjectQLEngine { // become a refusal. Readonly values are judged after the strip // (`validateRecordInScope(…, 'only')`, below). normalizeMultiValueFields(updateSchema, hookContext.input.data as Record, 'skip'); - validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, valueShapeStrict, messages: updateMsgCtx, onAdmittedValueShapeViolation }); + // [#22183] `keptOptionValues`: the option values an import kept for this write. + validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, valueShapeStrict, messages: updateMsgCtx, onAdmittedValueShapeViolation, keptOptionValues: opCtx.context?.keptOptionValues }); // [#2982] The middleware-composed AST — asserted present and // bound to the memoized row read in the pre-phase above, so the // injected row-scoping (RLS write filter, sharing's diff --git a/packages/objectql/src/import-kept-option-values.test.ts b/packages/objectql/src/import-kept-option-values.test.ts new file mode 100644 index 00000000000..34d50bdacc8 --- /dev/null +++ b/packages/objectql/src/import-kept-option-values.test.ts @@ -0,0 +1,275 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// +// #22183 — `ImportRequest.createMissingOptions`, end to end through the REAL +// engine. +// +// ## The defect +// +// With the option on, the import's cell coercion kept a `select` / +// `multiselect` cell that matched no option, and the engine's write-path option +// check then refused the same value ("Priority must be one of: high, low"). So +// on every writable field the option changed WHICH stage refused the row, never +// WHETHER it was refused. +// +// ## The fix +// +// Coercion reports the values it kept; the import runner hands exactly those to +// each write as `ExecutionContext.keptOptionValues`; the record validator's +// option arms admit a listed value on that write and still refuse every other +// value outside the options. The field's option list is never written. +// +// ## Pins +// +// (a) the import door: with the option on, a kept select value and a kept +// multiselect item are STORED, on the batched path, the per-row path, an +// update-mode import and the dry run; +// (b) control: with the option off the same cells are refused at coercion; +// (c) the admission is the import's only: a plain write of the same value is +// refused, and so is a write naming a different kept value; +// (d) a LATER edit of such a row: one that leaves the field out is accepted, +// one that sends the kept value back is judged against the options again +// and refused, and one that picks an option is accepted — the row stays +// editable and the option list is unchanged. +// +// (a) was red before the fix: every row came back refused by the engine. + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { ObjectKernel, runImport, type ImportProtocolLike } from '@objectstack/core'; +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; +import { ObjectQLPlugin } from './plugin.js'; +import { ObjectQL } from './engine.js'; +import { ValidationError } from './validation/record-validator.js'; + +const OBJECT = 'kept_ticket'; +const PRIORITY_OPTIONS = [{ value: 'high', label: 'High' }, { value: 'low', label: 'Low' }]; +const TAG_OPTIONS = [{ value: 'important', label: 'Important' }, { value: 'review', label: 'Review' }]; + +/** A store-backed stub driver: the stored row IS the verdict. */ +function makeStubDriver() { + const rows = new Map>(); + const matches = (row: any, where: any): boolean => { + if (!where || typeof where !== 'object') return true; + return Object.entries(where).every(([k, v]: [string, any]) => { + if (k === '$and') return (v as any[]).every((w) => matches(row, w)); + if (k.startsWith('$')) throw new Error(`stub driver: unsupported combinator ${k}`); + if (v !== null && typeof v === 'object' && !Array.isArray(v)) { + return Object.entries(v).every(([op, target]) => { + if (op === '$eq') return row?.[k] === target; + if (op === '$in') return Array.isArray(target) && target.includes(row?.[k]); + // REFUSE, never silently match. + throw new Error(`stub driver: unsupported operator ${op}`); + }); + } + return row?.[k] === v; + }); + }; + let n = 0; + const driver: any = { + name: 'kept-store', version: '0.0.0', supports: {}, + async connect() {}, async disconnect() {}, async checkHealth() { return true; }, + async syncSchema() {}, + async find(_o: string, ast: any) { + const hits = Array.from(rows.values()).filter((r) => matches(r, ast?.where)); + const page = typeof ast?.limit === 'number' ? hits.slice(0, ast.limit) : hits; + return page.map((r) => ({ ...r })); + }, + async findOne(_o: string, ast: any) { + for (const r of rows.values()) if (matches(r, ast?.where)) return { ...r }; + return null; + }, + async create(_o: string, data: Record) { + n += 1; + const id = (data.id as string) ?? `rec_${n}`; + const row = { ...data, id }; + rows.set(id, row); + return { ...row }; + }, + async update(_o: string, id: string, data: Record) { + const row = { ...rows.get(id), ...data, id }; + rows.set(id, row); + return { ...row }; + }, + async delete(_o: string, id: string) { return rows.delete(id); }, + async count(_o: string, ast: any) { + return Array.from(rows.values()).filter((r) => matches(r, ast?.where)).length; + }, + }; + return { driver, rows }; +} + +/** The field findings of an engine refusal, or `null` when the write landed. */ +async function refusal(write: () => Promise) { + try { + await write(); + return null; + } catch (e) { + expect(e).toBeInstanceOf(ValidationError); + const err = e as ValidationError; + return { code: err.code, fields: err.fields.map((f) => ({ field: f.field, code: f.code })) }; + } +} + +describe('#22183 — createMissingOptions keeps an unmatched option value through the engine', () => { + let kernel: ObjectKernel; + let objectql: ObjectQL; + let stub: ReturnType; + + beforeEach(async () => { + kernel = new ObjectKernel({ logger: { level: 'silent' }, gracefulShutdown: false }); + stub = makeStubDriver(); + await kernel.use({ + name: 'kept-store-plugin', type: 'driver', version: '1.0.0', + init: async (ctx: any) => { ctx.registerService('driver.kept-store', stub.driver); }, + } as any); + await kernel.use(new ObjectQLPlugin()); + await kernel.bootstrap(); + objectql = kernel.getService('objectql'); + objectql.registry.registerObject({ + name: OBJECT, + label: 'Kept Ticket', + datasource: 'kept-store', + fields: { + code: { name: 'code', label: 'Code', type: 'text' }, + title: { name: 'title', label: 'Title', type: 'text' }, + priority: { name: 'priority', label: 'Priority', type: 'select', options: PRIORITY_OPTIONS }, + tags: { name: 'tags', label: 'Tags', type: 'multiselect', options: TAG_OPTIONS }, + }, + } as any, 'test', 'test'); + }); + + afterEach(async () => { + if (kernel.getState() === 'running') await kernel.shutdown(); + }); + + const storedByCode = (code: string) => { + const hits = Array.from(stub.rows.values()).filter((r) => r.code === code); + expect(hits.length, `exactly one stored row for code ${code}`).toBe(1); + return hits[0]!; + }; + + const metaMap = new Map([ + ['code', { name: 'code', type: 'text' }], + ['title', { name: 'title', type: 'text' }], + ['priority', { name: 'priority', type: 'select', options: PRIORITY_OPTIONS }], + ['tags', { name: 'tags', type: 'multiselect', options: TAG_OPTIONS }], + ]); + + const ROWS = [ + { code: 'sel', title: 'kept select', priority: 'Bogus' }, + { code: 'multi', title: 'kept multiselect item', tags: 'Important, Bogus' }, + { code: 'plain', title: 'matched', priority: 'High' }, + ]; + + function importer(p: ImportProtocolLike) { + return (over: Record) => runImport({ + p, + objectName: OBJECT, + metaMap, + writeMode: 'insert', + matchFields: ['code'], + dryRun: false, + runAutomations: false, + trimWhitespace: true, + createMissingOptions: true, + skipBlankMatchKey: false, + context: { userId: 'usr_importer' }, + rows: ROWS, + ...over, + } as any); + } + + /** The real protocol: batched creates (`insertManyData`), `validateData`. */ + const fullProtocol = () => new ObjectStackProtocolImplementation(objectql as never) as unknown as ImportProtocolLike; + /** The same protocol offering only the per-row calls, so each row is its own `createData`. */ + const perRowProtocol = (): ImportProtocolLike => { + const impl = fullProtocol(); + return { + findData: (args) => impl.findData(args), + createData: (args) => impl.createData(args), + updateData: (args) => impl.updateData(args), + }; + }; + + for (const [label, protocol] of [['batched', fullProtocol], ['per-row', perRowProtocol]] as const) { + it(`(a) ${label}: a kept select value and a kept multiselect item are stored`, async () => { + const summary = await importer(protocol())({}); + expect(summary.results.map((r) => ({ ok: r.ok, action: r.action, code: r.code }))).toEqual([ + { ok: true, action: 'created', code: undefined }, + { ok: true, action: 'created', code: undefined }, + { ok: true, action: 'created', code: undefined }, + ]); + expect(storedByCode('sel').priority).toBe('Bogus'); + expect(storedByCode('multi').tags).toEqual(['important', 'Bogus']); + expect(storedByCode('plain').priority).toBe('high'); + // The option list is the author's, unchanged. + expect((objectql.registry.getObject(OBJECT) as any).fields.priority.options.map((o: any) => o.value)).toEqual(['high', 'low']); + }); + } + + it('(a) the dry run admits what the write would, and stores nothing', async () => { + const summary = await importer(fullProtocol())({ dryRun: true }); + expect(summary.results.map((r) => ({ ok: r.ok, action: r.action }))).toEqual([ + { ok: true, action: 'created' }, + { ok: true, action: 'created' }, + { ok: true, action: 'created' }, + ]); + expect(stub.rows.size).toBe(0); + }); + + it('(a) an update-mode import stores the kept value on the matched row', async () => { + stub.rows.set('rec_u', { id: 'rec_u', code: 'u', title: 'existing', priority: 'low' }); + const summary = await importer(fullProtocol())({ + writeMode: 'update', rows: [{ code: 'u', priority: 'Legacy', tags: 'Review; Archived' }], + }); + expect(summary.results.map((r) => ({ ok: r.ok, action: r.action }))).toEqual([{ ok: true, action: 'updated' }]); + expect(storedByCode('u')).toMatchObject({ priority: 'Legacy', tags: ['review', 'Archived'] }); + }); + + it('(b) control: with the option off the same cells are refused at coercion', async () => { + const summary = await importer(fullProtocol())({ createMissingOptions: false }); + expect(summary.results.map((r) => ({ ok: r.ok, code: r.code, field: r.field }))).toEqual([ + { ok: false, code: 'invalid_option', field: 'priority' }, + { ok: false, code: 'invalid_option', field: 'tags' }, + { ok: true, code: undefined, field: undefined }, + ]); + expect(Array.from(stub.rows.values()).map((r) => r.code)).toEqual(['plain']); + }); + + it('(c) the admission is the import\'s only: a plain write, or a write naming another value, is refused', async () => { + const ctx = { userId: 'usr_editor' }; + expect(await refusal(() => objectql.insert(OBJECT, { code: 'w1', priority: 'Bogus' }, { context: ctx } as any))).toEqual({ + code: 'VALIDATION_FAILED', fields: [{ field: 'priority', code: 'invalid_option' }], + }); + expect(await refusal(() => objectql.insert( + OBJECT, { code: 'w2', priority: 'Bogus' }, { context: { ...ctx, keptOptionValues: { priority: ['Other'] } } } as any, + ))).toEqual({ code: 'VALIDATION_FAILED', fields: [{ field: 'priority', code: 'invalid_option' }] }); + expect(stub.rows.size).toBe(0); + }); + + it('(d) a later edit: leaving the field out is accepted, sending the kept value back is judged again, picking an option is accepted', async () => { + await importer(fullProtocol())({}); + const sel = storedByCode('sel'); + const multi = storedByCode('multi'); + const protocol = new ObjectStackProtocolImplementation(objectql as never); + const ctx = { userId: 'usr_editor' }; + + // The record form's edit writes only the fields that changed. + await protocol.updateData({ object: OBJECT, id: sel.id, data: { title: 'edited' }, context: ctx } as any); + expect(storedByCode('sel')).toMatchObject({ title: 'edited', priority: 'Bogus' }); + await protocol.updateData({ object: OBJECT, id: multi.id, data: { title: 'edited too' }, context: ctx } as any); + expect(storedByCode('multi')).toMatchObject({ title: 'edited too', tags: ['important', 'Bogus'] }); + + // A write that sends the field is judged against the options, as any write is. + expect(await refusal(() => protocol.updateData({ + object: OBJECT, id: sel.id, data: { title: 'echo', priority: 'Bogus' }, context: ctx, + } as any))).toEqual({ code: 'VALIDATION_FAILED', fields: [{ field: 'priority', code: 'invalid_option' }] }); + expect(await refusal(() => protocol.updateData({ + object: OBJECT, id: multi.id, data: { tags: ['important', 'Bogus', 'review'] }, context: ctx, + } as any))).toEqual({ code: 'VALIDATION_FAILED', fields: [{ field: 'tags', code: 'invalid_option' }] }); + expect(storedByCode('sel').title).toBe('edited'); + + // Picking an option is accepted. + await protocol.updateData({ object: OBJECT, id: sel.id, data: { priority: 'low' }, context: ctx } as any); + expect(storedByCode('sel').priority).toBe('low'); + }); +}); diff --git a/packages/objectql/src/validation/record-validator.kept-option-values.test.ts b/packages/objectql/src/validation/record-validator.kept-option-values.test.ts new file mode 100644 index 00000000000..fc4a8f257fd --- /dev/null +++ b/packages/objectql/src/validation/record-validator.kept-option-values.test.ts @@ -0,0 +1,74 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; +import { validateRecord, ValidationError, type ValidateRecordOptions } from './record-validator.js'; + +/** + * #22183 — the option arms admit the values a write names in + * `keptOptionValues` (the cells an import kept under `createMissingOptions`), + * and nothing else. + * + * The engine passes the write's `ExecutionContext.keptOptionValues` here; the + * end-to-end half (the import runner through the real engine, and what a later + * edit of the row does) is `../import-kept-option-values.test.ts`. + */ +const schema = { + fields: { + priority: { name: 'priority', label: 'Priority', type: 'select', options: [{ value: 'high', label: 'High' }, { value: 'low', label: 'Low' }] }, + tags: { name: 'tags', label: 'Tags', type: 'multiselect', options: [{ value: 'important', label: 'Important' }, { value: 'review', label: 'Review' }] }, + stage: { name: 'stage', label: 'Stage', type: 'select', picklist: 'deal_stages', options: [] }, + }, +}; + +/** The refusal's envelope and its field findings, or `null` when the call accepted. */ +function verdict(data: Record, mode: 'insert' | 'update', options: ValidateRecordOptions = {}) { + try { + validateRecord(schema, data, mode, options); + return null; + } catch (e) { + expect(e).toBeInstanceOf(ValidationError); + const err = e as ValidationError; + return { code: err.code, fields: err.fields.map((f) => ({ field: f.field, code: f.code })) }; + } +} + +describe('validateRecord — keptOptionValues (#22183)', () => { + for (const mode of ['insert', 'update'] as const) { + it(`${mode}: a single-value select admits the kept value and still refuses any other value outside the options`, () => { + const kept = { keptOptionValues: { priority: ['Bogus'] } }; + expect(verdict({ priority: 'Bogus' }, mode, kept)).toBeNull(); + expect(verdict({ priority: 'high' }, mode, kept)).toBeNull(); + expect(verdict({ priority: 'Other' }, mode, kept)).toEqual({ + code: 'VALIDATION_FAILED', fields: [{ field: 'priority', code: 'invalid_option' }], + }); + // Control: without the admission the same value is refused, as before. + expect(verdict({ priority: 'Bogus' }, mode)).toEqual({ + code: 'VALIDATION_FAILED', fields: [{ field: 'priority', code: 'invalid_option' }], + }); + }); + + it(`${mode}: a multi-value field admits each kept item and refuses an item that was not kept`, () => { + const kept = { keptOptionValues: { tags: ['Other'] } }; + expect(verdict({ tags: ['important', 'Other'] }, mode, kept)).toBeNull(); + expect(verdict({ tags: ['Other', 'Stray'] }, mode, kept)).toEqual({ + code: 'VALIDATION_FAILED', fields: [{ field: 'tags', code: 'invalid_option' }], + }); + expect(verdict({ tags: ['important', 'Other'] }, mode)).toEqual({ + code: 'VALIDATION_FAILED', fields: [{ field: 'tags', code: 'invalid_option' }], + }); + }); + } + + it('an admission is per field: a value kept for one field does not pass on another', () => { + expect(verdict({ priority: 'Other', tags: ['Bogus'] }, 'insert', { keptOptionValues: { priority: ['Bogus'], tags: ['Other'] } })).toEqual({ + code: 'VALIDATION_FAILED', + fields: [{ field: 'priority', code: 'invalid_option' }, { field: 'tags', code: 'invalid_option' }], + }); + }); + + it('a picklist that did not resolve stays refused, whatever the write admits', () => { + expect(verdict({ stage: 'won' }, 'update', { keptOptionValues: { stage: ['won'] } })).toEqual({ + code: 'VALIDATION_FAILED', fields: [{ field: 'stage', code: 'invalid_option' }], + }); + }); +}); diff --git a/packages/objectql/src/validation/record-validator.ts b/packages/objectql/src/validation/record-validator.ts index d57cbd30780..c30d7977f3b 100644 --- a/packages/objectql/src/validation/record-validator.ts +++ b/packages/objectql/src/validation/record-validator.ts @@ -52,7 +52,9 @@ * scale: 2` refuses `1234.5`. A fraction-stored `percent` is * always counted two places further right. No column change. * - format email / url / phone (lightweight RFC-aware regex) - * - select / multiselect: value must appear in `options` + * - select / multiselect: value must appear in `options`, or be one the write + * names for the field in `keptOptionValues` (an import's kept + * cell, #22183) * - boolean / toggle: must coerce to boolean * - date / datetime: must be ISO-parsable, naming a year from 0001 to 9999 * for a `date` and from 1000 to 9999 for a `datetime` (#20280); @@ -992,6 +994,10 @@ function validateOne( // constraint — see {@link ReadonlyValueScope}. Each `if (shapeOnly) return // null` below sits where an arm's shape test ends and its constraints begin. shapeOnly = false, + // [#22183] The values outside `def.options` this write admits for this field + // — the cells an import kept (`ValidateRecordOptions.keptOptionValues`). + // Read by the two option arms only; every other arm ignores it. + keptOptions: readonly string[] = [], ): FieldValidationError | null { const fail = ( code: FieldErrorCode, @@ -1496,7 +1502,9 @@ function validateOne( if (shapeOnly) return null; const allowed = optionValues(def.options); if (picklist !== undefined && allowed.length === 0) return picklistUnresolved(); - if (allowed.length > 0 && !allowed.includes(String(value))) { + // [#22183] A value an import kept is admitted on that write; the list is + // not widened, so the refusal below still names the declared options. + if (allowed.length > 0 && !allowed.includes(String(value)) && !keptOptions.includes(String(value))) { return fail( 'invalid_option', { allowed: allowed.join(', ') }, @@ -1528,7 +1536,8 @@ function validateOne( return picklist !== undefined && value.length > 0 ? picklistUnresolved() : null; } for (const v of value) { - if (!allowed.includes(String(v))) { + // [#22183] Each item an import kept is admitted on that write, and only it. + if (!allowed.includes(String(v)) && !keptOptions.includes(String(v))) { return fail( 'invalid_option', { allowed: allowed.join(', ') }, @@ -1861,6 +1870,33 @@ export interface ValidateRecordOptions { * truthiness test on a path that is already building an error message. */ onAdmittedValueShapeViolation?: AdmittedValueShapeViolationSink; + + /** + * [#22183] Per field name, the values outside the field's options that THIS + * write admits: the cells a data import kept because they matched no option + * (`ImportRequest.createMissingOptions`). The engine passes the write's + * `ExecutionContext.keptOptionValues` here; the import runner is its one + * producer. + * + * Read by the `select` / `radio` and multi-value option arms only, and only + * as "this exact value is admitted": every other value outside the options is + * refused as before, the refusal still names the declared options, and a + * picklist that did not resolve stays refused. Omitted — every write but an + * import that kept a value — means no value is admitted beyond the options. + */ + keptOptionValues?: Readonly>; +} + +/** + * [#22183] The kept values {@link ValidateRecordOptions.keptOptionValues} names + * for one field, or none. Only an array is a list of kept values. + */ +function keptOptionsFor( + kept: ValidateRecordOptions['keptOptionValues'], + name: string, +): readonly string[] { + const values = kept?.[name]; + return Array.isArray(values) ? values : []; } /** @@ -1911,6 +1947,7 @@ export function validateRecordInScope( const valueStrict = options.valueShapeStrict === true; const messages = options.messages; const onAdmitted = options.onAdmittedValueShapeViolation; + const kept = options.keptOptionValues; if (mode === 'insert') { // Walk all declared fields — required check applies even when @@ -1920,7 +1957,7 @@ export function validateRecordInScope( // [#21663] A readonly value: its SHAPE, never required (the engine owns // its presence) and never a constraint — see ReadonlyValueScope. const shapeOnly = def.readonly === true; - const err = validateOne(name, def, data[name], shapeOnly, mediaStrict, messages, valueStrict, onAdmitted, shapeOnly); + const err = validateOne(name, def, data[name], shapeOnly, mediaStrict, messages, valueStrict, onAdmitted, shapeOnly, keptOptionsFor(kept, name)); if (err) errors.push(err); } } else { @@ -1955,7 +1992,7 @@ export function validateRecordInScope( // skipRequired: PATCH-omitted fields must not 400. (No def clone — the // registry's own field object flows through so the ADR-0104 value-shape // schema cache, keyed on def identity, hits.) - const err = validateOne(name, def, value, true, mediaStrict, messages, valueStrict, onAdmitted); + const err = validateOne(name, def, value, true, mediaStrict, messages, valueStrict, onAdmitted, false, keptOptionsFor(kept, name)); if (err) errors.push(err); } } diff --git a/packages/spec/authorable-surface/kernel.json b/packages/spec/authorable-surface/kernel.json index 33f19c39275..2d331ad6c24 100644 --- a/packages/spec/authorable-surface/kernel.json +++ b/packages/spec/authorable-surface/kernel.json @@ -177,6 +177,7 @@ "kernel/ExecutionContext:email", "kernel/ExecutionContext:flowRunId", "kernel/ExecutionContext:isSystem", + "kernel/ExecutionContext:keptOptionValues", "kernel/ExecutionContext:locale", "kernel/ExecutionContext:oauthScopes", "kernel/ExecutionContext:onBehalfOf", diff --git a/packages/spec/src/api/export.zod.ts b/packages/spec/src/api/export.zod.ts index 9dcdf9e56da..fc6c5bb09a1 100644 --- a/packages/spec/src/api/export.zod.ts +++ b/packages/spec/src/api/export.zod.ts @@ -300,7 +300,14 @@ export const ImportRequestSchema = lazySchema(() => z.object({ nullValues: z.array(z.string()).optional() .describe('Strings treated as null/blank (besides empty string)'), createMissingOptions: z.boolean().default(false) - .describe('Keep unmatched select values instead of failing the row'), + .describe( + 'Keep a select / radio / multiselect cell that matches none of the field\'s options instead of ' + + 'failing the row. The cell is stored as written (trimmed; for a multi-value cell, each unmatched item), ' + + 'and the engine\'s option check admits exactly the values this import kept, on this import\'s writes ' + + '(dry run included). The field\'s option list is not changed: a later write that sends the field is ' + + 'judged against the options again, while one that leaves the field out is not affected. A field bound ' + + 'to a shared picklist that did not resolve is still refused.', + ), skipBlankMatchKey: z.boolean().default(false) .describe('Skip rows whose matchFields are blank (default: upsert creates them, update skips them)'), }).refine((body) => !(body.mappingName && body.mapping), { diff --git a/packages/spec/src/kernel/execution-context.zod.ts b/packages/spec/src/kernel/execution-context.zod.ts index dfd41bd95b9..6c8a272e682 100644 --- a/packages/spec/src/kernel/execution-context.zod.ts +++ b/packages/spec/src/kernel/execution-context.zod.ts @@ -440,6 +440,31 @@ export const ExecutionContextSchema = lazySchema(() => z.object({ */ preserveAudit: z.boolean().optional().describe('Historical import: preserve the ORIGINAL audit timeline for this write instead of stamping it "now". Opt-in and server-constructed only, never client-supplied. On the UPDATE path it admits a whitelist — the audit/timestamp family (created_at / created_by / updated_at / updated_by) plus author-declared business `readonly` fields — while platform-managed `system` columns (tenancy, generated) stay stripped. On INSERT the exemption does NOT apply: the create-side static `readonly` strip runs inside `engine.insert` itself (after the `beforeInsert` hooks, before validation — the 2026-09-03 ruling; the DataProtocol ingress copy it replaced is deleted) and reads only `context.isSystem`, so a non-system create carrying `preserveAudit` still has those fields stripped and is warned (WARN) that the exemption is UPDATE-only — replaying archival readonly facts on create requires a system context. Permissions / RLS / field-level security are unaffected.'), + /** + * The option values a data import KEPT for this write, by field name (#22183) + * — the other half of `ImportRequest.createMissingOptions`. + * + * With that option on, the import's cell coercion keeps a `select` / `radio` + * / `multiselect` cell that matches none of the field's options instead of + * failing the row, and the import runner puts exactly the kept values here: + * one entry per field, listing the unmatched cells (each unmatched item of a + * multi-value cell). The engine's write-path option check admits a value + * outside the field's options when it is listed here for that field, on this + * write only, and still refuses every other value outside them. A field bound + * to a shared picklist that did not resolve stays refused: that is a list + * that failed to load, not a value the list lacks. + * + * Nothing else changes: the field's option list is not written, permissions / + * RLS / field-level security are unaffected, and a LATER write that sends the + * field is judged against the options again (one that leaves the field out is + * not judged on it at all). + * + * Server-constructed only, never client-supplied — the import runner sets it + * from the cells it kept, exactly like {@link skipStateMachine} and + * {@link preserveAudit} from `treatAsHistorical`. + */ + keptOptionValues: z.record(z.string(), z.array(z.string())).optional().describe('Import with `createMissingOptions`: per field name, the option values the import kept because they matched none of the field\'s options. The write-path option check admits exactly these values on this write (and still refuses every other value outside the options); the field\'s options are not changed, and a later write that sends the field is judged against them again. Server-constructed only (set by the import runner), never client-supplied.'), + /** * OAuth 2.1 scopes granted to the access token that authenticated this * request, when the principal was resolved from an OAuth bearer token