Skip to content

Align builtin reduction callback controls - #9718

Merged
youknowone merged 8 commits into
RustPython:mainfrom
youknowdot:fix/minmax-streaming-controls-20261104
Nov 7, 2026
Merged

youknowone merged 8 commits into
RustPython:mainfrom
youknowdot:fix/minmax-streaming-controls-20261104

Conversation

@youknowdot

@youknowdot youknowdot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Group CPython 3.15 min/max streaming, sum and all/any iteration, argument binding and callback ownership controls in 274 changed lines across three production Rust files.

  • Consume a single iterable one item at a time, interleaving key and comparison callbacks without materialization or length-hint calls. Multiple positional candidates use borrowed caller slots.

  • Keep raw caller arguments alive until internal cleanup finishes. Share one min/max engine with explicit winner/current item and key ownership and correct success/error release order.

  • Reuse ArgSpec through a borrowed keyword-only entry point. Validate arity, keyword count/names and default restrictions before obtaining an iterator.

  • Scope CPython-style typo suggestions and lossless WTF-8 keyword diagnostics to the new entry point. Reuse the existing suggestion algorithm with UTF-8 byte-length thresholds and suppress suggestions for surrogate-bearing names. Existing parser and suggestion consumers retain their behavior.

  • Bind sum inputs before iterability checks, acquire and validate its iterator before rejected-start errors, and retain the original start through all callbacks.

  • Preserve generic sum accumulator-before-item-before-iterator error cleanup without changing compensated arithmetic or numeric fast paths.

  • Share one all/any engine with canonical iterator validation and one cached next slot. Python slot wrappers continue to resolve a replaced/deleted next dynamically, including mutation from truth callbacks and item finalizers.

  • Preserve raw inputs, item-before-next cleanup, short circuiting and length-hint avoidance without changing generic iteration or min/max/sum traversal policy.

Validation

Pinned CPython reference: d220409a13b77de7a6c30abe16ea90db01528edc. The focused oracle passed 22 grouped assertions and recorded 30 outcomes/traces, covering streaming, mutation, length-hint avoidance, binding, defaults, comparisons, exception propagation and internal finalizer order. These are reference/source checks, not RustPython runtime tests.

Independent source review passed, including 344 additional CPython diagnostic assertions over 172 keyword names. The initial compiler run caught a Result-versus-Option guard method mismatch; the corrected final source passed threaded cargo check in 1m05s and all-targets Clippy in 2m58s. All 12 unchanged baseline diagnostics match exactly, without suppression. Ordinary commit hooks passed. No tests, Lib files, expected failures or workflows are changed.

Limits and integration

The original validation used reference and source checks. Current native validation is recorded in the borrowed-source follow-up below. Full runtime differential tests and separate platform/feature graphs remain outside that validation; hosted CI is pending.

Caller-frame argument destruction order can depend on the original call syntax, which FuncArgs no longer records. This patch retains raw owners through internal cleanup; it does not impose one universal caller-tail ordering or claim to solve that wider lifetime issue. Duplicate-keyword detection belongs to VM call binding before FuncArgs' unique-key map.

PR #9668 makes ArgSpec fields and parse_with public. Read-only three-way source composition succeeds while retaining those public interfaces and the new shared parser internals; the combined graph was not compiled. #9699 does not implement min/max behavior, and the examined older builtin changes do not duplicate this group. Existing non-opted-in parser consumers' surrogate diagnostics and suggestion behavior remain unchanged.

AI-assisted contribution by OpenAI Codex; assistance is disclosed in the commit trailer.

Sum follow-up validation

Independent exact-source review passed 30 pinned CPython oracle cases, 14 additional numeric error controls and the existing arithmetic snippet on CPython. Final threaded cargo check passed in 1m05s and all-targets Clippy in 2m17s; all 12 baseline diagnostics match unchanged. An initial redundant-clone diagnostic was fixed by making the existing iterator-then-raw-start release order explicit, without suppression or removing retention. Normal commit hooks passed. These earlier checks were reference/source and compilation checks; the current native results are recorded below.

The sum wrapper continues to use shared FromArgs binding. Exact total-arity/duplicate-argument diagnostic precedence and keyword typo suggestions are deferred; this does not alter shared binder behavior for other functions.

All/any follow-up validation

Independent exact-source review passed bounded CPython controls for truth and item-finalizer next-slot mutation (eight cases), invalid iterator results, short circuiting, errors and length-hint avoidance. The existing test_all and test_any methods were extracted unchanged and passed on pinned CPython. Final threaded host-env/VM/stdlib cargo check passed in 1m11s, all-targets Clippy in 2m47s, and all 12 baseline diagnostics match unchanged. Normal commit hooks passed. A newly unused adapter import was removed before final validation. No native RustPython runtime test was included in that earlier validation; current results are recorded below.

Terminal StopIteration conversion still uses the shared wrapper, which discards the original exception before returning its payload. This follow-up preserves the previous payload-before-iterator release order; full traceback-dependent terminal ownership parity remains deferred. Existing generic binder diagnostic precedence is unchanged.

Borrowed-source follow-up

Rebased independently onto main at fb57e68, preserving all seven prior commits patch-for-patch. The follow-up changes only the shared all/any helper: borrow &PyObject and obtain its iterator through get_iter, while each wrapper retains the original iterable through callbacks and cleanup. This removes coupling to an owned-reference type; it does not reduce reference-count clones, because get_iter still acquires an owned reference internally.

Plain PyIter conversion would release the original source too early. The existing ArgIterable adapter retains its source but does not validate the object returned by __iter__ when acquiring it. Restoring that adapter to sum would move invalid-iterator rejection after invalid-start checks. A common source-retaining, once-validated iterator argument remains a separate proposal; no new conversion framework is introduced here.

Independent review, formatting and ordinary commit hooks passed. This follow-up changes no tests, expected-failure markers or unrelated production files.

Validation on exact commit 29ef984:

  • Full workspace tests with the documented WASM, venvlauncher and C-API exclusions: 1,513 passed, 20 ignored across unit and documentation suites.
  • Full workspace all-target Clippy and separate C-API all-target Clippy: passed with zero warnings. Both used ordinary Clippy without -D warnings.
  • Separate C-API tests: 149 passed, 2 ignored.
  • Fresh release RustPython -m test test_builtin: SUCCESS, 157 tests run and 14 skipped.

Development and Rust test profiles used CARGO_PROFILE_DEV_DEBUG=0 and CARGO_PROFILE_TEST_DEBUG=0; release validation used the normal release profile. The safe existing CPython 3.12.14 host interpreter was verified in the fresh PyO3 build configuration. C-API checks used the repository's RustPython 3.15 configuration without an external interpreter. The CPython 3.15 comparison for this follow-up was source-only; no CPython 3.15 executable was run. Hosted CI for the updated head remains pending.

@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@youknowdot youknowdot changed the title Stream min and max with shared callback controls Align min, max and sum callback controls Nov 4, 2026
@youknowdot youknowdot changed the title Align min, max and sum callback controls Align builtin reduction callback controls Nov 4, 2026
@youknowdot
youknowdot force-pushed the fix/minmax-streaming-controls-20261104 branch from a8ae033 to 215c7f0 Compare November 6, 2026 04:24
@codspeed

codspeed Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 62 untouched benchmarks
⏩ 4 skipped benchmarks1


Comparing youknowdot:fix/minmax-streaming-controls-20261104 (44ebf49) with main (f1f38c1)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

pub(super) struct SumArgs {
#[pyarg(positional)]
iterable: ArgIterable,
iterable: PyObjectRef,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check this change is necessary after #9563

Comment thread crates/vm/src/stdlib/builtins.rs Outdated
}

fn all_or_any(
iterable: &PyObjectRef,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
iterable: &PyObjectRef,
iterable: &PyObject,

@youknowdot
youknowdot force-pushed the fix/minmax-streaming-controls-20261104 branch from 215c7f0 to 44ebf49 Compare November 6, 2026 13:52
youknowdot and others added 8 commits November 7, 2026 09:25
Preserve callback and internal owner order, retain raw caller arguments, and reuse keyword matching and suggestion machinery without changing existing parser consumers.

Assisted-by: OpenAI Codex:model version not exposed
Validate the iterator before rejecting the start type, retain raw arguments, and release generic accumulator operands in CPython order without changing numeric fast paths.

Assisted-by: OpenAI Codex:model version not exposed
Release the iterator before the original start on generic success, preserving callback lifetime order while making the required retention visible to Clippy.

Assisted-by: OpenAI Codex:model version not exposed
Validate iterator acquisition before traversal, cache the next slot while preserving dynamic method dispatch, and retain existing item and terminal cleanup boundaries.

Assisted-by: OpenAI Codex:model version not exposed
All builtin reduction entry points now retain raw objects, leaving this adapter import unused.

Assisted-by: OpenAI Codex:model version not exposed
Preserve borrowed keyword ownership and existing slot validation after the getargs migration.

Assisted-by: OpenAI Codex:model version not exposed
Preserve callback ownership, release ordering, and retained argument owners.

Assisted-by: Codex:model-version-unavailable
Use the shared get_iter entry point while the wrappers retain the original iterable through all callbacks. This changes helper ownership coupling, not reference-count traffic.

Assisted-by: OpenAI Codex:GPT-6
@youknowdot
youknowdot force-pushed the fix/minmax-streaming-controls-20261104 branch from 44ebf49 to 29ef984 Compare November 7, 2026 02:34
@youknowone
youknowone marked this pull request as ready for review November 7, 2026 07:58
@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-07T07:58:21.486289Z
We couldn’t complete this review. Comment @codex review to try again.
29ef984 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@youknowone
youknowone merged commit 9670d1b into RustPython:main Nov 7, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants