Skip to content

Remove Frame mutex and use DataStack bump allocator for LocalsPlus#7333

Merged
youknowone merged 14 commits intoRustPython:mainfrom
youknowone:worktree-frame-datastack
Mar 4, 2026
Merged

Remove Frame mutex and use DataStack bump allocator for LocalsPlus#7333
youknowone merged 14 commits intoRustPython:mainfrom
youknowone:worktree-frame-datastack

Conversation

@youknowone
Copy link
Member

@youknowone youknowone commented Mar 3, 2026

Summary by CodeRabbit

  • Refactor

    • Reworked frame/local storage to unify locals and evaluation stack for leaner execution.
  • Performance

    • Added a fast, bump-allocating data stack to reduce allocation overhead during execution paths.
  • Improvements

    • Better GC/thread-safety handling and updated synchronization behavior.
    • Minor vocabulary additions to the project spellchecker to recognize new identifiers.

@youknowone youknowone changed the title Worktree frame datastack frame datastack Mar 3, 2026
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 3, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a per-thread bump-allocated DataStack and refactors Frame storage to a new LocalsPlus (heap or datastack-backed). Integrates datastack allocation/cleanup into VM frame call paths, updates accessors across frames/builtins, adjusts GC and lock exports, and exposes datastack via the VM internals.

Changes

Cohort / File(s) Summary
Dictionary Configuration
​.cspell.dict/cpython.txt
Added four new spell-check entries: datastack, ncells, nlocalsplus, tstate.
Data Stack Module
crates/vm/src/datastack.rs, crates/vm/src/lib.rs
New public DataStack implementation: chunked bump allocator, push/pop APIs, alignment, chunk growth, Drop impl, and tests; module exported from crate.
Frame Core & LocalsPlus
crates/vm/src/frame.rs
Replaced FastLocals/FrameState with LocalsPlus + FrameUnsafeCell; unified locals+eval-stack storage supporting heap or datastack backing; added materialize API and refactored many opcode/frame accessors.
Frame Execution & Builtins
crates/vm/src/builtins/frame.rs, crates/vm/src/builtins/function.rs, crates/vm/src/builtins/super.rs
Frame construction updated (new use_datastack flag); switched fastlocals borrow access to accessors (fastlocals/fastlocals_mut); added datastack materialize + pop cleanup after frame execution on relevant call paths.
VM Integration
crates/vm/src/vm/mod.rs, crates/vm/src/vm/thread.rs
Added datastack: UnsafeCell<DataStack> to VirtualMachine, initialized per-VM; added datastack_push / datastack_pop helpers; updated Frame::new call sites to pass use_datastack flag.
GC & Locking Changes
crates/vm/src/gc_state.rs, crates/common/src/lock.rs
GcState gains unsafe Send (and conditional Sync) impls; global GC storage switched from OnceLock to static_cell!; LazyLock conditionally re-exported under feature flags and non-std fallback preserved.
Minor Builtins/Frame Adjustments
crates/vm/src/builtins/frame.rs (single change)
Switched one internal fastlocals borrow_mut use to fastlocals_mut().

Sequence Diagram(s)

sequenceDiagram
    participant Caller
    participant VM as VirtualMachine
    participant DS as DataStack
    participant Frame

    Caller->>VM: call function (vectorcall/invoke)
    VM->>DS: datastack_push(size)  -- allocate frame backing
    DS-->>VM: base_ptr
    VM->>Frame: Frame::new(..., use_datastack=true, base_ptr)
    VM->>Frame: run_frame(frame)
    Frame->>DS: (may use datastack-backed LocalsPlus)
    Frame-->>VM: return / unwind
    VM->>Frame: frame.materialize_localsplus()
    VM->>DS: unsafe datastack_pop(base_ptr)  -- cleanup LIFO
    VM-->>Caller: return value
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

Suggested reviewers

  • ShaharNaveh
  • fanninpm
  • coolreader18

Poem

🐰
A tiny stack I build and keep,
Bump and chunk where values sleep.
Frames hop light on LocalsPlus ground,
Push, then pop — LIFO sound.
Hooray, the VM leaps — a joyful bound!

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Title check ✅ Passed The title accurately describes the main changes: removing Frame mutex synchronization and implementing a DataStack bump allocator for LocalsPlus storage, which are the core architectural changes across the changeset.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@youknowone youknowone force-pushed the worktree-frame-datastack branch from a80854b to 01459ec Compare March 3, 2026 15:33
@github-actions
Copy link
Contributor

github-actions bot commented Mar 3, 2026

Code has been automatically formatted

The code in this PR has been formatted using:

  • cargo fmt --all
    Please pull the latest changes before pushing again:
git pull origin worktree-frame-datastack

@youknowone youknowone force-pushed the worktree-frame-datastack branch from f3eee57 to ec12e3d Compare March 3, 2026 20:41
@youknowone youknowone marked this pull request as ready for review March 3, 2026 20:56
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@crates/vm/src/frame.rs`:
- Around line 177-179: Update the stale lifecycle comment on
Frame::new_on_datastack (and the similar comment at the other location) to
reference the current API: instruct callers to call materialize_localsplus() to
migrate localsplus before invoking datastack_pop(), instead of mentioning the
old release_datastack(); keep the intent (drop values and return base pointer)
but replace the function name and order of operations to read something like
"The caller must call materialize_localsplus() when the frame finishes to
migrate localsplus before calling datastack_pop() to drop values and return the
base pointer."
- Around line 288-352: The stack helpers currently use debug_assert! which is
disabled in release and can lead to UB; make their bounds checks run in all
builds by replacing debug_assert! with runtime checks that panic on violation
(e.g., assert! or explicit if-check+panic) in stack_push, stack_pop,
stack_index, and stack_index_mut; keep stack_try_push as the non-panicking
alternative, but ensure stack_push checks idx < self.capacity(), stack_pop
checks self.stack_top > 0, and stack_index/stack_index_mut check idx <
self.stack_top as usize before performing the raw-pointer arithmetic so
out-of-bounds errors produce a controlled panic instead of undefined behavior.

ℹ️ Review info

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between bc108df and ec12e3d.

📒 Files selected for processing (9)
  • .cspell.dict/cpython.txt
  • crates/vm/src/builtins/frame.rs
  • crates/vm/src/builtins/function.rs
  • crates/vm/src/builtins/super.rs
  • crates/vm/src/datastack.rs
  • crates/vm/src/frame.rs
  • crates/vm/src/lib.rs
  • crates/vm/src/vm/mod.rs
  • crates/vm/src/vm/thread.rs

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (2)
crates/vm/src/frame.rs (2)

177-179: ⚠️ Potential issue | 🟡 Minor

Refresh datastack lifecycle comments to the current API.

These comments still instruct callers to use release_datastack(), but the migration path is now materialize_localsplus() before datastack_pop(). Please update both spots so the cleanup contract is accurate.
As per coding guidelines, "Do not delete or rewrite existing comments unless they are factually wrong or directly contradict the new code."

Also applies to: 470-472

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/vm/src/frame.rs` around lines 177 - 179, Update the datastack
lifecycle comments in the docs for the frame constructor functions (e.g.,
new_on_datastack and the other commented spot near the Frame methods) to reflect
the current API: instruct callers to call materialize_localsplus() to
materialize localsplus and then call datastack_pop() (or the
DataStack::datastack_pop equivalent) instead of the outdated
release_datastack(); keep the rest of the comment intact but replace the
release_datastack() reference with the new sequence (materialize_localsplus()
before datastack_pop()) so the cleanup contract matches the migrated API.

290-351: ⚠️ Potential issue | 🔴 Critical

Harden LocalsPlus stack bounds checks for release builds.

stack_push, stack_pop, stack_index, and stack_index_mut rely on debug_assert! before unsafe/raw-pointer access. In release builds, invariant breaks can become out-of-bounds UB instead of controlled failure.

Suggested hardening
@@
     fn stack_push(&mut self, val: Option<PyStackRef>) {
         let idx = self.nlocalsplus as usize + self.stack_top as usize;
-        debug_assert!(
+        assert!(
             idx < self.capacity(),
             "stack overflow: stack_top={}, capacity={}",
             self.stack_top,
             self.stack_capacity()
         );
@@
     fn stack_pop(&mut self) -> Option<PyStackRef> {
-        debug_assert!(self.stack_top > 0, "stack underflow");
+        assert!(self.stack_top > 0, "stack underflow");
         self.stack_top -= 1;
@@
     fn stack_index(&self, idx: usize) -> &Option<PyStackRef> {
-        debug_assert!(idx < self.stack_top as usize);
+        assert!(idx < self.stack_top as usize, "stack index out of bounds");
         let data = self.data_as_slice();
         let raw_idx = self.nlocalsplus as usize + idx;
         unsafe { &*(data.as_ptr().add(raw_idx) as *const Option<PyStackRef>) }
     }
@@
     fn stack_index_mut(&mut self, idx: usize) -> &mut Option<PyStackRef> {
-        debug_assert!(idx < self.stack_top as usize);
+        assert!(idx < self.stack_top as usize, "stack index out of bounds");
         let raw_idx = self.nlocalsplus as usize + idx;
         let data = self.data_as_mut_slice();
         unsafe { &mut *(data.as_mut_ptr().add(raw_idx) as *mut Option<PyStackRef>) }
     }

As per coding guidelines, "Follow Rust best practices for error handling and memory management."

#!/bin/bash
set -euo pipefail

echo "=== stack helper implementations ==="
nl -ba crates/vm/src/frame.rs | sed -n '288,352p'

echo
echo "=== debug_assert usage in stack helpers ==="
rg -n 'fn stack_(push|pop|index|index_mut)\(|debug_assert!\(' crates/vm/src/frame.rs -n -C2
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/vm/src/frame.rs` around lines 290 - 351, Replace reliance on
debug_assert! with explicit runtime checks that panic on violation so release
builds don't hit UB: in stack_push (function stack_push) check idx <
self.capacity() and panic! with the same "stack overflow..." message instead of
debug_assert; in stack_pop (function stack_pop) check self.stack_top > 0 and
panic! with "stack underflow"; in stack_index and stack_index_mut (functions
stack_index and stack_index_mut) check idx < self.stack_top as usize and panic!
with the same invariant message before doing the raw pointer casts; keep
stack_try_push as-is (it already returns Err) and preserve existing messages.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@crates/vm/src/frame.rs`:
- Around line 69-70: Update the lifecycle comment above FrameState to reference
the current field names: replace the stale "Frame.stack" mention with
"Frame.localsplus" (keep the other references to Frame.cells_frees and
Frame.prev_line unchanged) so the comment accurately describes that FrameState
fields are stored directly on Frame as UnsafeCell via Frame.localsplus,
Frame.cells_frees, and Frame.prev_line; ensure wording remains consistent with
safety reasoning and existing style.

---

Duplicate comments:
In `@crates/vm/src/frame.rs`:
- Around line 177-179: Update the datastack lifecycle comments in the docs for
the frame constructor functions (e.g., new_on_datastack and the other commented
spot near the Frame methods) to reflect the current API: instruct callers to
call materialize_localsplus() to materialize localsplus and then call
datastack_pop() (or the DataStack::datastack_pop equivalent) instead of the
outdated release_datastack(); keep the rest of the comment intact but replace
the release_datastack() reference with the new sequence
(materialize_localsplus() before datastack_pop()) so the cleanup contract
matches the migrated API.
- Around line 290-351: Replace reliance on debug_assert! with explicit runtime
checks that panic on violation so release builds don't hit UB: in stack_push
(function stack_push) check idx < self.capacity() and panic! with the same
"stack overflow..." message instead of debug_assert; in stack_pop (function
stack_pop) check self.stack_top > 0 and panic! with "stack underflow"; in
stack_index and stack_index_mut (functions stack_index and stack_index_mut)
check idx < self.stack_top as usize and panic! with the same invariant message
before doing the raw pointer casts; keep stack_try_push as-is (it already
returns Err) and preserve existing messages.

ℹ️ Review info

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ec12e3d and e94f2de.

📒 Files selected for processing (1)
  • crates/vm/src/frame.rs

@youknowone youknowone force-pushed the worktree-frame-datastack branch from e94f2de to 2f61dfc Compare March 4, 2026 05:29
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/vm/src/builtins/function.rs (1)

567-610: ⚠️ Potential issue | 🟠 Major

Datastack allocation is not unwound when argument binding fails.

Line 581 can return early via ? before datastack_pop() is reached, so datastack-backed frame storage is leaked on argument errors for non-generator/coroutine calls.

💡 Suggested fix (scope cleanup for all exits)
         let frame = Frame::new(
             code,
             Scope::new(locals, self.globals.clone()),
             self.builtins.clone(),
             self.closure.as_ref().map_or(&[], |c| c.as_slice()),
             Some(self.to_owned().into()),
             use_datastack,
             vm,
         )
         .into_ref(&vm.ctx);

+        scopeguard::defer! {
+            if use_datastack {
+                unsafe {
+                    if let Some(base) = frame.materialize_localsplus() {
+                        vm.datastack_pop(base);
+                    }
+                }
+            }
+        }
+
         self.fill_locals_from_args(&frame, func_args, vm)?;
         match (is_gen, is_coro) {
@@
-            (false, false) => {
-                let result = vm.run_frame(frame.clone());
-                // Release data stack memory after frame execution completes.
-                unsafe {
-                    if let Some(base) = frame.materialize_localsplus() {
-                        vm.datastack_pop(base);
-                    }
-                }
-                result
-            }
+            (false, false) => vm.run_frame(frame.clone())
         }

As per coding guidelines, "Follow Rust best practices for error handling and memory management."

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/vm/src/builtins/function.rs` around lines 567 - 610, The frame's
datastack can be leaked if fill_locals_from_args returns Err because early
returns skip the explicit vm.datastack_pop; to fix, create a small RAII guard
(e.g., DatastackUnwindGuard) that captures the Frame and Vm reference after
Frame::new and calls vm.datastack_pop(base) from Drop when
materialize_localsplus() yields Some(base), and give it a dismiss() method to
cancel the pop; instantiate this guard immediately after creating the frame and
before calling fill_locals_from_args(&frame,...), call guard.dismiss() in each
successful branch where you already manage the datastack manually (the
generator/coroutine branches and after vm.run_frame in the (false,false) branch
once you pop), and leave the guard to automatically pop on error paths so no
datastack-backed storage is leaked (refer to functions/values: Frame::new,
fill_locals_from_args, frame.materialize_localsplus, vm.datastack_pop).
♻️ Duplicate comments (1)
crates/vm/src/frame.rs (1)

288-352: ⚠️ Potential issue | 🔴 Critical

Harden stack helper bounds checks for release builds.

stack_push, stack_pop, stack_index, and stack_index_mut gate unsafe pointer access with debug_assert!. In release, these checks disappear, so any invariant break can become UB instead of a controlled panic.

Proposed hardening
 fn stack_push(&mut self, val: Option<PyStackRef>) {
     let idx = self.nlocalsplus as usize + self.stack_top as usize;
-    debug_assert!(
+    assert!(
         idx < self.capacity(),
         "stack overflow: stack_top={}, capacity={}",
         self.stack_top,
         self.stack_capacity()
     );
@@
 fn stack_pop(&mut self) -> Option<PyStackRef> {
-    debug_assert!(self.stack_top > 0, "stack underflow");
+    assert!(self.stack_top > 0, "stack underflow");
     self.stack_top -= 1;
@@
 fn stack_index(&self, idx: usize) -> &Option<PyStackRef> {
-    debug_assert!(idx < self.stack_top as usize);
+    assert!(idx < self.stack_top as usize, "stack index out of bounds");
     let data = self.data_as_slice();
     let raw_idx = self.nlocalsplus as usize + idx;
     unsafe { &*(data.as_ptr().add(raw_idx) as *const Option<PyStackRef>) }
 }
@@
 fn stack_index_mut(&mut self, idx: usize) -> &mut Option<PyStackRef> {
-    debug_assert!(idx < self.stack_top as usize);
+    assert!(idx < self.stack_top as usize, "stack index out of bounds");
     let raw_idx = self.nlocalsplus as usize + idx;
     let data = self.data_as_mut_slice();
     unsafe { &mut *(data.as_mut_ptr().add(raw_idx) as *mut Option<PyStackRef>) }
 }
#!/bin/bash
set -euo pipefail

echo "=== Stack helpers using debug-only checks ==="
nl -ba crates/vm/src/frame.rs | sed -n '288,352p'

echo
echo "=== Nearby subtraction-based stack indexing callsites ==="
rg -n 'stack_len\s*-\s*|stack_index\(' crates/vm/src/frame.rs | sed -n '1,80p'

Expected result: Line 288-352 still shows debug_assert! in helper methods guarding unsafe accesses, confirming release-mode check elision risk.

As per coding guidelines, "Follow Rust best practices for error handling and memory management".

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/vm/src/frame.rs` around lines 288 - 352, The helper methods
(stack_push, stack_try_push already checks, stack_pop, stack_index,
stack_index_mut) use debug_assert! which is compiled out in release causing
potential UB; change those debug-only guards into release-time checks (use
assert! or explicit if .. { panic!(...) }) before performing unsafe pointer
accesses and ensure stack_push mirrors stack_try_push’s capacity check (panic on
overflow) so all unsafe transmute/slice/from_raw_parts calls are protected in
release builds.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@crates/vm/Cargo.toml`:
- Line 13: The default features list in crates/vm/Cargo.toml now enables
"threading" (the line default = ["compiler", "wasmbind", "gc", "host_env",
"stdio", "threading"]), which forces threading-related code and synchronization
overhead on all consumers; either remove "threading" from that default array so
it becomes opt-in, or add clear documentation and a commit message explaining
why threading must be default and the tradeoffs (including which modules like
stdlib.thread, core VM frame tracking, and Arc vs Rc changes are affected).
After changing the default, run cargo build/test with and without defaults to
confirm no accidental breakage and update READMEs or the crate-level
documentation to describe the threading feature and recommended usage for
single-threaded targets.

In `@crates/vm/src/datastack.rs`:
- Around line 77-79: The arithmetic for computing aligned sizes and growing
chunk_size in datastack's allocator (functions has_space(), push(), push_slow())
must be changed to checked arithmetic to avoid overflow before any pointer
.add() or subtraction: replace (size + ALIGN - 1) with a checked sequence using
size.checked_add(ALIGN - 1).and_then(|v| v.checked_mul(1)) or more simply
size.checked_add(ALIGN - 1).and_then(|v| v.checked_sub(v % ALIGN)) to compute
aligned_size (or use checked_add then checked_sub of remainder), bail/panic with
a clear message on None; in push() and has_space() use that checked aligned_size
and return/panic on overflow rather than letting wrap occur; in push_slow()
compute aligned_size and the total allocation request via checked_add for
MINIMUM_OVERHEAD and other offsets, and replace chunk_size *= 2 with chunk_size
= chunk_size.checked_mul(2).expect("overflow while growing chunk_size") so any
overflow panics before pointer arithmetic occurs. Ensure all uses of
aligned_size and chunk_size later in pointer.add()/offset math use the checked
results.

---

Outside diff comments:
In `@crates/vm/src/builtins/function.rs`:
- Around line 567-610: The frame's datastack can be leaked if
fill_locals_from_args returns Err because early returns skip the explicit
vm.datastack_pop; to fix, create a small RAII guard (e.g., DatastackUnwindGuard)
that captures the Frame and Vm reference after Frame::new and calls
vm.datastack_pop(base) from Drop when materialize_localsplus() yields
Some(base), and give it a dismiss() method to cancel the pop; instantiate this
guard immediately after creating the frame and before calling
fill_locals_from_args(&frame,...), call guard.dismiss() in each successful
branch where you already manage the datastack manually (the generator/coroutine
branches and after vm.run_frame in the (false,false) branch once you pop), and
leave the guard to automatically pop on error paths so no datastack-backed
storage is leaked (refer to functions/values: Frame::new, fill_locals_from_args,
frame.materialize_localsplus, vm.datastack_pop).

---

Duplicate comments:
In `@crates/vm/src/frame.rs`:
- Around line 288-352: The helper methods (stack_push, stack_try_push already
checks, stack_pop, stack_index, stack_index_mut) use debug_assert! which is
compiled out in release causing potential UB; change those debug-only guards
into release-time checks (use assert! or explicit if .. { panic!(...) }) before
performing unsafe pointer accesses and ensure stack_push mirrors
stack_try_push’s capacity check (panic on overflow) so all unsafe
transmute/slice/from_raw_parts calls are protected in release builds.

ℹ️ Review info
Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 56f1f9fe-c928-4835-94bd-a1f6f5d80c40

📥 Commits

Reviewing files that changed from the base of the PR and between e94f2de and 42a2a37.

📒 Files selected for processing (10)
  • .cspell.dict/cpython.txt
  • crates/vm/Cargo.toml
  • crates/vm/src/builtins/frame.rs
  • crates/vm/src/builtins/function.rs
  • crates/vm/src/builtins/super.rs
  • crates/vm/src/datastack.rs
  • crates/vm/src/frame.rs
  • crates/vm/src/lib.rs
  • crates/vm/src/vm/mod.rs
  • crates/vm/src/vm/thread.rs
🚧 Files skipped from review as they are similar to previous changes (4)
  • .cspell.dict/cpython.txt
  • crates/vm/src/builtins/super.rs
  • crates/vm/src/vm/thread.rs
  • crates/vm/src/lib.rs


[features]
default = ["compiler", "wasmbind", "gc", "host_env", "stdio"]
default = ["compiler", "wasmbind", "gc", "host_env", "stdio", "threading"]
Copy link
Contributor

@coderabbitai coderabbitai bot Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if there are any docs or CHANGELOG entries mentioning this default feature change
echo "=== Searching for docs/CHANGELOG entries about threading or default features ==="
find . -type f \( -name "*.md" -o -name "*.rst" -o -name "CHANGELOG*" \) 2>/dev/null | head -20

echo -e "\n=== Checking for mentions of threading in docs ==="
fd -e md -e rst 2>/dev/null | xargs rg -l "threading" 2>/dev/null | head -10

echo -e "\n=== Checking for mentions of default features in docs ==="
fd -e md -e rst 2>/dev/null | xargs rg -l "default.*feature" 2>/dev/null | head -10

Repository: RustPython/RustPython

Length of output: 918


🏁 Script executed:

#!/bin/bash
# Verify if datastack module has threading dependencies
echo "=== Checking if datastack.rs exists and its threading dependencies ==="
if [ -f "crates/vm/src/datastack.rs" ]; then
  echo "Found datastack.rs"
  rg -n "cfg.*threading" crates/vm/src/datastack.rs 2>/dev/null || echo "No threading cfg in datastack.rs"
  echo -e "\n=== Checking datastack.rs size ==="
  wc -l crates/vm/src/datastack.rs
else
  echo "datastack.rs not found at crates/vm/src/datastack.rs"
  find . -name "datastack.rs" -type f 2>/dev/null
fi

Repository: RustPython/RustPython

Length of output: 254


🏁 Script executed:

#!/bin/bash
# Check if there are tests that explicitly disable threading
echo "=== Tests/configs with default-features disabled ==="
rg -n 'default-features\s*=\s*false' --type toml 2>/dev/null | head -20

echo -e "\n=== Checking Cargo.toml for threading feature definition ==="
rg -A 5 "^\[features\]" crates/vm/Cargo.toml

Repository: RustPython/RustPython

Length of output: 2257


🏁 Script executed:

#!/bin/bash
# Check git history for this specific line change
echo "=== Recent commits affecting line 13 in crates/vm/Cargo.toml ==="
git log --oneline -n 20 -- crates/vm/Cargo.toml 2>/dev/null

echo -e "\n=== Check if there's a commit message explaining this ==="
git log -p -n 1 --grep="threading" -- crates/vm/Cargo.toml 2>/dev/null || echo "No matching commit found"

Repository: RustPython/RustPython

Length of output: 226


🏁 Script executed:

#!/bin/bash
# Find all cfg(feature = "threading") gates in the codebase
echo "=== All uses of threading feature guard ==="
rg '#\[cfg\(.*feature.*=.*"threading"' -A 2 | head -40

echo -e "\n=== Count of threading feature gates ==="
rg '#\[cfg\(.*feature.*=.*"threading"' | wc -l

# Check if threading gate is on module level
echo -e "\n=== Module-level threading gates ==="
rg '^#\[cfg\(feature = "threading"\)\]' -B 2

Repository: RustPython/RustPython

Length of output: 6753


🏁 Script executed:

#!/bin/bash
# Check what the threading feature actually gates
echo "=== Searching for optional threading dependencies in Cargo.toml ==="
cat crates/vm/Cargo.toml | grep -A 30 "\[features\]"

echo -e "\n=== Check if threading is an empty feature or has deps ==="
rg "threading" crates/vm/Cargo.toml

Repository: RustPython/RustPython

Length of output: 1396


🏁 Script executed:

#!/bin/bash
# Look at the actual diff/context around the change
echo "=== Check recent changes to default features ==="
git log -p --all -S "threading" -- crates/vm/Cargo.toml 2>/dev/null | head -50 || echo "No threading-related commits"

# Check if there's a related PR or branch
echo -e "\n=== Check git branches for datastack ==="
git branch -a 2>/dev/null | grep -i datastack | head -10 || echo "No datastack branches"

# Get git show for the current state
echo -e "\n=== Current git diff for crates/vm/Cargo.toml ==="
git diff HEAD -- crates/vm/Cargo.toml 2>/dev/null || echo "No uncommitted changes"

Repository: RustPython/RustPython

Length of output: 206


Document or reconsider adding threading to default features.

This change enables threading-related code throughout the VM by default—not just for datastack (which has no threading guards). The threading feature gates 101+ code paths including:

  • Entire stdlib.thread module and sys._current_frames()/sys._current_exceptions()
  • Global thread registry and frame tracking in core VM
  • Arc vs Rc usage across rustpython-common (replacing atomic-free reference counting)
  • Thread-safe Send implementations for frame types

All users building with default features will now incur the synchronization overhead of threading, even single-threaded embedded use cases. Examples like wasm32_without_js disable defaults precisely to avoid this.

No commit message or documentation explains why threading was added to defaults or its necessity. If this was intentional, clarify the rationale and impact. If unintentional, consider reverting or making it opt-in.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/vm/Cargo.toml` at line 13, The default features list in
crates/vm/Cargo.toml now enables "threading" (the line default = ["compiler",
"wasmbind", "gc", "host_env", "stdio", "threading"]), which forces
threading-related code and synchronization overhead on all consumers; either
remove "threading" from that default array so it becomes opt-in, or add clear
documentation and a commit message explaining why threading must be default and
the tradeoffs (including which modules like stdlib.thread, core VM frame
tracking, and Arc vs Rc changes are affected). After changing the default, run
cargo build/test with and without defaults to confirm no accidental breakage and
update READMEs or the crate-level documentation to describe the threading
feature and recommended usage for single-threaded targets.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

False positive — threading is not in the default features. Line 13 of crates/vm/Cargo.toml:

default = ["compiler", "wasmbind", "gc", "host_env", "stdio"]

No change was made to defaults in this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@youknowone, looking at the diff included in this PR, threading does appear to be added to the default features:

-default = ["compiler", "wasmbind", "gc", "host_env", "stdio"]
+default = ["compiler", "wasmbind", "gc", "host_env", "stdio", "threading"]

The file content shown in the review context reflects the base branch (main) state — which does not have threading in defaults. The PR branch (worktree-frame-datastack) introduces the change. The - line is what was there before, and the + line is what this PR introduces.

So the review comment stands — could you clarify whether adding threading to default features is intentional as part of this PR?

@youknowone youknowone force-pushed the worktree-frame-datastack branch from 42a2a37 to 3c00316 Compare March 4, 2026 05:54
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

♻️ Duplicate comments (1)
crates/vm/src/frame.rs (1)

290-352: ⚠️ Potential issue | 🔴 Critical

Harden stack helper bounds checks in release builds.

LocalsPlus::stack_push, stack_pop, stack_index, and stack_index_mut still rely on debug_assert! before unsafe pointer-based access. In release builds, invariant violations can become UB instead of a controlled failure.

Suggested hardening
 fn stack_push(&mut self, val: Option<PyStackRef>) {
     let idx = self.nlocalsplus as usize + self.stack_top as usize;
-    debug_assert!(
+    assert!(
         idx < self.capacity(),
         "stack overflow: stack_top={}, capacity={}",
         self.stack_top,
         self.stack_capacity()
     );
@@
 fn stack_pop(&mut self) -> Option<PyStackRef> {
-    debug_assert!(self.stack_top > 0, "stack underflow");
+    assert!(self.stack_top > 0, "stack underflow");
@@
 fn stack_index(&self, idx: usize) -> &Option<PyStackRef> {
-    debug_assert!(idx < self.stack_top as usize);
+    assert!(idx < self.stack_top as usize, "stack index out of bounds");
@@
 fn stack_index_mut(&mut self, idx: usize) -> &mut Option<PyStackRef> {
-    debug_assert!(idx < self.stack_top as usize);
+    assert!(idx < self.stack_top as usize, "stack index out of bounds");
#!/bin/bash
set -euo pipefail

echo "Inspect stack helper implementations:"
nl -ba crates/vm/src/frame.rs | sed -n '288,352p'

echo
echo "Spot stack_len - N arithmetic call-sites that depend on these checks:"
rg -nP --type=rust 'stack_len\s*-\s*|stack_index\(' crates/vm/src/frame.rs | head -n 80

As per coding guidelines, "Follow Rust best practices for error handling and memory management."

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/vm/src/frame.rs` around lines 290 - 352, The stack helpers use
debug_assert! which is compiled out in release and can lead to UB; replace those
debug-only asserts with explicit runtime checks that panic with clear messages
before performing any unsafe pointer or transmute operations. Concretely: in
stack_push check idx < self.capacity() and panic on overflow (stack_push still
panics; stack_try_push remains as-is), in stack_pop check self.stack_top > 0 and
panic on underflow before decrementing/reading, and in stack_index and
stack_index_mut check idx < self.stack_top and that raw_idx < self.capacity()
(or otherwise ensure the computed pointer is valid) and panic with a descriptive
message before casting the slice pointer returned by
data_as_slice()/data_as_mut_slice(); perform these checks immediately prior to
any unsafe use so no unsafe transmute/pointer arithmetic runs when invariants
are violated.
🧹 Nitpick comments (2)
crates/vm/src/gc_state.rs (1)

831-841: Consider adding a module-level documentation note about thread-local behavior in non-threading mode.

The thread-local GcState in non-threading mode is correctly implemented: static_cell! expands to thread_local! when the threading feature is disabled (confirmed at crates/common/src/static_cell.rs:113-125). This means each thread gets its own independent GcState, which is appropriate for satisfying !Sync constraints.

This behavior is already documented inline at line 139: "In non-threading mode, gc_state() is thread-local so Sync is not needed." However, adding a brief note to the module-level docs (lines 1-4) would help maintainers understand this design decision upfront. The reinit_after_fork() method is safely gated to threading-only mode, and callers do not make cross-thread assumptions about GC state visibility.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/vm/src/gc_state.rs` around lines 831 - 841, Add a short module-level
documentation note at the top of the gc_state.rs module explaining that
gc_state() returns a true global When the `threading` feature is enabled but
expands to a thread-local `GcState` when `threading` is disabled (because
`rustpython_common::static_cell!` becomes `thread_local!`), and that this design
satisfies `!Sync` constraints; reference the `gc_state()` function, the
`GcState` type, the `static_cell!` macro, and mention that `reinit_after_fork()`
is gated to threading-only behavior so callers must not assume cross-thread
visibility of GC state.
crates/vm/src/frame.rs (1)

195-207: Mark materialization result as #[must_use] to prevent lifecycle mistakes.

Ignoring the returned base pointer makes it easy to skip datastack_pop, which can leak datastack chunks.

Suggested hardening
+#[must_use = "call VirtualMachine::datastack_pop() with the returned base pointer"]
 fn materialize_to_heap(&mut self) -> Option<*mut u8> {
@@
+#[must_use = "call VirtualMachine::datastack_pop() with the returned base pointer"]
 pub(crate) unsafe fn materialize_localsplus(&self) -> Option<*mut u8> {

Also applies to: 736-738

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/vm/src/frame.rs` around lines 195 - 207, Mark the pointer-returning
materialization function with #[must_use] so callers cannot ignore the returned
base pointer; specifically add #[must_use] to the fn materialize_to_heap(&mut
self) -> Option<*mut u8> declaration and likewise annotate any other methods
that return a raw pointer from LocalsPlusData::DataStack (the same pattern
flagged around the other occurrence) to ensure callers handle
datastack_pop/lifecycle cleanup.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@crates/common/src/lock.rs`:
- Around line 31-33: Clarify and reinforce the unsafe Sync impl for LazyLock by
updating the doc comment near the LazyLock<T, F> declaration and the unsafe
impl<T, F> Sync for LazyLock<T, F> to state explicitly that this impl is only
safe when both the "std" and "threading" features are absent (i.e., truly
single-threaded embedded/WASM-only targets), and either add a compile-time
assertion or a marker trait check to prevent this impl being used when either
feature is enabled; mention the exact symbols LazyLock and the unsafe impl Sync
so reviewers can find and update the documentation and/or add the compile-time
guard.

In `@crates/vm/src/frame.rs`:
- Around line 165-188: The constructors for LocalsPlus (fn new and fn
new_on_datastack) perform unchecked arithmetic and casts: replace nlocalsplus +
stacksize with checked_add(...).expect(...) to detect overflow, replace capacity
* size_of::<usize>() with checked_mul(...).expect(...), and replace the usize ->
u32 casts for nlocalsplus with u32::try_from(...).expect(...) to avoid silent
truncation; update both new and new_on_datastack paths and include clear expect
messages mentioning the function/operation (e.g., "LocalsPlus::new: capacity
overflow") to aid debugging while preserving the existing behaviors on success.

---

Duplicate comments:
In `@crates/vm/src/frame.rs`:
- Around line 290-352: The stack helpers use debug_assert! which is compiled out
in release and can lead to UB; replace those debug-only asserts with explicit
runtime checks that panic with clear messages before performing any unsafe
pointer or transmute operations. Concretely: in stack_push check idx <
self.capacity() and panic on overflow (stack_push still panics; stack_try_push
remains as-is), in stack_pop check self.stack_top > 0 and panic on underflow
before decrementing/reading, and in stack_index and stack_index_mut check idx <
self.stack_top and that raw_idx < self.capacity() (or otherwise ensure the
computed pointer is valid) and panic with a descriptive message before casting
the slice pointer returned by data_as_slice()/data_as_mut_slice(); perform these
checks immediately prior to any unsafe use so no unsafe transmute/pointer
arithmetic runs when invariants are violated.

---

Nitpick comments:
In `@crates/vm/src/frame.rs`:
- Around line 195-207: Mark the pointer-returning materialization function with
#[must_use] so callers cannot ignore the returned base pointer; specifically add
#[must_use] to the fn materialize_to_heap(&mut self) -> Option<*mut u8>
declaration and likewise annotate any other methods that return a raw pointer
from LocalsPlusData::DataStack (the same pattern flagged around the other
occurrence) to ensure callers handle datastack_pop/lifecycle cleanup.

In `@crates/vm/src/gc_state.rs`:
- Around line 831-841: Add a short module-level documentation note at the top of
the gc_state.rs module explaining that gc_state() returns a true global When the
`threading` feature is enabled but expands to a thread-local `GcState` when
`threading` is disabled (because `rustpython_common::static_cell!` becomes
`thread_local!`), and that this design satisfies `!Sync` constraints; reference
the `gc_state()` function, the `GcState` type, the `static_cell!` macro, and
mention that `reinit_after_fork()` is gated to threading-only behavior so
callers must not assume cross-thread visibility of GC state.

ℹ️ Review info
Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 4f9a0b14-3c1d-49d6-ad14-a98933e00e4a

📥 Commits

Reviewing files that changed from the base of the PR and between 42a2a37 and 859b6f7.

📒 Files selected for processing (3)
  • crates/common/src/lock.rs
  • crates/vm/src/frame.rs
  • crates/vm/src/gc_state.rs

@youknowone youknowone force-pushed the worktree-frame-datastack branch from 859b6f7 to 09a5a5e Compare March 4, 2026 07:25
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/vm/src/builtins/function.rs (1)

579-607: ⚠️ Potential issue | 🟠 Major

Datastack allocation can leak on argument-binding errors.

At Line 579, fill_locals_from_args(...)? can return early while use_datastack is true, but cleanup currently only happens in the (false, false) arm (Lines 599-607). That leaves a datastack allocation unpopped on common TypeError paths.

Proposed fix (guard cleanup for all exits when datastack is used)
         let frame = Frame::new(
             code,
             Scope::new(locals, self.globals.clone()),
             self.builtins.clone(),
             self.closure.as_ref().map_or(&[], |c| c.as_slice()),
             Some(self.to_owned().into()),
             use_datastack,
             vm,
         )
         .into_ref(&vm.ctx);

+        let _datastack_cleanup = use_datastack.then(|| {
+            let frame = frame.clone();
+            scopeguard::guard((), move |_| unsafe {
+                if let Some(base) = frame.materialize_localsplus() {
+                    vm.datastack_pop(base);
+                }
+            })
+        });
+
         self.fill_locals_from_args(&frame, func_args, vm)?;
         match (is_gen, is_coro) {
@@
-            (false, false) => {
-                let result = vm.run_frame(frame.clone());
-                // Release data stack memory after frame execution completes.
-                unsafe {
-                    if let Some(base) = frame.materialize_localsplus() {
-                        vm.datastack_pop(base);
-                    }
-                }
-                result
-            }
+            (false, false) => vm.run_frame(frame.clone()),
         }

As per coding guidelines: "Follow Rust best practices for error handling and memory management".

♻️ Duplicate comments (1)
crates/vm/src/datastack.rs (1)

77-90: ⚠️ Potential issue | 🔴 Critical

Use checked alignment math and avoid pre-check .add() UB in the fast path.

Line 77 and Line 88 still use unchecked (size + ALIGN - 1), and Line 90 performs self.top.add(aligned_size) in the condition. A large size can wrap aligned_size, and .add() must only be used with a proven in-bounds offset.

Suggested fix
 impl DataStack {
+    #[inline(always)]
+    fn align_size(size: usize) -> usize {
+        size.checked_add(ALIGN - 1)
+            .map(|v| v & !(ALIGN - 1))
+            .expect("DataStack allocation size overflow")
+    }
+
     /// Check if the current chunk has at least `size` bytes available.
     #[inline(always)]
     pub fn has_space(&self, size: usize) -> bool {
-        let aligned_size = (size + ALIGN - 1) & !(ALIGN - 1);
+        let aligned_size = Self::align_size(size);
         (self.limit as usize).saturating_sub(self.top as usize) >= aligned_size
     }
@@
     #[inline(always)]
     pub fn push(&mut self, size: usize) -> *mut u8 {
-        let aligned_size = (size + ALIGN - 1) & !(ALIGN - 1);
+        let aligned_size = Self::align_size(size);
         unsafe {
-            if self.top.add(aligned_size) <= self.limit {
+            let available = (self.limit as usize).saturating_sub(self.top as usize);
+            if available >= aligned_size {
                 let ptr = self.top;
                 self.top = self.top.add(aligned_size);
                 ptr
             } else {
                 self.push_slow(aligned_size)
             }
         }
     }

As per coding guidelines: "Follow Rust best practices for error handling and memory management".

#!/bin/bash
set -euo pipefail

# Verify unchecked alignment arithmetic and pointer add in fast path.
rg -nP '\(size \+ ALIGN - 1\)|\.add\(aligned_size\)' crates/vm/src/datastack.rs

# Show the exact fast-path blocks for manual inspection.
sed -n '72,98p' crates/vm/src/datastack.rs
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/vm/src/datastack.rs` around lines 77 - 90, The push method currently
uses unchecked alignment math `(size + ALIGN - 1)` and calls
`self.top.add(aligned_size)` in the conditional, which can overflow/wrap for
large `size` and invokes .add() without proven in-bounds offsets; fix push in
datastack.rs by (1) computing aligned_size with checked arithmetic (use
checked_add/checked_sub/checked_mul or a helper that returns None on overflow)
and return a null or handle allocation failure on overflow, and (2) replace the
conditional that uses `self.top.add(aligned_size) <= self.limit` with a
bounds-safe check (compute a checked pointer offset or compare sizes using
`(self.limit as usize).saturating_sub(self.top as usize) >= aligned_size` or use
`offset`/`wrapping_offset` only after verifying aligned_size fits) so .add() is
only called when the offset is proven in-bounds; update push to return early on
overflow and ensure pop still receives a valid pointer.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@crates/common/src/lock.rs`:
- Around line 27-30: The build breaks when the "threading" feature is enabled
without "std" because lock.rs unconditionally re-exports std::sync::OnceLock and
std::sync::LazyLock; either make the feature graph include std by changing the
Cargo.toml entry threading = ["parking_lot"] to threading = ["parking_lot",
"std"], or add a compile-time guard in lock.rs that detects cfg(all(feature =
"threading", not(feature = "std"))) and emits compile_error! with a clear
message; locate the uses of OnceLock and LazyLock in lock.rs and apply one of
these fixes so the crate cannot enable threading without std.

In `@crates/vm/src/frame.rs`:
- Line 675: In Frame::new replace the unchecked addition that computes
nlocalsplus with checked arithmetic: compute nlocalsplus by using .checked_add
(e.g., first nlocals.checked_add(num_cells) and then .checked_add(nfrees)) and
handle the None case by returning an appropriate error (or propagating a Result)
instead of allowing wrapping; reference the variables nlocals, num_cells, nfrees
and the function Frame::new so you locate the calculation and ensure subsequent
uses of nlocalsplus assume a valid non-wrapped value.

In `@crates/vm/src/gc_state.rs`:
- Around line 831-840: The gc_state() helper currently uses
rustpython_common::static_cell! which becomes thread-local when the "threading"
feature is disabled, causing per-thread GcState sets and breaking global
allocation tracking; change gc_state() to initialize a single process-wide
static (e.g., a OnceLock or equivalent global static) so it always returns the
same &'static GcState across threads regardless of the "threading" feature,
using GcState::new for initialization; update the static declaration in
gc_state() (replace static_cell! usage) so allocation/GC logic (calls to
gc_state() from object/core.rs) see one shared GcState.
- Around line 137-142: The unconditional unsafe impl Send for GcState is unsound
in non-threading builds; change the declaration so Send is only implemented when
the threading feature is enabled (mirror the existing #[cfg(feature =
"threading")] used for Sync). Update the unsafe impl Send for GcState to be
gated with #[cfg(feature = "threading")] so that both Send and Sync are only
provided in threading builds, keeping the safety comment and behavior of GcState
unchanged.

---

Duplicate comments:
In `@crates/vm/src/datastack.rs`:
- Around line 77-90: The push method currently uses unchecked alignment math
`(size + ALIGN - 1)` and calls `self.top.add(aligned_size)` in the conditional,
which can overflow/wrap for large `size` and invokes .add() without proven
in-bounds offsets; fix push in datastack.rs by (1) computing aligned_size with
checked arithmetic (use checked_add/checked_sub/checked_mul or a helper that
returns None on overflow) and return a null or handle allocation failure on
overflow, and (2) replace the conditional that uses `self.top.add(aligned_size)
<= self.limit` with a bounds-safe check (compute a checked pointer offset or
compare sizes using `(self.limit as usize).saturating_sub(self.top as usize) >=
aligned_size` or use `offset`/`wrapping_offset` only after verifying
aligned_size fits) so .add() is only called when the offset is proven in-bounds;
update push to return early on overflow and ensure pop still receives a valid
pointer.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: d2e83ddf-e6e7-4b27-b9dd-ea4f192513a6

📥 Commits

Reviewing files that changed from the base of the PR and between 859b6f7 and 09a5a5e.

📒 Files selected for processing (11)
  • .cspell.dict/cpython.txt
  • crates/common/src/lock.rs
  • crates/vm/src/builtins/frame.rs
  • crates/vm/src/builtins/function.rs
  • crates/vm/src/builtins/super.rs
  • crates/vm/src/datastack.rs
  • crates/vm/src/frame.rs
  • crates/vm/src/gc_state.rs
  • crates/vm/src/lib.rs
  • crates/vm/src/vm/mod.rs
  • crates/vm/src/vm/thread.rs
🚧 Files skipped from review as they are similar to previous changes (4)
  • crates/vm/src/vm/thread.rs
  • .cspell.dict/cpython.txt
  • crates/vm/src/builtins/frame.rs
  • crates/vm/src/lib.rs

@youknowone youknowone force-pushed the worktree-frame-datastack branch from 7bb9cae to 3af0cf8 Compare March 4, 2026 11:05
Copy link
Member Author

@youknowone youknowone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressing review comments

@youknowone youknowone changed the title frame datastack Remove Frame mutex and use DataStack bump allocator for LocalsPlus Mar 4, 2026
Move stack, cells_frees, prev_line out of the mutex-protected FrameState
into Frame as FrameUnsafeCell fields. This eliminates mutex lock/unlock
overhead on every frame execution (with_exec).

Safety relies on the same single-threaded execution guarantee that
FastLocals already uses.
Introduce DataStack with linked chunks (16KB initial, doubling) and
push/pop bump allocation. Add datastack field to VirtualMachine.
Not yet wired to frame creation.
Replace separate FastLocals (Box<[Option<PyObjectRef>]>) and
BoxVec<Option<PyStackRef>> with a single LocalsPlus struct that
stores both in a contiguous Box<[usize]> array. The first
nlocalsplus slots are fastlocals and the rest is the evaluation
stack. Typed access is provided through transmute-based methods.

Remove BoxVec import from frame.rs.
Normal function calls now bump-allocate LocalsPlus data from the
per-thread DataStack instead of a separate heap allocation.
Generator/coroutine frames continue using heap allocation since
they outlive the call.

On frame exit, data is copied to the heap (materialize_to_heap)
to preserve locals for tracebacks, then the DataStack is popped.

VirtualMachine.datastack is wrapped in UnsafeCell for interior
mutability (safe because frame allocation is single-threaded LIFO).
Update vectorcall dispatch functions to use localsplus stack
accessors instead of direct stack field access. Add
stack_truncate method to LocalsPlus. Update vectorcall fast
path in function.rs to use datastack and fastlocals_mut().
Check both bounds of the current chunk when determining if a
pop base is in the current chunk. The previous check (base >=
chunk_start) fails on Windows where newer chunks may be
allocated at lower addresses than older ones.
Two fixes for Cell-based types used in static items under non-threading
mode, which cause data races when Rust test runner uses parallel threads:

1. LazyLock: use std::sync::LazyLock when std is available instead of
   wrapping core::cell::LazyCell with a false `unsafe impl Sync`.
   The LazyCell wrapper is kept only for no-std (truly single-threaded).

2. gc_state: use static_cell! (thread-local in non-threading mode)
   instead of OnceLock, so each thread gets its own GcState with
   Cell-based PyRwLock/PyMutex that are not accessed concurrently.
… gate

- Use checked arithmetic for nlocalsplus in Frame::new
- Add "std" to threading feature dependencies in rustpython-common
- Gate GcState Send impl with #[cfg(feature = "threading")]
@youknowone youknowone force-pushed the worktree-frame-datastack branch from 4051d4c to 22df0c6 Compare March 4, 2026 12:15
@youknowone youknowone merged commit 68ad332 into RustPython:main Mar 4, 2026
13 checks passed
@youknowone youknowone deleted the worktree-frame-datastack branch March 4, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant