Skip to content

Preserve tokenizer-only compiler warning order #9316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
youknowone merged 2 commits into from
Oct 23, 2026
Merged

Preserve tokenizer-only compiler warning order #9316

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
11c8175
Preserve tokenizer-only compiler warning order
youknowdot Oct 22, 2026
78a96fd
Merge main and preserve tokenizer-only warning ordering
youknowdot Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions crates/vm/src/vm/compile.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1868,6 +1868,7 @@ mod escape_warnings {
inherited_barry_as_flufl: bool,
emitted_escapes: &[usize],
) -> Result<(), CompileWarningError> {
let tokenizer_only_mode = matches!(mode, compiler::Mode::Exec | compiler::Mode::Eval);
let is_single = matches!(mode, compiler::Mode::Single);
let mode = match mode {
compiler::Mode::Eval => ruff_python_parser::Mode::Expression,
Expand All @@ -1882,9 +1883,21 @@ mod escape_warnings {
inherited_barry_as_flufl,
is_single,
);
if !collected.has_valid_syntax || !collected.tokenizer_recovery {
// Invalid source and interpolated tokenizer warnings need a
// separate recovery contract. Preserve their existing behavior.
// Tokenizer-only batches can preserve lexical warning order without
// parser-error recovery. Mixed parser/tokenizer events still need
// their separate fallback contract, as does Single's input boundary.
let tokenizer_only = tokenizer_only_mode

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include interpolation numeric events before batching

When an exec/eval t-string contains a numeric-keyword warning before a later escaped brace—for example compile("(t'{1is 2}', t'\\{{')", ..., "eval") with SyntaxWarning promoted to an error—this predicate still labels the batch tokenizer-only. collect_numeric_literal_warnings skips every quoted literal wholesale, including its interpolation body, and EscapeWarningVisitor does not add numeric events, so the new path raises the later \{ warning instead of CPython's earlier invalid decimal literal at 1is. Include interpolation-body numeric events (or exclude these batches) before using the collected vector for warning order.

Useful? React with 👍 / 👎.

&& collected.warnings.iter().any(|warning| {
matches!(
warning,
ParserWarning::Escape(_, EscapeOrigin::Tokenizer, _)
)
})
&& collected.warnings.iter().all(|warning| {
!matches!(warning, ParserWarning::Escape(_, EscapeOrigin::Parser, _))
});
if !collected.has_valid_syntax || (!collected.tokenizer_recovery && !tokenizer_only) {
// Preserve invalid-source and mixed-origin fallback behavior.
self.emit_tokenizer_syntax_warnings(source, filename, module)?;
return self.emit_string_escape_warnings(source, filename, module, emitted_escapes);
}
Expand All @@ -1896,7 +1909,7 @@ mod escape_warnings {
collected.single_boundary,
WarningEmission::Standard {
emitted_escapes,
tokenizer_recovery: true,
tokenizer_recovery: !tokenizer_only,
},
)
.map(|_| ())
Expand Down
Loading