Skip to content

Validate symbols against advertised object files - #1200

Open
mturac wants to merge 4 commits into
astral-sh:mainfrom
mturac:fix/issue-459
Open

Validate symbols against advertised object files#1200
mturac wants to merge 4 commits into
astral-sh:mainfrom
mturac:fix/issue-459

Conversation

@mturac

@mturac mturac commented Aug 4, 2026

Copy link
Copy Markdown

Summary

  • collect Python symbols from native libpython binaries and the object files advertised by PYTHON.json
  • report exported symbols, plus internally required ELF and Mach-O symbols, that cannot be reproduced from the advertised object set
  • validate advertised object format, kind, architecture, and endianness; recognize LTO bitcode containers without claiming unsupported symbol coverage

Closes #459.

Test plan

  • cargo fmt --check
  • cargo test --locked
  • cargo build --release --locked
  • cargo clippy --locked --all-targets
  • ./check.py
  • validated the 2024-10-16 regression artifact and observed exactly the nine missing tokenizer symbols
  • validated current 2026-07-28 debug and PGO+LTO artifacts without new errors

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 63104006d9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/validation.rs
Comment on lines +2297 to +2298
let supports_object_symbol_validation =
matches!(object_file_format.as_str(), "coff" | "elf" | "mach-o");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Match the declared object format to the target

This treats every recognized format as valid for every target, without comparing it to target_object_format(triple). For example, an x86-64 Linux archive that declares coff and contains x86-64 COFF objects can pass the subsequent kind, architecture, endianness, and symbol checks even though downstream Linux linkers cannot consume those advertised objects.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f7b1559.

Comment thread src/validation.rs Outdated
Comment on lines +2344 to +2346
if !is_static
&& (supports_object_symbol_validation || is_llvm_bitcode_format)
&& advertised_object_paths.contains(&path)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate advertised objects in static archives

For every +static distribution this guard skips collect_advertised_object_symbols, including its parsing, relocatable-kind, format, architecture, and endianness checks. Consequently, a referenced object in a static archive can contain arbitrary bytes or an unusable format (especially Windows COFF objects, which validate_possible_object_file otherwise ignores) and validation only confirms that its path exists.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f7b1559.

@jjhelmus

jjhelmus commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

validated the 2024-10-16 regression artifact and observed exactly the nine missing tokenizer symbols

Can you expand on this? What regression was detected?

mturac added 2 commits August 10, 2026 19:40
Treat structurally valid ANON_OBJECT_HEADER v1 files as partial symbol
coverage when validating Windows distributions. Keep malformed headers on
the existing parse-error path.
@mturac

mturac commented Aug 10, 2026

Copy link
Copy Markdown
Author

The regression is #378. The 2024-10-16 x86_64 Linux artifact ships a libpython3.13.so.1.0 that references nine tokenizer symbols, but none of the object paths advertised by PYTHON.json define them. The validator now reports those exact nine symbols, including _PyToken_Init, _PyTokenizer_FromFile, and _PyTokenizer_Get. This reproduces why rebuilding libpython from the advertised object set fails.

The Windows PGO failure on this PR was separate. MSVC /GL objects use an anonymous LTCG container instead of a normal COFF symbol table. The latest commit recognizes that container only when its signature, version, machine, class ID, and payload size all match.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update validation to check that all symbols are available in object files

2 participants