make sure we call check_auto in CI by tarekziade · Pull Request #45775 · huggingface/transformers

tarekziade · 2026-05-04T07:27:15Z

What does this PR do?

per #45774 we did not run check_auto in the CI

Commit 0e15778 ("Dynamic auto mapping" PR #45018, Apr 16 2026) is the one that created utils/check_auto.py in the first place. In that same commit, the Makefile was changed so that:

The CI-run targets (style, check-code-quality) had their old auto_mappings entry renamed to sort_auto_mappings i.e., they kept running only the simple alphabetical sorter
(utils/sort_auto_mappings.py), which doesn't have the tempfile bug.
The new auto_mappings entry (the buggy check_auto.py regenerator) was added only to check-repo and fix-repo; both local-developer-only targets.

So the buggy regenerator has never had CI coverage since it was introduced. Before PR #45018, the auto_mappings name in CI referred to the old simple sort script. It wasn't dropped from CI; it was effectively replaced under a different name when the new dynamic mechanism was bolted on.

The current PR fixes it by:

making sure we run it now in CI
share the same list of check across all make targets, so we don't regress again there
display a diff in check_auto so it's easier to debug
fixed a big in glob.blog ordering (system dependent)

tarekziade · 2026-05-04T07:28:43Z

Part of the problem is that the Makefile has too many targets that do the same things. I will make sure we de-dupe. WIll add more commits here

tarekziade · 2026-05-04T07:29:43Z

proof of wiring : https://app.circleci.com/pipelines/github/huggingface/transformers/173694/workflows/d70c1229-cb57-4444-9331-88c78581d942/jobs/2296112

HuggingFaceDocBuilderDev · 2026-05-04T07:39:29Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

github-actions · 2026-05-04T08:12:07Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto

tarekziade · 2026-05-04T08:12:52Z

run-slow: auto

github-actions · 2026-05-04T08:14:11Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/auto"]
quantizations: []

Cyrilvallez

Thanks! make fix-repo used to do less though no?

zucchini-nlp · 2026-05-04T08:19:43Z

+                is_natural = model_type == module_name
+                # If we already recorded a natural match for this model_type, don't let a
+                # non-natural one overwrite it — the natural class is the canonical owner.
+                if model_type in natural_types and not is_natural:
+                    continue
+


nice catch!

github-actions · 2026-05-04T08:21:52Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	afa24183	workflow commit (merge commit)
PR	e90065ec	branch commit (from PR)
main	54e90e08	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

tarekziade · 2026-05-04T08:41:50Z

Thanks! make fix-repo used to do less though no?

Fix thanks for spotting. recap:

check-code-quality: 6
check-repository-consistency: 18
check-repo: 24 (sum of both for local runs)
fix-repo: 15 (with fixes)

zucchini-nlp

approving for the check-auto, for the rest it'd be great to check-in with YihDar i believe. Can you also run-slow: maskformer to be sure ?

tarekziade · 2026-05-04T09:37:28Z

run-slow: maskformer

github-actions · 2026-05-04T09:39:07Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/maskformer"]
quantizations: []

github-actions · 2026-05-04T09:48:01Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	b736538f	workflow commit (merge commit)
PR	b20ced55	branch commit (from PR)
main	54e90e08	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

Cyrilvallez

Alright, thanks! Trusting you that the skips based on the --fix flag work! If you can confirm before merge please to be sure! 🙏🏻

Cyrilvallez · 2026-05-04T09:50:21Z

+    # In --fix mode, drop checkers that have no fix capability (fix_args is None) so
+    # they don't print bogus "(0.00s)" lines or inflate the final pass count. Print
+    # one transparency line listing what we're skipping.
+    if args.fix:
+        not_fixable = [n for n in names if CHECKERS[n][3] is None]
+        if not_fixable:
+            names = [n for n in names if CHECKERS[n][3] is not None]
+            print(
+                f"Skipping {len(not_fixable)} check-only checker(s) in fix mode: {', '.join(not_fixable)}\n",
+                flush=True,
+            )


Trusting you that this work (not 100% sure how args are forwarded here), but could you just drop the print please? It's expected to skip them, so not warranting any print

It's under args.fix, so keep a message here makes sense IMO.

Also it's a check utility (unlike modeling or loading code), so should not cause "a lot of warning" as we saw before.

this will generate a single line at the top, not adding all the checks in details, see an example:

(venv) m5 ➜transformers git:(tarek-fix-check-aut) ✗ make fix-repo Skipping 9 check-only checker(s) in fix mode: types, modeling_structure, imports, import_complexity, repo, inits, config_docstrings, config_attributes, update_metadata ✓ Ruff linting (0.09s) $ ruff check examples tests src utils scripts .circleci/create_circleci_config.py benchmark benchmark_v2 setup.py conftest.py --fix --exclude All checks passed! ✓ Ruff formatting (0.03s) $ ruff format examples tests src utils scripts .circleci/create_circleci_config.py benchmark benchmark_v2 setup.py conftest.py --exclude 4293 files left unchanged ✓ Import ordering (0.04s) $ python utils/custom_init_isort.py ✓ Sort auto mappings (0.03s) $ python utils/sort_auto_mappings.py ✓ Generate auto mappings (4.00s) $ python utils/check_auto.py --fix_and_overwrite ✓ Copied code consistency (5.24s) $ python utils/check_copies.py --fix_and_overwrite ✓ Modular file conversions (2.44s) $ python utils/check_modular_conversion.py --fix_and_overwrite ✓ Documentation table of contents (0.08s) $ python utils/check_doc_toc.py --fix_and_overwrite ✓ Modeling rules documentation (0.05s) $ python utils/check_modeling_rules_doc.py --rules-toml utils/rules.toml --fix_and_overwrite ✓ Docstring formatting (7.30s) $ python utils/check_docstrings.py --fix_and_overwrite ✓ Dummy objects (0.02s) $ python utils/check_dummies.py --fix_and_overwrite ✓ Pipeline type hints (2.59s) $ python utils/check_pipeline_typing.py --fix_and_overwrite ✓ Doctest list (0.02s) $ python utils/check_doctest_list.py --fix_and_overwrite ✓ Model dates (0.51s) $ python utils/add_dates.py ✓ Dependency versions table (0.12s) $ python setup.py deps_table_update running deps_table_update All 15 checks passed in 22.56s.

ydshieh · 2026-05-04T11:06:51Z

great to check-in with YihDar i believe

＠tarekziade, since you already got 2 ✅ + you are the author of these new "make / checker" stuffs, I believe in you too.

But if you prefer me to take a look on specific points, let me know.

tarekziade · 2026-05-04T11:56:13Z

Last run in CI, see python utils/check_auto.py

Generate auto mappings
$ python utils/check_auto.py
OK (15.04s)

Public imports
$ python -c "from transformers import *"
OK (30.07s)

Import complexity
$ python utils/check_import_complexity.py
Import complexity OK: 579 modules (max 1000)
OK (2.51s)

Copied code consistency
$ python utils/check_copies.py
OK (9.68s)

Modular file conversions
$ python utils/check_modular_conversion.py
OK (9.40s)

Documentation table of contents
$ python utils/check_doc_toc.py
OK (0.36s)

Modeling rules documentation
$ python utils/check_modeling_rules_doc.py --rules-toml utils/rules.toml
OK (0.56s)

Docstring formatting
$ python utils/check_docstrings.py
OK (29.44s)

Dummy objects
$ python utils/check_dummies.py
OK (0.17s)

Repository structure
$ python utils/check_repo.py
Repository-wide checks:
    - checking all models are included.
    - checking all models are public.
    - checking all models have tests.
    - checking all objects have documentation.
    - checking all models are in at least one auto class.
    - checking all names in auto name mappings are defined.
    - checking all keys in auto name mappings are defined in `CONFIG_MAPPING_NAMES`.
    - checking all auto mappings could be imported.
    - checking the DEPRECATED_MODELS constant is up to date.
    - checking all models accept **kwargs in their call.
OK (40.40s)

Init files
$ python utils/check_inits.py
OK (0.12s)

Pipeline type hints
$ python utils/check_pipeline_typing.py
OK (12.66s)

Config docstrings
$ python utils/check_config_docstrings.py
OK (15.93s)

Config attributes
$ python utils/check_config_attributes.py
OK (17.00s)

Doctest list
$ python utils/check_doctest_list.py
OK (0.14s)

Model metadata
$ python utils/update_metadata.py --check-only
OK (13.75s)

Model dates
$ python utils/add_dates.py --check-only
OK (2.56s)

Dependency versions table
$ python setup.py deps_table_update
running deps_table_update
OK (0.78s)


All 18 checks passed in 3m27.24s.

* make sure we call check_auto in CI * make sure we never drift again * improved the script so it actually displays what differs * make sure we pick the natural one first * added test coverage for check_auto * oops reverted this one by accident * keep it simple * fix-repo is only 15 checks * improved fix vs non-fix check runs

tarekziade requested review from Cyrilvallez and zucchini-nlp May 4, 2026 07:27

tarekziade self-assigned this May 4, 2026

tarekziade added 2 commits May 4, 2026 09:42

make sure we call check_auto in CI

48049ae

make sure we never drift again

28794b1

tarekziade force-pushed the tarek-fix-check-aut branch from b4bb3ed to 28794b1 Compare May 4, 2026 07:42

tarekziade added 2 commits May 4, 2026 10:01

improved the script so it actually displays what differs

741c690

make sure we pick the natural one first

e90065e

Cyrilvallez reviewed May 4, 2026

View reviewed changes

Comment thread Makefile Outdated

Comment thread Makefile Outdated

added test coverage for check_auto

c32a29f

zucchini-nlp reviewed May 4, 2026

View reviewed changes

tarekziade added 4 commits May 4, 2026 10:22

oops reverted this one by accident

b7742b6

keep it simple

1af800b

fix-repo is only 15 checks

96eaa07

improved fix vs non-fix check runs

b20ced5

zucchini-nlp approved these changes May 4, 2026

View reviewed changes

tarekziade requested a review from ydshieh May 4, 2026 09:37

Cyrilvallez approved these changes May 4, 2026

View reviewed changes

tarekziade added this pull request to the merge queue May 4, 2026

Merged via the queue into main with commit 8c004ec May 4, 2026
30 checks passed

tarekziade deleted the tarek-fix-check-aut branch May 4, 2026 12:14

Conversation

tarekziade commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

tarekziade commented May 4, 2026

Uh oh!

tarekziade commented May 4, 2026

Uh oh!

HuggingFaceDocBuilderDev commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026

Uh oh!

tarekziade commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026

Uh oh!

Cyrilvallez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zucchini-nlp May 4, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 4, 2026

CI Results

Commit Info

Uh oh!

tarekziade commented May 4, 2026

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

tarekziade commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026

CI Results

Commit Info

Uh oh!

Cyrilvallez left a comment

Choose a reason for hiding this comment

Uh oh!

Cyrilvallez May 4, 2026

Choose a reason for hiding this comment

Uh oh!

ydshieh May 4, 2026

Choose a reason for hiding this comment

Uh oh!

tarekziade May 4, 2026

Choose a reason for hiding this comment

Uh oh!

ydshieh commented May 4, 2026

Uh oh!

tarekziade commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tarekziade commented May 4, 2026 •

edited

Loading

tarekziade commented May 4, 2026 •

edited

Loading