Gemma4: fix failed test cases by kaixuanliu · Pull Request #45568 · huggingface/transformers

kaixuanliu · 2026-04-22T06:52:45Z

What does this PR do?

This PR did several things:

Skip some test cases that are not suitbale for gemma4 model
Fix bug when attention_mask is None(tests/models/gemma4/test_modeling_gemma4.py::Gemma4Audio2TextModelTest::test_eager_matches_fa2_generate)
fix some failed test cases related to test_flash_attn_x_from_config
Add XPU related Expectations

Fixes # (issue)

Code Agent Policy

I confirm that this is not a pure code agent PR.

Who can review?

@ydshieh pls help review

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

ydshieh · 2026-04-29T12:37:17Z

+        if attention_mask is not None:
+            attention_mask = self._convert_4d_mask_to_blocked_5d(attention_mask)


@Cyrilvallez any opinion.

From PR descriptioin

Fix bug when attention_mask is None(tests/models/gemma4/test_modeling_gemma4.py::Gemma4Audio2TextModelTest::test_eager_matches_fa2_generate)

ydshieh · 2026-04-29T12:43:04Z

+    @require_flash_attn
+    @require_torch_accelerator
+    @mark.flash_attn_test
+    @slow
+    def test_flash_attn_2_from_config(self):
+        # Gemma4 requires mm_token_type_ids in train mode, so we test in eval mode
+        self.flash_attn_from_config(attn_implementation="flash_attention_2", test_fwd_in_train=False)
+
+    @require_flash_attn_3
+    @require_torch_gpu
+    @mark.flash_attn_3_test
+    @slow
+    def test_flash_attn_3_from_config(self):
+        # Gemma4 requires mm_token_type_ids in train mode, so we test in eval mode
+        self.flash_attn_from_config(attn_implementation="flash_attention_3", test_fwd_in_train=False)


@kaixuanliu I didn't see these 2 failing on our Flash Attn CI job.

Could you share more info / error logs ?

Our flash attn ci doesn have FA3 - I think it's hard to install because you need to compile from source and it's much longer than FA2 build from source

Maybe we could add a separate FA4 CI - not sure how stable it is tho since it's still in beta

Well, for FA3 and FA4, on my env they are skipped as well. I can delete these two.

Ah no, see my comment below #45568 (comment)

No, I mean for

test_flash_attn_2_from_config

our CI is [PASSED]. So I am not sure why we need this fix, at least for FA2.

Our CI runner don't have FA3 or FA4, so they are skipped. But the question may still valid: do we really this fix?

well, the code is added before this PR is merged: #45454, it will crash for this case before here is removed. After this PR this case can pass. I will update the code.

cc @zucchini-nlp for viz, don't think it's super important but would still be nice to fix at some point ig

Gemma4 requires mm_token_type_ids in train mode, so we test in eval mode > was fixed already no?

If not, @kaixuanliu can you open an issue and ping me there. I might forget to come back when bugs are reported under PRs 😅

It's already fixed. And I have removed this part.

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

Qodo-Free-For-OSS · 2026-04-30T09:32:17Z

Hi, Integration tests add device-specific Expectations entries for XPU without a default fallback, so running these tests on an unsupported accelerator type (or an XPU generation not covered) can select an unintended expectation or raise if no expectation matches. This makes the tests more brittle to new device properties.

Severity: informational | Category: reliability

How to fix: Add default expectation fallback

Agent prompt to fix - you can give this to your LLM of choice:

Issue description

XPU expectations were added without a (None, None) default. This can make the test brittle when run on different XPU generations or unexpected device properties.

Fix Focus Areas

tests/models/gemma4/test_modeling_gemma4.py[534-542]

tests/models/gemma4/test_modeling_gemma4.py[575-593]

tests/models/gemma4/test_modeling_gemma4.py[621-629]

tests/models/gemma4/test_modeling_gemma4.py[675-681]

tests/models/gemma4/test_modeling_gemma4.py[745-755]

Recommended changes

Add a default expectation (None, None): <existing cuda expectation> if you want to preserve previous behavior for other devices.

Or, add additional XPU keys if multiple gens are expected to run these tests.

Ensure the intended device coverage is explicit to avoid accidental matching on future hardware.

Found by Qodo code review

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

vasqu

It kind of got lost, sorry let me run slow tests just for sanity checking and then merge

vasqu · 2026-05-05T14:12:10Z

run-slow: gemma4

github-actions · 2026-05-05T14:13:32Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: gemma4

github-actions · 2026-05-05T14:14:06Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/gemma4"]
quantizations: []

HuggingFaceDocBuilderDev · 2026-05-05T14:23:40Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

github-actions · 2026-05-05T14:50:58Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	a3c44435	workflow commit (merge commit)
PR	37b8baa4	branch commit (from PR)
main	a6ccf935	base commit (on `main`)

Model CI Report

❌ 1 new failed tests from this PR 😭

gemma4:
tests/models/gemma4/test_modeling_gemma4.py::Gemma4IntegrationTest::test_export_text_only (❌ ⟹ ❌)

* set eval mode for flash attn tests Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> * skip flash_attn tests Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> * fix bug when attention_mask is None Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> * add XPU expectations Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> * add deterministic decorator Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> * skip 2 compile related tests Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> * update Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> * update Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> * nice code Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> * fix code quality check Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> * update comment Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> --------- Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

kaixuanliu added 4 commits April 22, 2026 03:14

set eval mode for flash attn tests

078b908

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

skip flash_attn tests

7abaeef

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

fix bug when attention_mask is None

5eac346

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

add XPU expectations

edd29c4

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

kaixuanliu changed the title ~~Gemma4 fix~~ Gemma4: fix failed test cases Apr 22, 2026

kaixuanliu added 2 commits April 22, 2026 07:30

add deterministic decorator

1ef6f01

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

skip 2 compile related tests

51671d4

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

kaixuanliu marked this pull request as ready for review April 22, 2026 09:25

github-actions Bot requested review from Rocketknight1 and ydshieh April 22, 2026 09:25

kaixuanliu added 2 commits April 27, 2026 10:15

Merge branch 'main' into gemma4-fix

388ad09

update

6165de2

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

evalstate mentioned this pull request Apr 28, 2026

Cumulative defect fixes from recent Transformers PRs evalstate/transformers#41

Open

ydshieh reviewed Apr 29, 2026

View reviewed changes

Comment thread tests/models/gemma4/test_modeling_gemma4.py

ydshieh reviewed Apr 29, 2026

View reviewed changes

vasqu reviewed Apr 29, 2026

View reviewed changes

Comment thread tests/models/gemma4/test_modeling_gemma4.py Outdated

kaixuanliu added 3 commits April 29, 2026 14:58

update

5466654

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

nice code

c90ebaa

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

fix code quality check

26400da

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

This was referenced Apr 29, 2026

Cumulative feature and defect updates from recent Transformers PRs evalstate/transformers#42

Open

Cumulative defect fixes from recent Transformers PRs evalstate/transformers#43

Open

update comment

852118a

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

vasqu approved these changes May 5, 2026

View reviewed changes

Merge branch 'main' into gemma4-fix

37b8baa

vasqu added this pull request to the merge queue May 5, 2026

Merged via the queue into huggingface:main with commit df2f2b5 May 5, 2026
22 of 23 checks passed

kaixuanliu deleted the gemma4-fix branch May 8, 2026 01:56

		if attention_mask is not None:
		attention_mask = self._convert_4d_mask_to_blocked_5d(attention_mask)

Conversation

kaixuanliu commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Who can review?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaixuanliu Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Qodo-Free-For-OSS commented Apr 30, 2026

Issue description

Fix Focus Areas

Recommended changes

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

vasqu commented May 5, 2026

Uh oh!

github-actions Bot commented May 5, 2026

Uh oh!

github-actions Bot commented May 5, 2026

Uh oh!

HuggingFaceDocBuilderDev commented May 5, 2026

Uh oh!

github-actions Bot commented May 5, 2026

CI Results

Commit Info

Model CI Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kaixuanliu commented Apr 22, 2026 •

edited

Loading

kaixuanliu Apr 29, 2026 •

edited

Loading