You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
test_pickle_load[arm] (added in #1092) is bimodal: it flips between ~1.6 ms and ~1.8 ms across commits that don't touch the pickle path at all, so with the 2% threshold it randomly paints unrelated PRs red.
The baseline itself flipped between those two comparisons: main measured 1.6 ms at b36a4a1 (06-27) and 1.8 ms at 8f7ac06 (06-30), with exactly one commit in between (Add frozendict support for 3.15+ #1052, frozendict) that doesn't touch the pickle path either.
Since this is simulation mode (instruction counting), it's not run-to-run measurement noise: the benchmark appears sensitive to code layout/alignment in _core.c, so any code addition can toggle it between the two states. Related to the general variance discussion in #1075 and CodSpeedHQ/codspeed#409.
Options (all live in the CodSpeed dashboard, admin only - there's no repo-side config for this):
Set a per-benchmark regression threshold for test_pickle_load[arm] above the flip amplitude (say 15%). Keeps measuring, stops the false gating.
test_pickle_load[arm](added in #1092) is bimodal: it flips between ~1.6 ms and ~1.8 ms across commits that don't touch the pickle path at all, so with the 2% threshold it randomly paints unrelated PRs red.Evidence from this week:
decimal_formatoption #978 (decimal_format encoder option, head 045a25c vs main b36a4a1): flagged -10.05% (1.6 -> 1.8 ms), check red. The diff cannot affect struct pickling.to_builtinswithfrozendict#1105 (to_builtins frozendict fix, head 5a54d59 vs main 8f7ac06): same benchmark reported +10.28% (1.8 -> 1.6 ms), check green (improvements don't gate).frozendictsupport for 3.15+ #1052, frozendict) that doesn't touch the pickle path either.Since this is simulation mode (instruction counting), it's not run-to-run measurement noise: the benchmark appears sensitive to code layout/alignment in
_core.c, so any code addition can toggle it between the two states. Related to the general variance discussion in #1075 and CodSpeedHQ/codspeed#409.Options (all live in the CodSpeed dashboard, admin only - there's no repo-side config for this):
test_pickle_load[arm]above the flip amplitude (say 15%). Keeps measuring, stops the false gating.typing.ClassVarannotation #1097.Option 1 seems the most targeted. Worth checking whether the sibling
test_pickle_*benchmarks from #1092 show the same behavior before picking.