Skip to content

CodSpeed: test_pickle_load[arm] is bimodal and randomly flags unrelated PRs #1110

Description

@Siyet

test_pickle_load[arm] (added in #1092) is bimodal: it flips between ~1.6 ms and ~1.8 ms across commits that don't touch the pickle path at all, so with the 2% threshold it randomly paints unrelated PRs red.

Evidence from this week:

Since this is simulation mode (instruction counting), it's not run-to-run measurement noise: the benchmark appears sensitive to code layout/alignment in _core.c, so any code addition can toggle it between the two states. Related to the general variance discussion in #1075 and CodSpeedHQ/codspeed#409.

Options (all live in the CodSpeed dashboard, admin only - there's no repo-side config for this):

  1. Set a per-benchmark regression threshold for test_pickle_load[arm] above the flip amplitude (say 15%). Keeps measuring, stops the false gating.
  2. Ignore the benchmark entirely in CodSpeed.
  3. The repo-wide "informational status check" toggle, which would also address the "CI looks perpetually broken" complaint from Fix crash on incorrect typing.ClassVar annotation #1097.

Option 1 seems the most targeted. Worth checking whether the sibling test_pickle_* benchmarks from #1092 show the same behavior before picking.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions