Restore pytest-benchmark for local and manual profiling - #1113
Conversation
ofek
left a comment
There was a problem hiding this comment.
Instead, can you please bring back support for the old, simpler benchmarks so that we can run them locally and with the manual CI trigger? I was going to myself actually so I would feel bad merging this and then adding much of this again shortly after.
f40e1b6 to
513cd64
Compare
|
@ofek good call, reworked to your suggestion. Instead of dropping the pytest-benchmark scaffolding, this just re-adds That restores the simple benchmarks for local runs and the manual profile trigger. I confirmed the two plugins coexist: without One thing still open from the old setup: the No rush on my end, just let me know what you think whenever you get a chance. |
Merging this PR will improve performance by 11.26%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | test_decode_type[arm-msgpack-True] |
824.3 µs | 735.6 µs | +12.06% |
| ⚡ | Simulation | test_decode_type[arm-json-None] |
872.8 µs | 782.2 µs | +11.58% |
| ⚡ | Simulation | test_decode_type[arm-json-False] |
960.9 µs | 872.4 µs | +10.14% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing fix-profile-workflow-benchmark-flags (513cd64) with main (9ddc0e9)
Footnotes
-
135 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
|
I'll take on the work to bring them back, thanks! |
|
Whatever's easiest for you: feel free to merge this and start on yours, or close it and just repeat the one line in your own PR since it's the only change here. I'll leave the fate of this one in your hands. |
The
Profile/Manual Profileworkflow crashes on every run:just test-perfappends--benchmark-name short --benchmark-disable-gc, but those are pytest-benchmark options and pytest-benchmark was dropped fromtest-profwhen the perf suite moved to pytest-codspeed (#1075). So every matrix job dies at collection:(all 5 OS/arch jobs, e.g. run 27883637649).
Rather than strip the pytest-benchmark scaffolding out, this restores it:
pytest-benchmarkis added back to thetest-profgroup. That fixes the crash and brings back the simple, locally-runnable benchmarks forjust test-perfand the manual profile trigger, while leaving continuous benchmarking on pytest-codspeed untouched.The two plugins coexist cleanly (verified locally, both installed):
just test-perf all ...(no--codspeed): pytest-benchmark provides thebenchmarkfixture and prints its walltime table. The--benchmark-*/--calibrate/--roundsflags are recognized again.codspeed.yml(--codspeed): pytest-codspeed instruments the same fixture; pytest-benchmark steps aside. No conflict.uv.lockis gitignored, so the diff is just the dependency line; CI resolves the env frompyproject.toml.