Switch notebook tests to nbval#987
Conversation
Codecov Report
@@ Coverage Diff @@
## master #987 +/- ##
=======================================
Coverage 80.28% 80.28%
=======================================
Files 136 136
Lines 14468 14468
=======================================
Hits 11616 11616
Misses 2852 2852 Continue to review full report at Codecov.
|
|
close/reopen here because it seems that we never recovered from a github actions outage this morning |
|
This is ready for review and comment. I will leave it open for at least 72 hours, merging no earlier than Thu 04 Mar 15:00 GMT (16:00 local). EDIT: Changed my mind from the 24 hour window -- this is a big enough change that I'll give a 72 hour window. It does involve a fundamental change to our testing procedure. |
sroet
left a comment
There was a problem hiding this comment.
LGTM, 1 consistency question, please don't hang this PR on that.
| date | ||
| ipynbtest.py --strict "test_pyemma.ipynb" || testfail=1 | ||
| date | ||
| py.test --nbval --current-env \ |
There was a problem hiding this comment.
Was the -v consciously dropped here?
| py.test --nbval --current-env \ | |
| py.test --nbval --current-env -v \ |
There was a problem hiding this comment.
Was the
-vconsciously dropped here?
Yes, this was intentional. The verbose output is not very useful with nbval, because each test is just named Cell 0, Cell 1, etc. (This is what I liked much more about ipynbtest and its parsing of the Markdown headings.) I added the -v for the slower notebooks, because I'm pretty sure that before too long I'll know exactly which cells are expected to take a while, and it'll be nice to get line-by-line updates for them. For fast-running notebooks, I didn't see a need for it.
We currently run several notebooks as tests -- mostly system-scale smoke tests, though also some more unit-level testing. These tests are run using a tool that @jhprinz wrote, ipynbtest.
Unfortunately, ipynbtest hasn't been maintained for a quite a while. Additionally, after @jhprinz developed ipynbtest, another tool, nbval, became a widely used pytest plugin for using Jupyter notebooks in testing.
With no one to maintain ipynbtest, I think we need to switch to nbval. This PR does that. I'm sad about this, because I think ipynbtest is a nicer tool to use (I like the way it reports Markdown headings as a progress indicator while running.) However, this is now the only thing blocking us from testing on Python 3.8 and 3.9, so the time has come.