Flaky test failures: 'stream' != 'streamstream' about nbval HOT 18 CLOSED

As a workaround, I'm using --sanitize-with=workaround.txt, where workaround.txt contains

regex: (stream)+
replace: stream

from nbval.

I suspect this is related to the number of stream outputs produced by the cell varying (each stdout.flush produces a message. This number can depend on timing). nbval should be merging consecutive stream outputs into one (what we call coalesce_streams in nbconvert), so that the number of messages doesn't affect the output to be compared.

from nbval.

It looks like my workaround isn't working: the tests are flaky. Does anyone have a better workaround?

from nbval.

@minrk: Will stream outputs be coalesced in the saved notebook's output, or does nbval need to process those as well?

from nbval.

I guess we might also have to handle \r. Should all streams with the same name be combined, or only subsequent ones? Are there any other gotchas? Is there a library call for handling these things (should there be)?

from nbval.

We should also handle backspace characters, according to this code: https://github.com/jupyter/notebook/blob/e77d7ebfb060d7ce45fb85e865a2c6d8e7635900/notebook/static/base/js/utils.js#L467. However, it seems nbconvert does not do this.

from nbval.

Is this notebook behavior going to be an issue?

from nbval.

To highlight this better, the outputs of this cell are not deterministic:

import time
import numpy as np
print('Bar!', file=sys.stderr)
print('Combo breaker!', file=sys.stdout)
time.sleep(np.random.rand())
print('Foo!', file=sys.stderr)

from nbval.

PS: The stream vs streamstream means that one cell had one stream, while the other cell had two. This can definitely be improved!

from nbval.

I think we can only combine consecutive stream outputs from the same stream, to make it more consistent.

I think we had discussed a kernel change so that printing on one stream would effectively flush the other, to make output order more deterministic.

from nbval.

@vidartf to answer your question, yes consecutive outputs from the same stream will be merged into one output in the live notebook and in nbconvert during nbconvert --execute. The nbconvert implementation is here.

from nbval.

Fix suggested in #57.

from nbval.

Sorry, I thought #57 was the one we already merged.

from nbval.

Thanks for fixing this! Do you have an ETA on publishing the fix? (I'm still getting flaky errors from nbval 0.5 on pypi.)

from nbval.

@vidartf is there anything else you want to work on before we cut 0.6?

from nbval.

I think we should be OK. I actually meant to get a release during PyData, but forgot about it.

from nbval.

I have released 0.6. :-)

from nbval.

Thank you! 👍

from nbval.