flaky unit test: test_build_linkcheck.test_connect_to_selfsigned_fails about sphinx HOT 11 OPEN

Can you check whether this happened with previous commits? (might be hard to find it in the logs but I hope my fix for the intl flaky tests is not the culprit actually)

from sphinx.

Hm, now that I check, I do see another occurrence of it two weeks ago here: https://github.com/sphinx-doc/sphinx/actions/runs/7902157054/job/21567184034

I'm 99.9% confident that it's nothing to do with your changes; I think that the flakiness bug you fixed was a much more common cause of test failures and is properly resolved.

from sphinx.

The timeouts that are occuring here are reported by the requests library, and are a timeout while waiting to receive response bytes from the server.

There doesn't appear to be any systematic problem during creation of test in-process HTTP(S) servers -- generally that seems to work. For whatever reason, though, this particular test appears flaky. My guess is that it's because it introduces marginally more complexity than some other tests that would otherwise, in theory, be equally affected.

Given the timeout-detection logic, it may be possible to wait for socket-response connectivity before yielding the server thread for testing -- basically, to run a form of test-server healthcheck on the thread to ensure that it's up-and-running before allowing the test suite to communicate with it.

In practice however, that would reduce to introducing another level of timeout; we can't wait forever for that healthcheck, because then the test suite itself could become susceptible to indefinite delays.

So I think what we should do is simply increase the timeout for the affected test. If we see it occur elsewhere perhaps we can increase the timeouts there too -- but I would prefer to localize the changes so that we can more easily determine whether there is indeed any common root cause or whether this is happenstance.

edit: add test server setup source code hyperlink

from sphinx.

In practice however, that would reduce to introducing another level of timeout; we can't wait forever for that healthcheck, because then the test suite itself could become susceptible to indefinite delays.

It seems that simply increasing the timeout for the affected test does not work reliably; other tests are indeed affected too (see test failures in #12050 for commit 396b8df for example).

So perhaps it would be worthwhile to add a healthcheck step, despite the duplication of timeouts. A larger timeout could be configured for the test HTTP(S) server healthcheck/setup phase, with a smaller timeout configured for the Sphinx linkcheck build test cases.

from sphinx.

Would it work if we use different ports and not always the 7777? We coule rotate the ports being used from tests to tests.

from sphinx.

Would it work if we use different ports and not always the 7777? We coule rotate the ports being used from tests to tests.

I think that would only make a difference if the operating system waits/retries when binding to that specific port -- my understanding is that most operating systems do not do that, and simply return an error code if the port is already bound (in use). Since we don't see that error occur, the timeout appears to be unrelated.

from sphinx.

(I do think we might want to consider port rotation when parallelizing the tests.. but that's a different story :))

from sphinx.