Comments (9)
Installing dependencies is also a big slowdown with the current CI. Perhaps that can be sped up somehow? Using a Docker image with everything preinstalled, for instance?
from sanic.
@iAndriy This is correct, the vast majority of time is consumed in pytest. Different platforms are run in parallel but it still takes long, and this is cumbersome when running them locally, too.
Many tests use
sleep
and/or timeouts that would greatly benefit of parallel execution (or of mock clock). There are parts where unit tests rather than test client could be used, still ensuring coverage and desired tests, but making it faster, while currently the vast majority of tests are integration tests though the test client. There surely is also unnecessary overlap with different tests, and some might not even be relevant anymore.But going through them to optimize and improve, making sure not to miss anything important, is a huge task that I doubt there is manpower to. Test parallelization might yield some easy results but it still needs care not to make the tests fail at random. Also looking at the few particularly slow tests in detail might yield useful speedup. As would reducing the number of combinations in parametrized tests. And reducing sleep/timeout durations (as long as no random failures occur then).
@Tronic Thanks for elaborating, a lot of useful details I haven't know about.
yeah, it's important to keep tests deterministic, flakiness is a huge problem and being conscious of it during optimisation is important. As per your earlier suggestion re-organizing tests and adding dynamic ports to enable parallelism shall do the work. Surprisingly, some of the tests fail on my local machine but work during CI/CD, so maybe adding some utilities to run tests in docker (on local machine) would be useful too.
I intend to focus more on the dynamic port allocation and addressing sleeps, once we have this fully added start with the xdist/re-org of tests into specific folders.
from sanic.
Quite many tests run the server on some hardcoded port, so a bit of care might be needed to enable parallel tests.
from sanic.
Quite many tests run the server on some hardcoded port, so a bit of care might be needed to enable parallel tests.
this makes sense, sounds like interesting problem to overcome, perhaps we can ask for the available port from os π€
from sanic.
An excellent goal, but that is going to be a painful task. As mentioned, because we're doing a lot of io, there are a lot of tests that are much more integration than unit. There's a lot of baked in synchronization and state management. It will be a huge effort to overcome and im not sure we'd be able to cover all of the same use cases. Port selection alone isn't the only obstacle.
Perhaps a more realistic goal may be to separate the test suites into a few subgroups and run each of those simultaneously. But not sure that'd be much better as it would explode the number of runners.
from sanic.
That's a great idea. I'm all for this. We did this at work and it was a huge time saver.
from sanic.
@ahopkins @Tronic Perhaps Can you please share some links with examples ?
From my findings the majority of time is spend on running tests.
Here is example with annotations on screenshots. As you can see job took 4mins 12 secs( 252 seconds), within this time tests has been executed in 245.45 seconds.
from sanic.
@iAndriy This is correct, the vast majority of time is consumed in pytest. Different platforms are run in parallel but it still takes long, and this is cumbersome when running them locally, too.
Many tests use sleep
and/or timeouts that would greatly benefit of parallel execution (or of mock clock). There are parts where unit tests rather than test client could be used, still ensuring coverage and desired tests, but making it faster, while currently the vast majority of tests are integration tests though the test client. There surely is also unnecessary overlap with different tests, and some might not even be relevant anymore.
But going through them to optimize and improve, making sure not to miss anything important, is a huge task that I doubt there is manpower to. Test parallelization might yield some easy results but it still needs care not to make the tests fail at random. Also looking at the few particularly slow tests in detail might yield useful speedup. As would reducing the number of combinations in parametrized tests. And reducing sleep/timeout durations (as long as no random failures occur then).
from sanic.
I found quite many tests failing both on local MacOS and some versions of Linux. These were addressed in #2844 in case you want to dig up the diff or specific commits from that branch. With those changes I got the tests passing on MacOS and various Linux distros, not sure if I tried Windows though. The current main still fails many tests on my Macbook.
from sanic.
Related Issues (20)
- How can I use the Response streaming technique in a POST method? HOT 2
- Shutdown never finishes when background tasks are cancelled but need some time to finish
- EOFError in Server Manager HOT 2
- Start-up exception HOT 1
- openapi json exception
- The RESTful API has redundant path parameters HOT 1
- Response streaming produces [ERROR] Invalid response type None (need HTTPResponse)
- Unexpected behavior with bp.middleware() using classes HOT 2
- δΈδΈͺζ±ε© HOT 1
- "body not consumed" error on GET request with content-length=0 HOT 1
- mypy error type for async get function in HTTPMethodView HOT 2
- Unexpected behaviour with REQUEST_MAX_HEADER_SIZE HOT 1
- Unexpected behavior when using blueprint groups and app.url_for HOT 1
- How to avoid abort when the code throw an Exception HOT 1
- Sanic drops part of HTTP response data HOT 8
- REQUEST_TIMEOUT & RESPONSE_TIMEOUT not working
- Documentation Hidden by Menu HOT 1
- Native Gzip & Brotli compression HOT 3
- AttributeError: 'Sanic' object has no attribute 'multiplexer' HOT 4
- help pls, when func A and func B are in defference Sanic-Server, same global asyncio.Event() var in them cant sync HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sanic.