Comments (14)
I suspect this is related to dojo/loader#39, though it needs more digging.
from core.
Based on what @vansimke said in dojo/loader#39, I don't think this is related to that. But what he said is probably related to this.
from core.
Added comment that was mistakenly added to dojo/loader#39:
I've played around with this and I think this may be related to a performance issue in CI. If I set the timeout for the .get() test to 60 seconds and the others to 30 seconds, then all the tests pass in Chrome and Android. Not sure if that is an acceptable solution though since this means that these tests are going to be prone to fail at seemingly random intervals.
from core.
Wow, 60 seconds. That seems crazy, especially when I think the other browsers complete that just fine. I haven't looked at the timing on the other browsers, but I bet it is a lot less than 30-60 seconds.
It seems like something else is causing a problem right?
from core.
I would expect so. Chrome is fine with 30 and 10 (didn't play to see how low I could go, but 10s failed with .get()
). I went from 30s to 60s for Android on the get()
as a last gasp to see if I could get them all to pass.
I'm going to dig into the guts of the tests next and see if there is something odd going on in there. I can't imagine what though.
from core.
I wonder if we're clogging the browser's request pipeline? Maybe we're making too many HTTP requests as part of running the suite and it is just taking time for them to clean themselves up.
from core.
The resources allocated for cloud testing can be extremely limited, making timing-sensitive tests very unreliable. I think our best bet is to give tests very generous timeouts, but have them resolve themselves asap when the test conditions are met.
from core.
I think that I might be on to something. When I run the full suite, the Android browser takes 45+ seconds to return, but running only the tests/unit/request.ts tests brings it down to less than 2s. I'm pretty sure the tests are fighting with Intern for network resources.
from core.
@jason0x43 any thoughts on this? I know we had challenges with IE9 and reordered the way we return the results, but now it seems like on other large test suites we are having something similar and have reliably triggered it now on both Chrome and Android. Maybe something @jacobroufa and @vansimke could work on together?
from core.
@vansimke is likely correct that the browser's request pipeline is being saturated. The WebDriver reporter in intern master significantly improves the situation by buffering messages and sending them using a single active connection at a time. This appeared to improve the stability of the core tests in several runs I made this weekend. At least, I got significantly fewer failures with intern master than with 3.0.6.
In 3.0.6 you can set waitForRunner
to true. This can significantly slow down the testing process, but by causing the testing system to send lifecycle messages synchronously with the testing process (rather than batching them and sending them asynchronously), it removes the network contention. This also appeared to improve the stability of the request tests.
Assuming network contention really is the issue, it may be useful for Intern to provide a method allowing network-sensitive tests to clear out the lifecycle message queue before the test starts.
Even though the above two changes (using intern master and using waitForRunner
in 3.0.6) did improve stability, I still did get occasional failures. As @msssk points out, using cloud resources can make relying on timing, particularly network timing, a bit of a nightmare. Sometimes a request to the echo server would simply take longer than the timeout (which I left set at 5000ms).
from core.
Setting waitForRunner
to true did, in fact, stabilize the tests without having to change the timeout, but the entire suite took much longer to run (about 6x). The .get() test ran in only 0.281s vs 10s+ on Win10/Chrome and 0.272s vs ~45s for Android with this set.
I'm going to keep working with this to see if we can improve the stability without compromising the execution time, but for now, I would recommend setting the waitForRunner
flag to true so that results can be relied upon.
from core.
@vansimke I wouldn't be opposed either to trying out "intern": "theintern/intern"
as the development dependency to utilise what is in master. If that also stabilises things, then it obviously puts pressure on us to get the next release of Intern out.
from core.
@kitsonk That is what I'm experimenting with now. I'll update this issue when I get results.
from core.
sinon does not do well in certain async situations which was the root cause to this.
from core.
Related Issues (20)
- Update tests to Intern 4
- Deprecate (maybe remove) load
- Deprecate lang/assign (and refactor usage)
- old dojox.storage documentation is broken HOT 2
- `utils#throttle()` test fails in Edge HOT 1
- Xhr unit tests intermittently fail on IE11
- I just would like to use dojo2 as a library, it looks a little complex, how could I use it like a react or dojo1? HOT 9
- tslib must be a dependency and not a devDependency HOT 8
- Evented is bloated
- Problem since TypeScript 2.7.1 – Type 'M[keyof M]' does not satisfy the constraint 'EventObject<keyof M>' HOT 9
- UrlSearchParams toString() should not add queryParams if they have no value set
- has.ts causes loading the main bundle to crash in Internet Explorer 11 in https HOT 4
- Support the abortable fetch API HOT 1
- dojo/widget-core typo in code snippet HOT 1
- README: outdated Promise doc
- Docs for UrlSearchParam are wrong
- Upload progress events add preflight request
- Standardize Constructor exports
- Evented listener destroying its own handle won't run all listeners
- lang.mixin errors with null target
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from core.