Comments (9)
Is there a graceful way to handle failed link checks, without failing the whole test suite?
from kolibri.
I'd rather keep running the whole test suite and just remove link checking, but it raises the question of when will we run "occasional" tests?
from kolibri.
@MCGallaspy why do you wanna remove it? It's extremely useful to avoid dead links in the docs, that's why Sphinx has that command built in ;)
it raises the question of when will we run "occasional" tests?
Yup, the conditional thing is conditional :D My suggestion is to base it off git changesets, mean if you change something in docs/
then we run the link checker.
from kolibri.
It's useful, but if it occasionally times out and fails the whole suite, its utility goes down in my eyes. I just want to ensure that all the unit tests/integration tests/etc are run regardless of the changeset, as opposed to just a specific app label. However if you're proposing that we only run the link checker when the docs are in the changeset, then I think that's a fine idea.
from kolibri.
It's useful, but if it occasionally times out and fails the whole suite, its utility goes down in my eyes.
But that's the point of this issue?
I just want to ensure that all the unit tests/integration tests/etc are run regardless of the changeset
If you have good heuristics, you don't need to bloat the CI :)
if you're proposing that we only run the link checker when the docs are in the changeset, then I think that's a fine idea.
That's the first proposal. But I hope we expand on it so we can make tests faster. I'd personally like a [skip-bdd]
tag for commit messages :)
from kolibri.
Just a few points:
- My opinion is that running all the tests for every PR is not bloat, but is rather using a test server precisely as intended. If anything, we should run a limited set of tests locally before pushing (if the test suit gets large enough to warrant it) and let the test server do the heavy work.
- IMO link checking is of marginal utility, which is why I'm happy to only run it occasionally. But I'm opposed to only running tests occasionally. It creates the possibility of catching bugs too late.
- Wanting to skip tests is a symptom of a poor test suite -- but Kolibri doesn't have a poor test suite. In fact, it's only got a handful of tests at all, and they run really fast and aren't brittle. When we introduce tests that have a large # of possible failure points (like browser tests with Selenium) let's be conscientious enough to design easily testable systems. (One thing that makes KA Lite's browser tests brittle is that the design is not loosely coupled, making it testable only with difficulty.)
Remember This is not KA Lite.
from kolibri.
Comments from the meeting:
- JS tests are expected to be very fast
- We shouldn't skip tests when it affects Coverage negatively
- We can possibly do tests based on which branch is pushed to
from kolibri.
My opinion is that running all the tests for every PR is not bloat, but is rather using a test server precisely as intended. If anything, we should run a limited set of tests locally before pushing (if the test suit gets large enough to warrant it) and let the test server do the heavy work.
I really agree with this, it's a matter of adjusting it with very explicit exceptions. And if we cannot identify an exception to the rule (test everything), then we test everything :)
Reality has been quite cruel.. for KA Lite. Even though we tested stuff locally to avoid waiting 20-40 minutes for the CI, a lot of tears were spilled over shaky test frameworks.. because they caused communication work regarding false negatives.. and some traumatizing debugging :) First case of this same phenomena in Kolibri is the sphinx link checker.. which IMO is really useful but should only be executed when explicitly asking for a full set of tests or when changing .rst files.
Btw. I maintain an open source project in which only some of the tests were running but all lights were green... when we finally fixed the test discoverer, we had at least a couple severe issues :) -> https://github.com/django-money/django-money/pull/162/files ...with that said, I'm alert to putting too much complexity into the testing frameworks in case should be unknowing that our software has become Real Media Player because our test framework says otherwise :)
from kolibri.
It's done!
from kolibri.
Related Issues (20)
- EQM - Newly created quizzes should appear on top of the table in Plan > Quizzes HOT 1
- EQM - Select a quiz - Missing 'Are you sure you want to leave this page?' modal when I click the browser's Back button HOT 2
- EQM - The Section settings modal closes after clicking 'Add more questions' HOT 2
- Hover and Click effect on Demo & Download button for better user interface and experience HOT 1
- EQM - Cannot preview the answer history of a completed quiz in mobile view HOT 1
- EQM - Preview quiz - The selection effect is reversed HOT 2
- EQM - Reports - Seeing 'Resource not found on device' while the resources are actually on the device HOT 1
- EQM - Reports - Getting a 500 error when viewing the 'Difficult questions' for a not started quiz HOT 2
- Pex filename problems with pre-releases HOT 1
- Quiz creation does not properly show the authentication wrapper
- EQM - All question cards are disabled when there's just 1 question to add HOT 2
- EQM: Move side panel titles to top of side panel HOT 1
- Display issues for login language switcher HOT 1
- EQM - The 'Add section' button is not fully visible in some languages HOT 1
- EQM - Replace questions in RTL language - The checkboxes and arrows are overlapping HOT 5
- Create a linting rule to prevent RTL breaking inline styles
- EQM - Answer history in mobile view - Incorrect question numbers when viewing the sections HOT 2
- Implement `/api/v1/errors/report/` in telemetry
- Implement error capture into Kolibri's Tasks
- Error at importing CSV with users HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kolibri.