Parallel Threaded Execution of Scenarios about common HOT 20 CLOSED

There is a parallel runner in this repo, but as it states only java backend has been tested. But I've used this on projects with java & groovy and all was OK.

https://github.com/djb61/parallel-cucumber-jvm

from common.

Since you mentioned cucumber-jvm, here is a longish issue describing what changes are nesessary to make parallel execution of cucumber scenarios work: cucumber/cucumber-jvm#630
A possible problem with your suggestion is that the glue code instances (provided by the di framework of your choice) as well as "world" instances (as in groovy) won't be scenario scoped and therefore be subject to racing issues (whether it's a good idea or not to keep state in the glue) (at least thats what I understood from a quick look through your changes).
Unfortunately the issue mentioned above has been closed (not even with a "wont fix because: ..." but rather a "just write faster tests").

from common.

Yes, we don't store data in our glue code: we have a special ScenarioScopedData class managing the multi-threading issues of data stored per scenario / thread.

OK for closing the issue, but a lot of your users are using Cucumber with Selenium, and Web tests are not fast by design, it cannot be changed.

So we will continue to use our own fork of Cucumber.

from common.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

from common.

Still an issue. Still annoying. ping.

from common.

The core team doesn't currently have bandwidth to work on this. You can help by submitting a PR, or by offering financial support: https://cucumber.io/blog/2017/10/03/invest-in-cucumber

from common.

I think the topic is too big for single PRs but requires a good vision and coordination (as discussed in cucumber/cucumber-jvm#630). It is okay if it is not a priority right now but I just wanted to prevent the issue from being closed due to inactivity. Because it still exist and because there are still users out there who would love to have it fixed :)

from common.

This is the sort of feature that isn't done in a jiffy. It would probably involve a considerable amount of changes to the codebase (whether it's ruby, java or javascript), and could easily take weeks of work, and involve several people.

This is the kind of work that is only likely to happen if it's funded through the opencollective.

In order to incentivise more organisations and people to provide financial support for big efforts like this, I plan to set up a voting scheme similar to what webpack has done.

Sponsors gets more votes, increasing the likelihood of people picking up what they voted/paid for.

from common.

Some thoughts:

• An internal tool (e.g. CLI parallel runner options) would have to be made for each version of Cucumber because they are different internal languages whereas external tools (e.g. feature file splitters and report aggregators) can work with all versions of Cucumber because they all use the same Gherkin language and standard (mostly) formatters. That is a large multiplication of effort and maintenance.

• Testing at such scale that parallelization becomes critical is also the kind of situation where people tend to be rolling their own CI pipelines anyway, so the odds of there being more than one Cucumber process anyway on lots of machines (as opposed to one process with loads of threads on one machine) goes up. Smaller, composable tools become a lot more valuable than one tool tool that tries to do everything.

• If you are not satisfied with existing external tools, make better ones. That's what most of my open source projects are. :)

from common.

What version of cucumber-groovy/cucumber-java did you use @boaty82 ?

from common.

@seanleblanc, this tool uses cucumber-jvm 1.2.2(gherkin-2.12.2) 
see new PR over here (if you haven't found it yet) cucumber/cucumber-jvm#1357

from common.

@seanleblanc not sure if you mean my forked repo (and recent PR) or the earlier quoted library

my fork of cucumber-jvm is working off of the latest code. I hope it will be reviewed soon and I either get feedback or it is considered to be merged in

parallel-cucumber-jvm uses cucumber-jvm 1.2.5 (I've used the groovy plugin along side this) - I did look into upgrading this to support the newer libraries, but it was messy at best and didn't account for the comments around performance e.g. not performing expensive operations multiple times

from common.

This is being worked on for Cucumber-JVM over at cucumber/cucumber-jvm#1389 so I am closing this.

from common.

Hello Quick Question
Trying to add the threads = 3 inside Junit runner not working check the screen shot
let me know what i am making mistake

@RunWith(Cucumber.class)
@CucumberOptions(
features = { "src/test/java/samplefeature/sampletest.feature" },
glue = { "samplesteps" },
tags = { "@regression"},
plugin = {"pretty"}, threads = 3
)

from common.

Hello @mdrasul,

With the parallel fork, scenarios are executed out of order. The produced report.json is aggregated as if all scenarios were ran in sequential order. As a result, this aggregation can only take place at the very end of the process, when all scenarios ran. For this reason, parallel run is not supported for IDE JUnit panels. I should probably remove the "threads" parameter in @CucumberOptions. Or add some complicated logic to report scenarios in order as soon as a direct chain of scenarios is done, but with @synchronized-* scenarios are run first (to optimize parallelism), this complicated logic could prove useless in several cases.

Parallel is only supported with the CLI runner.

Also, I released a final release (non-SNAPSHOT) a few days ago: be sure to use 1.2.4 instead of the old 1.2.4-SNAPSHOT.

from common.

@danielwegener Regarding your comment of 10 Dec 2016, the new version I released a few days ago fixes this for Spring: the Spring DI layer is now thread-safe: each thread has its own Glue scope instance.

Also, from what I see in:

• cucumber/cucumber-jvm#630
• cucumber/cucumber-jvm#1357
• cucumber/cucumber-jvm#1389

... Cucumber now supports parallelization. Not on a scenario or pickle (or even rule, I've discovered) level yet, but at a feature level.

And not with @synchronized-* annotations (but this one, I understand you would not want to support this requirement).

That's a very good news! Congrats for all the work done these last months.
And for the big refactoring that will come next: https://docs.google.com/document/d/12Y3Ut-SVSGdw9L_aW6NGQY9JaCcn3z0alY4d7kzOndU .

from common.

... Cucumber now supports parallelization. Not on a scenario or pickle (or even rule, I've discovered) level yet, but at a feature level.

This depends on the runner you use. The CLI and TestNG support parallel execution of pickles, JUnit per only supports parallel execution of features. I would have liked to support the parallel execution of pickles by JUnit too but JUnit 4s architecture simply does not allow it.

from common.

Oh, thanks for the precision, that's wonderful (and feature-level is already very helpful in JUnit, if used only during development).
Do you know if it would be possible in JUnit 5?

from common.

Being a test runner Cucumber would use the JUnit Platform by implementing the JUnit Engine rather then integrate with JUnit Jupiter.

I am currently not aware of anything in the JUnit Engine that would prevent it.

from common.

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

from common.