mpinardi / cucumber-performance Goto Github PK

I have a very simple feature with 1 scenario with 4 steps, each of which performs a trivial task (variable assignment, addition, etc.).

I have a working plan which executes the feature over 5 seconds, with 10 threads.

One execution produced the following results:

Feature: test.feature Count: 194 Avg: 155 Min: 13 Max: 1095
Scenario: Test addition Count: 194 Avg: 0 Min: 0 Max: 24
Step: x is 1 Count: 194 Avg: 0 Min: 0 Max: 2
Step: y is 2 Count: 194 Avg: 0 Min: 0 Max: 0
Step: I add x + y Count: 194 Avg: 0 Min: 0 Max: 13
Step: I get the value 3 Count: 194 Avg: 0 Min: 0 Max: 7

I wanted to query how the metrics are calculated, for example:

• the max value of 1095ms for the feature appears to be significantly higher than the max run for the single scenario (24ms) - why is that? I'd expect the metric to reference the time taken within the step implementations.
• the max value for the scenario (24ms) is not equal to the combined max values for the steps (2 + 0 + 13 + 7 = 22ms). I appreciate 2ms is relatively trivial in this context, however it raises concerns over the integrity of the results.

Perhaps the 13ms for an integer addition can be put down to Java threads competing for the same CPU thread (?), but is this the time it took to only execute the step method, or does it also include time taken to execute internal Cucumber code?

Hi,

Some methods are now unsupported in Cucumber 3 :

package cucumber.runner;

class HookTestStep extends TestStep implements cucumber.api.HookTestStep {
....

    @Override
    @Deprecated
    public String getStepText() {
        throw new UnsupportedOperationException();
    }
}

So if you have a Hook in your scenario it does not work, and we get this execption (bu debbugin):

java.lang.UnsupportedOperationException
	at cucumber.runner.HookTestStep.getStepText(HookTestStep.java:68)
	at cucumber.api.perf.result.StepResultListener$2.receive(StepResultListener.java:30)
	at cucumber.api.perf.result.StepResultListener$2.receive(StepResultListener.java:1)
	at cucumber.runner.EventBus.send(EventBus.java:28)
	at cucumber.runner.TestStep.run(TestStep.java:38)
	at cucumber.runner.TestCase.run(TestCase.java:41)
	at cucumber.runner.Runner.runPickle(Runner.java:44)
	at cucumber.api.perf.PerfCucumberRunner.runScenario(PerfCucumberRunner.java:169)
	at cucumber.api.perf.PerfCucumberRunner.runCucumber(PerfCucumberRunner.java:155)
	at cucumber.api.perf.PerfCucumberRunner.call(PerfCucumberRunner.java:133)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

Please fix

Hi, I am trying to use your library however when initializing an instance of Cucumber Perf, I am getting this error:

cucumber.runtime.FeaturePathFeatureSupplier.<init>(Lcucumber/runtime/model/FeatureLoader;Lcucumber/runtime/RuntimeOptions;)V

This is being triggered within the constructor of CucumberPerf(clazz) on the line:
this.features = FeatureBuilder.getFeatures(ro);

The interesting thing is, that if I debug and go inside getFeatures() and I evaluate the expression new FeaturePathFeatureSupplier(featureLoader, runtimeOptions).get() in debug mode (which is actually the return value of getFeatures() ) it evaluates correctly with an ArrayList containing the feature files details.

I am using io.cucumber cucumber-java 4.2.0 and cucumber-perf 1.3.1

Any clue what the issue might be?

I had this Exception after Following all steps described in the documentation https://github.com/mpinardi/cucumber-performance/wiki/Running-Cucumber-Perf

Note: I have tried 3 way to run and got the same result

Exception in thread "main" java.lang.NoSuchMethodError: cucumber.runtime.RuntimeOptions.cucumberFeatures(Lcucumber/runtime/io/ResourceLoader;Lcucumber/runner/EventBus;)Ljava/util/List;
        at cucumber.api.perf.FeatureBuilder.getFeatures(FeatureBuilder.java:32)
        at cucumber.api.perf.CucumberPerf.<init>(CucumberPerf.java:119)
        at cucumber.api.perf.cli.Main.main(Main.java:7)
        at core.RunCukesTest.main(RunCukesTest.java:36)

(Sorry about the formatting, not sure how to get stuff with < and > around it to render properly....)

Hi, I need some more beginner-level information in the Wiki to get my stuff running.

I'm using Intellij IDEA and am trying to get create a Java class to run my existing Cucumber feature files.

1. What extension, if any, do I need on the Plan file? A ".txt" doesn't seem to be working.

2. I've got lots of dependencies in my Maven POM, including

My main class is

import cucumber.api.junit.Cucumber;
import cucumber.api.perf.CucumberPerf;
import cucumber.api.perf.PerfRuntimeOptions;
import org.junit.runner.RunWith;
import java.util.Arrays;

@RunWith(Cucumber.class)

public class CucumberPerfTest {
public static void main(String[] args) throws Throwable
{
PerfRuntimeOptions options = new PerfRuntimeOptions();
options.addTagFilters(Arrays.asList(new String[]{"@DennisPerfTest"}));
options.addPlanPaths(Arrays.asList(new String[]{"src/test/resources/features"}));
options.addCucumberOptions(Arrays.asList(new String[]{"features = {CUCUMBER_FEATURES}, " + "" +
"plugin = { CUCUMBER_FORMAT_TYPE, CUCUMBER_FORMAT_TEST_RESULTS_PATH, " +
"CUCUMBER_JSON_PATH, CUCUMBER_JUNIT_PATH, CUCUMBER_REPORT_PATH}, " +
"monochrome = CUCUMBER_MONACHROME, dryRun = CUCUMBER_DRY_RUN)"}));
CucumberPerf cukePerf = new CucumberPerf(options);

    cukePerf.runThreads();

}

}

My plan is
@DennisPerfTest

Plan: plan
Simulation: simulation 1
Group browse_places_filter.feature
#slices
#these values will replace property "value out"
|value out|
|changed value 1|
|changed value 2|
#number of threads
Runners: 2
#total number of threads to run.
Count: 2
#a optional random wait mean for before thread runs tests.
#thread will wait between +-50% of this mean
RandomWait: 00:00:02

#Will run all groups for the period below
Simulation Period: simulation 2 period
Group us_persons_banner.feature
|value out|
|changed value |
Threads: 5
#count is ignored in a simulation period
Count: 1
#run time
Time: 00:00:30
RampUp: 00:00:10
RampDown: 00:00:10

but I get

exception in thread "main" java.lang.NoClassDefFoundError: cucumber/runtime/io/URLOutputStream
at cucumber.api.perf.PerfRuntimeOptions.(PerfRuntimeOptions.java:70)
at cucumber.api.perf.PerfRuntimeOptions.(PerfRuntimeOptions.java:60)
at gov.ic.aspace.autotest.stepdefs.CucumberPerfTest.main(CucumberPerfTest.java:23)
Caused by: java.lang.ClassNotFoundException: cucumber.runtime.io.URLOutputStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 3 more

Process finished with exit code 1

Any help is appreciated. Thanks!

Dennis

We tried to implement the performance testing framework for the cucumber test cases and was able to do that. However the problem is all the scenarios in the feature file are getting executed even though the tags were defined. Please help.

//code
options.addCucumberOptions(Arrays.asList(new String[]{"-g","com.cru.cucumber.stepdefinations","-t","@smoke","src/test/java/resources","-p","json:target/cucumber-reports/Cucumber.json","--plugin","html:target/cucumber-htmlreport"}));

Output:
2 Scenarios (�[32m2 passed�[0m)
6 Steps (�[32m6 passed�[0m)
0m25.403s

2 Scenarios (�[32m2 passed�[0m)
6 Steps (�[32m6 passed�[0m)
0m25.357s

Ideally only one scenario should be executed. But 2 scenarios in the feature file are getting executed irrespective of the tag or order.

Executed a single plan via a Main class with the Cucumber perf annotation, and it didn't seem to end the process.

Main class:

@CucumberOptions(plugin = {"null_summary", "junit:target/junit-report.xml"}, //
features = {"src/test/resources/features"}, glue = "org.example.step")
@CucumberPerfOptions(plans = {"src/test/resources/plans"})
public class RunAsMain {

    private static final Logger LOG = LoggerFactory.getLogger(RunAsMain.class);

    public static void main(final String args[]) {
        CucumberPerf perf = new CucumberPerf(RunAsMain.class);
        try {
            perf.runThreads();
        } catch (Throwable e) {
            LOG.error("Error: " + e.getMessage(), e);
        }
    }
}

The Java process remains even after the results are printed.

How to import the maven dependencies in the gradle project??

Is there any way we can report metrics for specific steps rather than all steps? Say if I have 5 steps, I'm more interested in the metrics reported for Step 3 and Step 5 rather than all. I'm curious if it can be controlled in a robust manner. Sometimes I want to see the metric for Step 3 & 5 and other times only Step 4. Please let me know.

Hi there. I'm interested in adopting your cucumber performance test library for a project.
On the surface it looks very impressive, however I am having problems setting the plugins you've mentioned in the Wiki. More specifically, I'm looking to render a better set of reports, rather than the default summary.

Your wiki mentions plugins such as "summary_text:file://C:/test/summary", for example, however I cannot see where that will be used, when using the @CucumberPerfOptions annotation.
I have checked through your codebase and cannot see any reference to the 'plugin' property of the @CucumberPerfOptions annotation.
The PerfRuntimeOptionsFactory class does not appear to use it - should it?

Tested on version 1.3.0.

Hello, I am very keen to use this package and have followed your readme instructions to set it all up with a feature file.

However, no matter what I try the runner simply doesn't seem to do anything. Not tests are being found:

Total Ran: 0
RunTime: PT0.109S

I have this set in a runner file:

public class RunPerformanceTests {
public static void main(String[] args)
{
PerfRuntimeOptions options = new PerfRuntimeOptions();
options.addTagFilters(Arrays.asList(new String[]{"not @Bskip","@performancetest"}));
options.addPlanPaths(Arrays.asList(new String[]{"src/test/java/graphql/testing/features"}));
options.addCucumberOptions(Arrays.asList(new String[]{"-g","features","src/test/java/graphql/testing/features"}));
CucumberPerf cukePerf = new CucumberPerf(options);
System.out.println("Total Ran: "+cukePerf.getTotalRanCount());
System.out.println("RunTime: "+cukePerf.getRunTime());
System.exit(0);
}
}

The paths are correct.

I am using cucumber version 3.0.2. Is this supported or do you suggest a different version?