cuppa-framework / cuppa Goto Github PK

Its not currently possible to determine whether a test run was successful or not. When invoking programmatically this is important to know so as to pass along the appropriate exit code as needed.

Runner.run does not reset the tests running flag after finishing a test run, thus preventing Cuppa from running tests twice in one process. In the case of JUnit, this prevents annotating multiple classes with @RunWith.

Cuppa's own unit tests use the internal API to perform their testing. This makes it difficult to write tests for Cuppa extensions, which shouldn't depend on the internal API. For example, testing a test block transform should be straight forward. Ideally, the tests can use the Cuppa test definition API to create the input data for the transform. We should ensure that the unit tests can be written in Cuppa too.

As awesome as Guava is, Cuppa probably shouldn't depend on it or else we risk introducing classpath conflicts if our users want to depend on a different version of Guava.

The project website, http://cuppa.forgerock.org, is currently unavailable.

Using maven-surefire-plugin 2.18 works as expected, however using the latest version of 2.19.1 will result in the following stack trace during Maven clean install:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on project html: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test failed: There was an error in the forked process
[ERROR] org.apache.maven.surefire.util.SurefireReflectionException: java.lang.NoSuchMethodError: org.apache.maven.surefire.providerapi.ProviderParameters.getProviderProperties()Ljava/util/Properties;
[ERROR] at org.apache.maven.surefire.util.ReflectionUtils.instantiateOneArg(ReflectionUtils.java:135)
[ERROR] at org.apache.maven.surefire.booter.ForkedBooter.createProviderInCurrentClassloader(ForkedBooter.java:321)
[ERROR] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
[ERROR] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
[ERROR] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
[ERROR] Caused by: java.lang.NoSuchMethodError: org.apache.maven.surefire.providerapi.ProviderParameters.getProviderProperties()Ljava/util/Properties;
[ERROR] at org.forgerock.cuppa.maven.surefire.CuppaSurefireProvider.<init>(CuppaSurefireProvider.java:52)
[ERROR] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[ERROR] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[ERROR] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[ERROR] at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
[ERROR] at org.apache.maven.surefire.util.ReflectionUtils.instantiateOneArg(ReflectionUtils.java:131)
[ERROR] ... 4 more
[ERROR] -> [Help 1]

Workaround: Switch back to 2.1

If a build tool like Maven or Gradle is not being used, the user will want a simple way to run their Cuppa tests. We should provide a simple command-line client for running tests in a project.

When attempting to use the latest version 2.20.1 of the Maven Surefire Plugin to run the JUnit tests the following error occurs.

I've not investigated it yet to understand the cause.

Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test failed: An API incompatibility was encountered > while executing org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test: java.lang.NoSuchMethodError: > org.apache.maven.surefire.report.SafeThrowable.(Ljava/lang/String;)V
[ERROR] -----------------------------------------------------
[ERROR] realm = plugin>org.apache.maven.plugins:maven-surefire-plugin:2.20.1
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/Users/.../org/apache/maven/plugins/maven-surefire-plugin/2.20.1/maven-surefire-plugin-2.20.1.jar
[ERROR] urls[1] = file:/Users/.../org/forgerock/cuppa/cuppa-surefire/1.3.1/cuppa-surefire-1.3.1.jar
[ERROR] urls[2] = file:/Users/.../org/apache/maven/surefire/common-java5/2.19.1/common-java5-2.19.1.jar
[ERROR] urls[3] = file:/Users/.../org/forgerock/cuppa/cuppa/1.3.1/cuppa-1.3.1.jar
[ERROR] urls[4] = file:/Users/.../org/apache/maven/surefire/maven-surefire-common/2.20.1/maven-surefire-common-2.20.1.jar
[ERROR] urls[5] = file:/Users/.../org/apache/maven/plugin-tools/maven-plugin-annotations/3.3/maven-plugin-annotations-3.3.jar
[ERROR] urls[6] = file:/Users/.../org/apache/maven/surefire/surefire-api/2.20.1/surefire-api-2.20.1.jar
[ERROR] urls[7] = file:/Users/.../org/apache/maven/surefire/surefire-logger-api/2.20.1/surefire-logger-api-2.20.1.jar
[ERROR] urls[8] = file:/Users/.../org/apache/maven/surefire/surefire-booter/2.20.1/surefire-booter-2.20.1.jar
[ERROR] urls[9] = file:/Users/.../org/codehaus/plexus/plexus-utils/1.5.15/plexus-utils-1.5.15.jar
[ERROR] urls[10] = file:/Users/.../junit/junit/4.12/junit-4.12.jar
[ERROR] urls[11] = file:/Users/.../org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
[ERROR] urls[12] = file:/Users/.../backport-util-concurrent/backport-util-concurrent/3.1/backport-util-concurrent-3.1.jar
[ERROR] urls[13] = file:/Users/.../org/codehaus/plexus/plexus-interpolation/1.11/plexus-interpolation-1.11.jar
[ERROR] urls[14] = file:/Users/.../org/slf4j/slf4j-jdk14/1.5.6/slf4j-jdk14-1.5.6.jar
[ERROR] urls[15] = file:/Users/.../org/slf4j/jcl-over-slf4j/1.5.6/jcl-over-slf4j-1.5.6.jar
[ERROR] urls[16] = file:/Users/.../org/apache/maven/reporting/maven-reporting-api/3.0/maven-reporting-api-3.0.jar
[ERROR] urls[17] = file:/Users/.../org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar
[ERROR] urls[18] = file:/Users/.../org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import from realm ClassRealm[maven.api, parent: null]]

Using 2.19.1 works ok.

We need to write a Gradle plugin to make it possible for Gradle projects to run Cuppa tests.

It would be nice to be able to pass data between blocks, so:

{
  describe("addition", () -> {
    when("adding two numbers"), () -> {
      beforeEach(() -> {
        return new Calculator.enter(5).plus().enter(5).equals();
      }).it("has added them", (calc) -> {
        assertThat(calc.value()).isEqualTo(10);
      });
    });
  });
}

https://plugins.jetbrains.com/plugin/8254

Currently Cuppa runs before and beforeEach hooks before a skipped or pending test and after and afterEach hooks after it. This is inefficient and can lead to unexpected test failures if any of the hooks happen to fail.

Need more details to investigate this. @philostler @phillcunnington Can you provide any more details?

The threshold for what is considered slow should be configurable.

We need to know what where our testing blindspots are.

Would be awesome to integrate with coveralls.io too.

Hi. Thanks for such an awesome framework!

As I go on learning the framework, there are things that I can't find an obvious solution for (maybe I'm not thinking correctly on how to leverage the notion of a test block). Regardless, is there a straightforward way to:

1. Run multiple @Test annotated classes inside a single test suite? For example, using Spock and JUnit, I can say:

FILE: CombinedTestSuite.groovy

import org.junit.runner.RunWith
import org.junit.runners.Suite

@RunWith(Suite)
@Suite.SuiteClasses([MyTestClassWhichExtendsSpocksSpecificationBaseClass1, MyTestClassWhichExtendsSpocksSpecificationBaseClass2, MyTestClassWhichExtendsSpocksSpecificationBaseClass3])
class CombinedTestSuite {}

2. If I have a single CombinedTestSuite @Test annotated class under scr/test/java/integration, Maven will pick it up and run its defined tests if I use mvn test. However, if I say: mvn test site, the generated site documentation doesn't include the Cuppa test class.

3. Also, I'm trying to run Cuppa tests through Jacoco, also with mvn test site, but it's saying:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running Cuppa
  This is a test
LOL I AM A TEST
    ✓ does some cool stuff...
  1 passing
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in Cuppa

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-site-plugin:3.6:site (default-site) @ web-backend-test ---
[INFO] configuring report plugin org.apache.maven.plugins:maven-project-info-reports-plugin:2.7
[INFO] configuring report plugin org.apache.maven.plugins:maven-checkstyle-plugin:2.17
[INFO] configuring report plugin org.apache.maven.plugins:maven-jxr-plugin:2.3
[INFO] configuring report plugin org.apache.maven.plugins:maven-pmd-plugin:3.0.1
[INFO] configuring report plugin org.codehaus.mojo:findbugs-maven-plugin:3.0.1
[INFO] configuring report plugin org.apache.maven.plugins:maven-javadoc-plugin:2.10.3
[INFO] configuring report plugin org.jacoco:jacoco-maven-plugin:0.7.7.201606060606
[INFO] configuring report plugin org.apache.maven.plugins:maven-changes-plugin:2.12
[INFO] configuring report plugin org.apache.maven.plugins:maven-surefire-report-plugin:2.19.1
[INFO] Fork Value is true
[INFO] Done FindBugs Analysis....
[INFO] Skipping JaCoCo execution due to missing execution data file.
[ERROR] No Issue Management set. No JIRA Report will be generated.

Thanks!

I can't see any real benefit to this distinction.

Currently Cuppa's default reporter does not make it clear which hook was responsible for a test failure.

We should provide an Eclipse plugin to help with development of Cuppa tests.

None of the core contributors to Cuppa use Eclipse so we are looking for help from the community to do this.

We should build it in an iterative way, so the first version should be relatively simple and act to demonstrate that we can do a release to the Eclipse plugin repository.

See the IntelliJ plugin for ideas.

Currently the Behaviour enum is in a different package (model) and IntelliJ can't discover the import automatically.

This may be reason to consider moving them to static methods on Cuppa.

Calling TestBuilder#skip() and TestBuilder#only() should be mutually exclusive, but it is possible to call both methods on the same object. We should fail fast in this case.

when("a", () -> {
    it("test1");
});
when("a", () -> {
    it("test2");
});

Expected:

Reports a single a when block with two tests, test1 and test2, nested under it.

Actual:

Reports two a when blocks with one test nested under each.

When you have two tests with the same name, JUnit reports the second test incorrectly.

when("a", () -> {
    it("test");
});
when("b", () -> {
    it("test");
});

Expected:

JUnit reports test nested in "suite" a and another test nested in "suite" b.

Actual:

JUnit reports test nested in "suite" a and another test at the root level.

TestNG doesn't seem to offer any mechanism for integration like JUnit's @RunWith. One option would be for us to provide a tool that runs TestNG and Cuppa tests and aggregates the results (like Surefire does).

We're open to ideas.

Cuppa should have a choice of reporters out of the box to suite different needs.

Right now, if a test class that defines Cuppa tests is instantiated by something other than Cuppa it fails with a confusing error (something to do with the test class being null). We should catch this case early and let the user know that something else is instantiating their test class by mistake.

The timeout should be configurable, preferably using the existing with() mechanism. E.g.

with(testTimeout(5, TimeUnit.SECONDS)).
it("...", () -> {
  // ...
});

The following code should be equivalent but is currently not. The result is that some tags are overridden.

with(tags("a"), tags("b"))

with(tags("a", "b"))

If CuppaRunner also attempted to run the class through the default JUnit runner, then users should be able to freely mix the two types of tests in one class, which may aid adoption.