hyperskill / hs-test Goto Github PK

View Code? Open in Web Editor NEW

31.0 9.0 10.0 1.52 MB

A framework that simplifies testing educational projects for Hyperskill. It is built on top of JUnit.

Java 98.40% Dockerfile 0.01% Go 0.52% JavaScript 0.44% Python 0.59% Shell 0.04%

package

hs-test's Introduction

hs-test

It is a framework that simplifies testing educational projects for Hyperskill. It is built on top of JUnit.

It is required to use for Hyperskill projects. The main features are:

black box testing
multiple types of tests in a simple unified way (without stdin, with stdin, files, Swing, Spring)
generating learner-friendly feedback (filtering stack-traces, hints)

To learn how to use this library you can go here: https://github.com/hyperskill/hs-test/wiki

hs-test's People

Contributors

Stargazers

Watchers

Forkers

servb tony-gk enrodroid msmilkshake lalinsch rabestro iriday misterbucketman polischuks luzerne676

hs-test's Issues

Library in not working with Java 12

Any static variable in the solution causes

java.lang.NoSuchFieldException: modifiers at java.base/java.lang.Class.getDeclaredField(Class.java:2417) at

Fatal Error - IllegalAccessException using Enums

Infinite loop creates Cannot check the submission

Cannot check the submission. Please send the report to Hyperskill team.
stdout:

stderr:

Out of memory creates Cannot check the submission

Error text:

"Cannot check the submission. Please send the report to Hyperskill team. 
stdout:

stderr:

FAILURE: Build failed with an exception.

* What went wrong: 
Execution failed for task ':test'. 
> Process 'Gradle Test Executor 1' finished with non-zero exit value 137 
This problem might be caused by incorrect test process configuration. 
Please refer to the test execution section in the User Manual at https://docs.gradle.org/5.3.1/userguide/java_testing.html#sec:test_execution

* Try: 
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 35s"

Should show to the user full input and output of the test

For example, there can be 1/20 th chance of getting such an output.

Start test 1
Hello
>Hello
Hello
Start test 2
Hello
>Hi
Hi
Start test 3
Hello
>Hihi1
Hihi1
Start test 4
Hello
>

Start test 5
Hello
>1
>Hey
Hey
Start test 6
Hello
>2
2

Use reflection to get user's main method, not direct link from the importing

Make WrongAnswer and TestSuccess mid test (using dynamic input)

Fail while cloning objects results on strange exception

nextInt() + nextLine() + nextInt() = InputMismatch

Need special feedback on this exception

Do not use System.err stream to print "Start test N"

It used because of IDEA marks this stream in red.
Instead, just use standard out stream, but use color
https://stackoverflow.com/questions/5762491/how-to-print-color-in-console-using-system-out-println

Add feedback for predefined io tests

Sometimes we need to create a test like:

input: "aa bb cc"
output "aabbcc"
feedback: input characters are separated by one space.

Refactor: drop ability for testing any method, keep only main

No projects seem to test custom methods (apart of Swing project, but it is easy to fix)
Black box testing ideology is not compatible with knowing that some methods exist in the user code

Prohibit using Asserts in tests

Need to catch AssertionError in solve method and raise Fatal error

Test number (and better, TestCase) should be available during checking

Should start using the "Releases" function in github.

Refactor: remove solve, checkSolve, generatePredefinedIO

Instead of them use lambda functions in the constructor (or builder) of TestCase

Implement testing Android projects

Implenemt testing Spring applications

Some testing files stay after tests, but should be deleted

Not between every test (since sometimes we need to save some data between tests e.g. TextEditor) but after all the tests

If an exception happens during generating tests output shows just (Wrong Answer)

Add native support for feedback in TestCase

Solves #1

Multiple instances of Scanner causes NoSuchElementException but should be OK

Androis is not worling with Java 11

Need to downgrade to Java 8

User didn't close Scanner on file causes java.nio.file.FileSystemException when deleting files

Should add special feedback on this

private main method creates FatalError but should just Error with feedback

Implement dynamically appendable input

Input should be appendable mid-testing based on current output

Allow disabling time limits

For example, if the time limit is negative then consider it to be infinite

Add checkstyle plugin

https://docs.gradle.org/current/userguide/checkstyle_plugin.html

Add ability to catch certain types of Exceptions and return reasonable feedback

Additional test files should not override user's files even if they have the same name

System.exit call should end up with ending current test, not in exception "Tried to exit"

https://github.com/hyperskill/hs-test/blob/dev/src/main/java/org/hyperskill/hstest/dev/exception/FailureHandler.java#L201

Allow throwing some error as the way to fail the test

Now the test is assumed to be failed only if methods .addInput, check return CheckResult object. This can be hard if checking goes deep into several methods (for example, parsing output) and encounteres that something is wrong. Since return statement can be made only in .addInput, check you actually need to write something like:

... deep into methods
       throw new Exception(feedback);
...

... check method
List<Grid> grids;
try {
     grids = Grid.parse(out);
} catch (Exception ex) {
     return CheckResult.FALSE(ex.getMessage());
}
...

but with this issue implemented it can be something like this (less boilerplate):

... deep into methods
       throw new FailTestError(feedback);
...

... check method
List<Grid> grids = Grid.parse(out);
...

static Scanner + any .hasNext* method causes erorrs

Scanner remembers that there aren't any numbers in the input and in the second test provides no input, but should provide correct input.

Add ability to run tests in parallel

Fail if user created additional threads and didn't close them before leaving main

Hide "'>' symbol means the start of the input" line when there was no input

All class membes is equals to null when generating tests.

Example:

// TODO: why if this field in not static then it is null, not String[]?
static final String[] inputs = new String[] {
    "1 1", "1 2", "1 3",
    "2 1", "2 2", "2 3",
    "3 1", "3 2", "3 3"
};

This happens because tests are generated in constructor when no member fields initialized

Fatal error during testing, please send the report to Hyperskill team.

OS Mac OS X
Java 12.0.1
Vendor Oracle Corporation
Testing library version 5

Cannot be cloned: nothing
Cannot be reset: nothing
Cannot deserialize: nothing
Cannot serialize: nothing
Circular links: nothing


java.lang.Exception: No main method found
        at org.hyperskill.hstest.v5.stage.BaseStageTest.initTests(BaseStageTest.java:67)
        at org.hyperskill.hstest.v5.stage.BaseStageTest.start(BaseStageTest.java:120)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:567)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)