joaoopereira / dotnet-test-rerun Goto Github PK

First of all, thank you for creating this tool! I recently started using it to rerun our integration tests in our Azure DevOps pipeline. This suite suffers from some ordering effects. We are trying to resolve that issue, but in the meantime we would like to keep our build green. dotnet-test-rerun seems to be made for that use case!

However, I noticed something does not seem to work as intended -- at least in our pipeline. After each test, test-rerun runs the entire test suite again, not just the failed tests. (That is the intended behaviour, right?)

This is the output I see in the integration test task:

Rerun attempt 3/3
Found Failed tests. Rerun filter: [...failing tests...]
working directory: /
forking test [...path...]/[...application...].IntegrationTests.dll --logger "trx" --results-directory [...directory...]

Now I suspect that I should also see --filter in that last line of code -- is that correct? I've been investigating the code base to confirm my suspicion, but I haven't found the place where the filter is set just yet.

This is the script we execute in our build task:

dotnet tool install --global dotnet-test-rerun
test-rerun [...path...]/[...application...].IntegrationTests.dll --results-directory ../../_temp

We need to specify the results directory because the task did not have write access at the original location.

So, in short, two questions:

1. How does dotnet-test-rerun decide which tests to rerun? Where in the code can I find this logic?
2. Do you have any idea why it reruns our entire integration test suite, instead of just the failing tests?

Currently, the DotNet test ADO task allows to run test projects using a glob pattern. For example,

 - task: DotNetCoreCLI@2
    displayName: Run tests
    inputs:
      command: 'test'
      projects: 'src/**/*.[Tt]est?(s).csproj'

This will recursively search all folders for files matching this pattern. It would be great if the tool supported this.

PS. There is a workaround to add all test .csproj files to a solution (.sln). However, this increases the execution time, since on retry, the tool will try and search for the failed tests in all .csproj projects in the .sln file.

Hi, it's me again! First off, thanks for your previous quick fix! Unfortunately, another problem popped up when applying the filters, this time pertaining to our tests decorated with the DataTestMethod in MsTest:

Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
No test matches the given testcase filter `FullyQualifiedName~SomeTest (Argument1, Argument2) in path/Application.IntegrationTests.dll

The problem lies with the brackets (( and )) in the fully qualified name of the test. These should be escaped in order for the filter attribute to work properly; see https://github.com/Microsoft/vstest-docs/blob/main/docs/filter.md.

Now this could be as easy as replacing testsToRerun on line 74 of RerunCommand.cs with Regex.Replace(testsToRerun, @"[()]", @"\$&"), but that implementation is pretty naive and I have some doubts whether it might break some things, e.g. when another logger is used or when a fully qualified name is wrapped in brackets (see the last example in the link above). Any way, I hope this helps in pointing to a more robust fix.

If there's anything I can help with, please let me know!

dotnet test allows passing inline RunSettings as the last arguments on the command line after "-- "

https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test

That is missing in test-rerun. test-rerun only supports the --settings parameter for runsettings.

Context: assume that we use the tool on an .sln file that contains a lot of test (and, maybe, non-test projects).

Upon execution, tool will run dotnet test which will search for tests in all the test projects, execute them. If there are failures, the tool will re-run dotnet test on all the test projects searching for the failed tests, having a performance impact.

Would it be possible, on re-run, to run dotnet test on the test projects (or even the DLLs) that contain the failed tests?

Thanks again for this great library!

Hi, it's me again 😅

At the moment, when you run dotnet-test-rerun in an Azure pipeline, it will always mark the running of the task as successful, even in case of a testsuite that still contains failing tests after the specified amount of reruns.

Failed!  - Failed:     4, Passed:     3, Skipped:     0, Total:     7, Duration: 25 s - Application.IntegrationTests.dll (net6.0)


Finishing: Run integration tests

Is this, in your opinion, intended behaviour?

I would suspect that most users would expect the tool to signal failure in such a case. That would require terminating the process with ExitCode 1 instead of 0, which is what happens now. Would you agree with this assessment?

By the way, I believe the same issue now crops op when an unhandled exception is thrown from the code: it ends with ExitCode 0 and the pipeline simply carries on.

As always, thank you for your great work on maintaining this tool, and also for your patience when handling the host of issues I keep dropping. If there's any more information you need from me, please let me know.

When a theory's subset fails then dotnet-test-rerun re-runs all test cases of the theory.

For reference, I'm running on OSX. If I try and use a relative path for my test assembly I get:

➜  net6.0 git:(feature/DEV-14338) test-rerun ./MyTest.dll -d 1
  Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (arm64)
  The test source file "/./MyTest.dll" provided was not found.

If I specify a full path I get :

➜  net6.0 git:(feature/DEV-14338) test-rerun /Users/tim/Documents/code/some/full/path/bin/Debug/net6.0/MyTest.dll -d 1
  Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (arm64)
  Copyright (c) Microsoft Corporation.  All rights reserved.

  Starting test execution, please wait...

example:

--delay <seconds>: Specifies the delay between test runs in seconds. The default is 1000.

Firstly, many thanks for putting this tool together! It's absolutely excellent, and has saved a lot of man hours.

I have a dedicated test automation solution, built around MSTest, that consists of API and UI Tests.

These different categories of tests are organised into their own unit test projects, and are filtered by TestCategory.

For example “TestCategory=API & TestCategory=SmokeTests”, executes the tests from the API unit test project.

I have an issue however when it comes to the re-run mechanism kicking in, when I have similar test names acrosss both unit test projects, if “Test1” exists in both unit test projects, or is contained within the test name in another test project (something like “Test12” is sufficient), and I have a failure, the re-run doesn’t use the fully qualified name of the failed test, and thus tests across both projects, can be executed.

Log output looks like this, hence my thoughts are that it’s not using a fully qualified name:

Found Failed tests. Rerun filter: FullyQualifiedName~Test1

I would have expected it to be something like MyTests.API.Test1

I orchestrate my tests with docker compose, and typically with an API test stack, I do not deploy a web browser, as it is not needed. With the re-try however, if it re-runs a web test, this causes a failure. As a work around, I’ve added the browser to the API test stack, but this isn’t ideal.

I am trying to run this dotnet test command using this wrapper command but it seems some options doesn't work especially code coverage

dotnet test C:\a\1\s\UnitTests\UnitTests.csproj --logger trx --results-directory C:\a_temp --configuration Release --no-build --verbosity minimal --filter (TestCategory!=DataWarehouse&TestCategory!=Integration) /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:CoverletOutput=C:\a_temp\TestResults\Coverage\UnitTests-TestResults.opencover.xml

what is the equivalent command for above using test-rerun. Thanks.

at the moment it requires a project, but dotnet test does not, it will run also at root level of your solution, even if your tests are hidden in test folder. this would be desireable also for this tool

add support for all options:
https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test

Hi,

Options [-e|--environment <NAME="VALUE">] is not supported. https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test.

Can you please add support?

Thanks.

when running dotnet test without a path/project it will search for all projects within the current and childfolders for test projects. This is handy so when new test-projects are added, they are found automatically. Would be awesome if test-rerun [somepathtodll] [OPTIONS] the somepathtodll would be optional

Thanks for great piece of software!

Executing

test-rerun --logger trx --logger logger2

Returns error

Option '--logger' expects a single argument but 2 were provided.

dotnet test would allow to provide multiple logger arguments.

For some reason, running this on ADO doesn't do test rerun.

This is the way we are using it (altering some sensitive info from project names and variables):

- task: Bash@3
      continueOnError: true
      displayName: 'Run functional tests with retries'
      inputs:
        failOnStderr: false
        targetType: 'inline'
        script: |
          # https://github.com/joaoopereira/dotnet-test-rerun
          dotnet tool install --global dotnet-test-rerun --version 1.8.0 --no-cache
          
          project="$(Build.SourcesDirectory)/src/project.sln"
          filter="(TestCategory=SomeCategory)"
          
          # build the test solution
          dotnet build $project --configuration Debug
          
          # run the tests
         test-rerun $project --configuration Debug --loglevel Debug --delay 3 --rerunMaxAttempts 3 --no-build --results-directory $(Agent.TempDirectory) --settings "$(Build.SourcesDirectory)/src/test.runsettings" --logger trx --verbosity detailed --filter=$filter --inlineRunSettings '"TestRunParameters.Parameter(name=\"name\", value=\"value\")"'

Detailed log output:

...

Total tests: 287
Test Run Failed.
     Passed: 255
     Failed: 29
    Skipped: 3
 Total time: 21.8892 Minutes
     6>VSTest:
         MSB4181: The "Microsoft.TestPlatform.Build.Tasks.VSTestTask" task returned false but did not log an error.
       Done executing task "Microsoft.TestPlatform.Build.Tasks.VSTestTask" -- FAILED.
     6>Done building target "VSTest" in project "project.csproj" -- FAILED.
     6>Done Building Project "/mnt/vss/_work/2/s/src/project.csproj" (VSTest target(s)) -- FAILED.
     1>Done executing task "MSBuild" -- FAILED.
     1>Done building target "VSTest" in project "project.sln" -- FAILED.
     1>Done Building Project "/mnt/vss/_work/2/s/src/project.sln" (VSTest target(s)) -- FAILED.

Build FAILED.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:21:57.57


Test Run Failed.
Test Run Failed.

Exit code 1.
command:
dotnet test /mnt/vss/_work/2/s/src/project.sln --filter "(TestCategory=SomeCategory)" --settings "/mnt/vss/_work/2/s/src/test.runsettings" --logger "trx" --no-build -c "Debug" -v "Detailed" --results-directory "/mnt/vss/_work/_temp" -- "TestRunParameters.Parameter(name=\"name\", value=\"value\")" 
   at dotnet.test.rerun.DotNetRunner.DotNetTestRunner.HandleProcessEnd() in /home/runner/work/dotnet-test-rerun/dotnet-test-rerun/src/DotNetRunner/DotNetTestRunner.cs:line 83
   at dotnet.test.rerun.DotNetRunner.DotNetTestRunner.Run(String arguments) in /home/runner/work/dotnet-test-rerun/dotnet-test-rerun/src/DotNetRunner/DotNetTestRunner.cs:line 58
   at dotnet.test.rerun.DotNetRunner.DotNetTestRunner.Test(RerunCommandConfiguration config, String resultsDirectory) in /home/runner/work/dotnet-test-rerun/dotnet-test-rerun/src/DotNetRunner/DotNetTestRunner.cs:line 38
   at dotnet.test.rerun.RerunCommand.RerunCommand.Run() in /home/runner/work/dotnet-test-rerun/dotnet-test-rerun/src/RerunCommand/RerunCommand.cs:line 51
   at dotnet.test.rerun.RerunCommand.RerunCommand.<>c__DisplayClass6_0.<<-ctor>b__0>d.MoveNext() in /home/runner/work/dotnet-test-rerun/dotnet-test-rerun/src/RerunCommand/RerunCommand.cs:line 42
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()
##[error]Bash exited with code '1'.
Finishing: Run functional tests with retries

Hi.

I think there is some issue with the rerun logic when tests target multiple frameworks, because it doesn't just rerun the test in the framework it failed in, it reruns them both.

I created a simple test project with this content:

[TestClass]
public class UnitTest1
{
    [TestMethod]
    public void TestMethod1()
    {
 #if NET7_0
        Assert.IsTrue(true);
 #else
        Assert.IsTrue(false);
  #endif
     }
  } 
  
 and when I run the tests I get this output:

Determining projects to restore...
All projects are up-to-date for restore.
MultiFrameworkTest -> C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net6.0\MultiFrameworkTest.dll
MultiFrameworkTest -> C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net7.0\MultiFrameworkTest.dll
Test run for C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net6.0\MultiFrameworkTest.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.6.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Failed TestMethod1 [19 ms]
Error Message:
Assert.IsTrue failed.
Stack Trace:
at MultiFrameworkTest.UnitTest1.TestMethod1() in C:\Users\sam.holder\source\repos\MultiFrameworkTest\UnitTest1.cs:line 12

Results File: C:\Users\sam.holder\source\repos\MultiFrameworkTest\sam.holder_CW-LAP-SAM2_2023-10-23_15_28_16.trx

Failed! - Failed: 1, Passed: 0, Skipped: 0, Total: 1, Duration: 30 ms - MultiFrameworkTest.dll (net6.0)
Test run for C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net7.0\MultiFrameworkTest.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.6.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: C:\Users\sam.holder\source\repos\MultiFrameworkTest\sam.holder_CW-LAP-SAM2_2023-10-23_15_28_17.trx

Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 9 ms - MultiFrameworkTest.dll (net7.0)

No tests found with the Outcome Failed in file sam.holder_CW-LAP-SAM2_2023-10-23_15_28_17.trx
Rerun attempt 1/3
Found Failed tests: FullyQualifiedNameMultiFrameworkTest.UnitTest1.TestMethod1
Rerun filter: FullyQualifiedNameMultiFrameworkTest.UnitTest1.TestMethod1
Determining projects to restore...
All projects are up-to-date for restore.
MultiFrameworkTest -> C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net6.0\MultiFrameworkTest.dll
MultiFrameworkTest -> C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net7.0\MultiFrameworkTest.dll
Test run for C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net6.0\MultiFrameworkTest.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.6.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Failed TestMethod1 [15 ms]
Error Message:
Assert.IsTrue failed.
Stack Trace:
at MultiFrameworkTest.UnitTest1.TestMethod1() in C:\Users\sam.holder\source\repos\MultiFrameworkTest\UnitTest1.cs:line 12

Results File: C:\Users\sam.holder\source\repos\MultiFrameworkTest\sam.holder_CW-LAP-SAM2_2023-10-23_15_28_19.trx

Failed! - Failed: 1, Passed: 0, Skipped: 0, Total: 1, Duration: 26 ms - MultiFrameworkTest.dll (net6.0)
Test run for C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net7.0\MultiFrameworkTest.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.6.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: C:\Users\sam.holder\source\repos\MultiFrameworkTest\sam.holder_CW-LAP-SAM2_2023-10-23_15_28_20.trx

Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 9 ms - MultiFrameworkTest.dll (net7.0)

No tests found with the Outcome Failed in file sam.holder_CW-LAP-SAM2_2023-10-23_15_28_20.trx
Rerun attempt 2/3
Found Failed tests: FullyQualifiedNameMultiFrameworkTest.UnitTest1.TestMethod1
Rerun filter: FullyQualifiedNameMultiFrameworkTest.UnitTest1.TestMethod1
Determining projects to restore...
All projects are up-to-date for restore.
MultiFrameworkTest -> C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net6.0\MultiFrameworkTest.dll
MultiFrameworkTest -> C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net7.0\MultiFrameworkTest.dll
Test run for C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net6.0\MultiFrameworkTest.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.6.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Failed TestMethod1 [15 ms]
Error Message:
Assert.IsTrue failed.
Stack Trace:
at MultiFrameworkTest.UnitTest1.TestMethod1() in C:\Users\sam.holder\source\repos\MultiFrameworkTest\UnitTest1.cs:line 12

Results File: C:\Users\sam.holder\source\repos\MultiFrameworkTest\sam.holder_CW-LAP-SAM2_2023-10-23_15_28_23.trx

Failed! - Failed: 1, Passed: 0, Skipped: 0, Total: 1, Duration: 28 ms - MultiFrameworkTest.dll (net6.0)
Test run for C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net7.0\MultiFrameworkTest.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.6.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: C:\Users\sam.holder\source\repos\MultiFrameworkTest\sam.holder_CW-LAP-SAM2_2023-10-23_15_28_24.trx

Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 9 ms - MultiFrameworkTest.dll (net7.0)

No tests found with the Outcome Failed in file sam.holder_CW-LAP-SAM2_2023-10-23_15_28_24.trx
Rerun attempt 3/3
Found Failed tests: FullyQualifiedNameMultiFrameworkTest.UnitTest1.TestMethod1
Rerun filter: FullyQualifiedNameMultiFrameworkTest.UnitTest1.TestMethod1
Determining projects to restore...
All projects are up-to-date for restore.
MultiFrameworkTest -> C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net6.0\MultiFrameworkTest.dll
MultiFrameworkTest -> C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net7.0\MultiFrameworkTest.dll
Test run for C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net6.0\MultiFrameworkTest.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.6.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Failed TestMethod1 [14 ms]
Error Message:
Assert.IsTrue failed.
Stack Trace:
at MultiFrameworkTest.UnitTest1.TestMethod1() in C:\Users\sam.holder\source\repos\MultiFrameworkTest\UnitTest1.cs:line 12

Results File: C:\Users\sam.holder\source\repos\MultiFrameworkTest\sam.holder_CW-LAP-SAM2_2023-10-23_15_28_26.trx

Failed! - Failed: 1, Passed: 0, Skipped: 0, Total: 1, Duration: 25 ms - MultiFrameworkTest.dll (net6.0)
Test run for C:\Users\sam.holder\source\repos\MultiFrameworkTest\bin\Debug\net7.0\MultiFrameworkTest.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.6.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: C:\Users\sam.holder\source\repos\MultiFrameworkTest\sam.holder_CW-LAP-SAM2_2023-10-23_15_28_27.trx

Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 9 ms - MultiFrameworkTest.dll (net7.0)

As you can see even though the test only fails in .net 6.0 its rerun in 7.0 every time as well.

I think you can provide the --framework option to dotnet test so maybe we should parse the framework out of the path for the test results in the trx and use that to rerun only the failing tests