mutate: dextool does not detect killed mutant about dextool HOT 7 CLOSED

I generated a mutant by id using dextool mutate generate --id <id>. After extending my test case, I tested that the mutant is killed by manually running the build, test, and analyze script:

mutate/build.sh && mutate/test-all.sh 1>out 2>err && mutate/analyze.py out err

But when I ran the test through dextool, the mutant is not detected as killed. What can be the cause here? Is the steps I used to manually run the tests reflects how dextool runs the tests?

You are correct in your description of how it is executed:

1. build.sh
2. test-all.sh
3. if test-all.sh return exit code != 0 the mutant is marked as "killed" and analyzer is called.

Is the exit status of test-all.sh != 0 when the test cases fail?

from dextool.

Is the exit status of test-all.sh != 0 when the test cases fail?

I think this is why the analyze script is not called. My test script always returns 0 unless the last test case failed. Fixed my test script and now the analyze script is called correctly :)

I'll gladly take any suggestions you have for what you are missing from the documentation :)

These steps should be a good addition.

1. build.sh

2. test-all.sh

3. **if** test-all.sh return exit code != 0 the mutant is marked as "killed" and analyzer is called.

An explananation for killedByCompiler and how it is detected would be good too. I assume, like the test script, it depends on the build script, if the build script returns exit code != 0 then it is marked as killedByCompiler?

from dextool.

Thanks for the explanation btw :)

from dextool.

An explananation for killedByCompiler and how it is detected would be good too. I assume, like the test script, it depends on the build script, if the build script returns exit code != 0 then it is marked as killedByCompiler?

Done

from dextool.

I logged each time the build, test, and analyze script is called and noticed that the analyze script is called only once at the start. I think this is where I misunderstood. I thought before this that the build, test, and analyze script is run for each mutant but according to my log, the analyze script is run only at the start once to detect new/dropped test cases, etc. So that satisfies the docs from README:

To be able to find test cases that kill zero mutants, detect new test cases, and dropped test cases it needs to find all these test cases at the beginning when it is measuring the performance of the test suite. This is why the passed: is important.

And then the next paragraph talks about how the failed test cases are detected:

To be able to map which test case killed which mutant it needs help finding the test cases that failed when the mutant where injected. This is where the failed: part comes in

My question then, how can dextool detects which test case killed which mutant, if the analyze script is only run once at the start?

I set use_schemata to false in the config file.

from dextool.

I logged each time the build, test, and analyze script is called and noticed that the analyze script is called only once at the start. I think this is where I misunderstood. I thought before this that the build, test, and analyze script is run for each mutant but according to my log, the analyze script is run only at the start once to detect new/dropped test cases, etc. So that satisfies the docs from README:

To be able to find test cases that kill zero mutants, detect new test cases, and dropped test cases it needs to find all these test cases at the beginning when it is measuring the performance of the test suite. This is why the passed: is important.

And then the next paragraph talks about how the failed test cases are detected:

To be able to map which test case killed which mutant it needs help finding the test cases that failed when the mutant where injected. This is where the failed: part comes in

My question then, how can dextool detects which test case killed which mutant, if the analyze script is only run once at the start?

If that is the case then there is a bug in dextool that i'll fix! The analyze script shall execute on all killed mutants.

I set use_schemata to false in the config file.

from dextool.

I'll gladly take any suggestions you have for what you are missing from the documentation :)

from dextool.