Support of Visual Studio Code Coverage reports about sonar-cxx HOT 18 CLOSED

Hi Guenter,

can you provide some background to what would we gain. Does it have advantages over the channels / formats we already have. What formats does VS provide, which tools would one use etc.

from sonar-cxx.

Hi,

Use Case:
Support Visual Studio coverage files with C++ plugin

Steps to do:

1. Generate a Visual Studio 2012 coverage file with: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" collect /output:out.coverage (...)

Where (...) is performed like (could be also another test runner)
mstest.exe /testmetadata:.\some.vsmdi /testlist:03_COMP_smoke/clientAndServer/pass /runconfig:.\localtestrun2010.testrunconfig /resultsfile:testResults.trx

The CodeCoverage.exe is run as master of the unit test runner.

2. Generate a native xml file from the coverage file:
   C:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" analyze /output:mstest_coverage.xml out.coverage

3. The mstest_coverage.xml file now has to be processed for SonarQube.

Thinking awhile over this I think it would be sufficient to add also a possibility for a XSL transformation for coverage data:

sonar.cxx.coverage.xsltURL= …

Regards

from sonar-cxx.

i might be interested in this feature, we are paying bullseye licences to
get coverage and we also pay visual studio licenses. So might be that we
scrap one, so there is definitely the need to visual studio users.

On Wed, Dec 4, 2013 at 12:02 PM, Günter Wirth [email protected]:

Hi,

Use Case:
Support Visual Studio coverage files with C++ plugin

Steps to do:

1. Generate a Visual Studio 2012 coverage file with: C:\Program Files
   (x86)\Microsoft Visual Studio 11.0\Team Tools\Dynamic Code Coverage
   Tools\CodeCoverage.exe" collect /output:out.coverage (...)

Where (...) is performed like (could be also another test runner)
mstest.exe /testmetadata:.\some.vsmdi
/testlist:03_COMP_smoke/clientAndServer/pass
/runconfig:.\localtestrun2010.testrunconfig /resultsfile:testResults.trx

The CodeCoverage.exe is run as master of the unit test runner.

2. Generate a native xml file from the coverage file:
   C:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Dynamic
   Code Coverage Tools\CodeCoverage.exe" analyze /output:mstest_coverage.xml
   out.coverage

3. The mstest_coverage.xml file now has to be processed for SonarQube.

Thinking awhile over this I think it would be sufficient to add also a
possibility for a XSL transformation for coverage data:

sonar.cxx.coverage.xsltURL= …

Regards

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/79#issuecomment-29792458
.

from sonar-cxx.

@guwirth Thanks for summarizing this up. I could add another point: the gcov-chain on Linux (at least for the releases we tried so far) does not produce high-quality results. The branch counts are often wrong, there are branches listed on places where there are'nt any (at least in source code) etc. Apropos: how does the MS toolchain feels like? Does it produce acceptable results?

The other tool-chain we support right now is bullseye. For me, its not ideal either: it doesnt support line coverage and doesnt map good to the Sonar model.

As for the 'sonar.cxx.coverage.xsltURL' thing: I dont like it. My feeling (and experience, partly) is that it doesnt give us much. I think, transformation should be done outside of the plugin (using Python, Perl, ..., and whatever approach you like, including XSLT, if you're a fan of).

from sonar-cxx.

Support for MS-test results are also interesting for me and we use bullseye until now which could then be discontinued. By the way, the C# plug-in V2.2. (Feb. 2014) will support MS-Test.

SONARDOTNT-357 - Make it possible to reuse MSTest reports
SONARDOTNT-358 - Make it possible to reuse MS Coverage reports

from sonar-cxx.

To draw a conclusion: that would make sense, apparently. Im not interested in this, personally, so if you guys are, promote the issue to 'enhancement' and give it a shot.

The implementation choices from my point of view are: either write something external to convert to cobertura format or add another parser to the coverage sensor. Converting to bullseye format doesnt make sense as the format maps badly to the sonar model. BTW: does MS coverage support both line and branch coverage? cannot quite understand what I find via google.

from sonar-cxx.

... how does the MS toolchain feels like?

After configuring tool in the right way results are not bad but also sometimes suprising:
See http://blogs.msdn.com/b/jsocha/archive/2011/08/17/interpreting-c-code-coverage-results.aspx

... bullseye ... doesnt support line coverage

Visual Studio is supporting:

Block-based statement coverage
For the purposes of the tools, a block is defined as a sequence of instructions that have a single entry point and a single exit point. Exit points include branch instructions, a function call, a return instruction, or, for managed code, a throw instruction.

Line-based coverage
For line-based coverage, the tools identify all of the blocks that make up a line and then use this information to determine the level of coverage for the line. If all of the blocks that make up the line are covered, then the tools report that the line is covered. If no blocks in the line are covered, then the tools report that the line is not covered. If some, but not all, of the blocks in the line are covered, then the tools report that the line is partially covered.

See also http://msdn.microsoft.com/en-us/library/dd537628.aspx

As for the 'sonar.cxx.coverage.xsltURL' thing: I dont like it.

I think plugin should be consistent. Transformation should be available for all or none XML reports.

from sonar-cxx.

On 12/05/2013 10:42 PM, Günter Wirth wrote:

    ... how does the MS toolchain feels like?

After configuring tool in the right way results are not bad but also
sometimes suprising:
See
http://blogs.msdn.com/b/jsocha/archive/2011/08/17/interpreting-c-code-coverage-results.aspx

Thanks for the link, Guenter. I expected something like this: there are
no simple things in C++ land ;)
But its all good as long we can get it compressed/converted to the Sonar
model.

    ... bullseye ... doesnt support line coverage

Visual Studio is supporting:

Block-based statement coverage
For the purposes of the tools, a block is defined as a sequence of
instructions that have a single entry point and a single exit point.
Exit points include branch instructions, a function call, a return
instruction, or, for managed code, a throw instruction.

Line-based coverage
For line-based coverage, the tools identify all of the blocks that make
up a line and then use this information to determine the level of
coverage for the line. If all of the blocks that make up the line are
covered, then the tools report that the line is covered. If no blocks in
the line are covered, then the tools report that the line is not
covered. If some, but not all, of the blocks in the line are covered,
then the tools report that the line is partially covered.

See also http://msdn.microsoft.com/en-us/library/dd537628.aspx

    As for the 'sonar.cxx.coverage.xsltURL' thing: I dont like it.

I think plugin should be consistent. Transformation should be available
for all or none XML reports.

So think I. And as I strive for the second alternative (Transformation
should be available for ... none XML reports) in the mid-term, Id like
to not build more stuff on it.

—
Reply to this email directly or view it on GitHub
#79 (comment).

from sonar-cxx.

One of the biggest advantages of the community plugin is the openness. So if there is an easy way in Java or Sonar to support transformation I would vote for it. As more out of the box is supported as easier the plugin is to use.

from sonar-cxx.

Conceptually true, but in this case: the is an "easy way to support transformation in Java or Sonar" namely:
$ | conversion | sonar_runner
Just put the conversion step logically between the 'report creator' and Sonar. I'd say it's as easy as the current Cxx-solution and even more flexible, because you are not constrained to use XSLT. The first appoach also looks better from the engineering POV because of looser coupling.

from sonar-cxx.

Would be helpful to add a 'Design Decisions' page to Wiki to document such things.

from sonar-cxx.

The other tool-chain we support right now is bullseye. For me, its not ideal either: it doesnt support line coverage and doesnt map good to the Sonar model.

@wenns What exactly is the problem with Bullseye?

BullseyeCoverage Measurement Technique: http://www.bullseye.com/measurementTechnique.html
Why do you think line coverage is missing (http://www.bullseye.com/coverage.html#basic_statement)? Isn't Condition/Decision Coverage the better choice?

from sonar-cxx.

Bullseye provides function coverage + condition/decision coverage only.

Line coverage is not available.

Conceptually the sonar model is completely different than this, in sonar
they count lines hits + branch coverage. Branch coverage in sonar sucks
because it does not give you which condition are covered only the absolute
number.

In bullseye this is available and its really helpull to create new test
cases to produce more coverage.

So what we done was to approximate the 2 models, the condition/decision is
well mapped to sonar model although we loose the more information. And the
function coverage is mapped to line coverage, which is not visually correct
I sonar since you will see lines inside functions without any coverage
information
On Feb 22, 2014 5:35 PM, "Günter Wirth" [email protected] wrote:

The other tool-chain we support right now is bullseye. For me, its not
ideal either: it doesnt support line coverage and doesnt map good to the
Sonar model.

@wenns https://github.com/wenns What exactly is the problem with
Bullseye?

BullseyeCoverage Measurement Technique:
http://www.bullseye.com/measurementTechnique.html
Why do you think line coverage is missing (
http://www.bullseye.com/coverage.html#basic_statement)? Isn't Condition/Decision
Coverage the better choice?

Reply to this email directly or view it on GitHubhttps://github.com//issues/79#issuecomment-35805367
.

from sonar-cxx.

Another think I forgot, when using bullseye sensor line coverage is
actually function coverage in the sonar ui. This is a bit confusing if you
use the sonar ui since it calls it line coverage
On Feb 22, 2014 5:50 PM, "Jorge Costa" [email protected] wrote:

Bullseye provides function coverage + condition/decision coverage only.

Line coverage is not available.

Conceptually the sonar model is completely different than this, in sonar
they count lines hits + branch coverage. Branch coverage in sonar sucks
because it does not give you which condition are covered only the absolute
number.

In bullseye this is available and its really helpull to create new test
cases to produce more coverage.

So what we done was to approximate the 2 models, the condition/decision is
well mapped to sonar model although we loose the more information. And the
function coverage is mapped to line coverage, which is not visually correct
I sonar since you will see lines inside functions without any coverage
information
On Feb 22, 2014 5:35 PM, "Günter Wirth" [email protected] wrote:

The other tool-chain we support right now is bullseye. For me, its not
ideal either: it doesnt support line coverage and doesnt map good to the
Sonar model.

@wenns https://github.com/wenns What exactly is the problem with
Bullseye?

BullseyeCoverage Measurement Technique:
http://www.bullseye.com/measurementTechnique.html
Why do you think line coverage is missing (
http://www.bullseye.com/coverage.html#basic_statement)? Isn't Condition/Decision
Coverage the better choice?

Reply to this email directly or view it on GitHubhttps://github.com//issues/79#issuecomment-35805367
.

from sonar-cxx.

Like to add Visual Studio coverage support. See below an example for coverage result calling with myFunc(0, 1):

#include "stdafx.h"

void myFunc(int a, int b)
{
    int x = 0;
    int y = 0;

    if( a ) {
        x = 1;
        y = 1;
    }

    if( b )
    {
        x = 2;
        y = 2;
    }

    if( a &&
        b
        ) {
            x = 3;
            y = 3;
    }
}

Resulting Visual Studio XML coverage file:

<?xml version="1.0" encoding="UTF-8" ?>
<results>
  <modules>
    <module name="unittest1.dll" path="unittest1.dll" id="93AA4138F7AFB24C91DC9614B700B83C07000000" block_coverage="72.73" line_coverage="68.75" blocks_covered="8" blocks_not_covered="3" lines_covered="11" lines_partially_covered="1" lines_not_covered="4">
      <functions>
        <function id="77200" name="myFunc" type_name="" block_coverage="62.50" line_coverage="61.54" blocks_covered="5" blocks_not_covered="3" lines_covered="8" lines_partially_covered="1" lines_not_covered="4">
          <ranges>
            <range source_id="0" covered="yes" start_line="4" start_column="0" end_line="4" end_column="0" />
            <range source_id="0" covered="yes" start_line="5" start_column="0" end_line="5" end_column="0" />
            <range source_id="0" covered="yes" start_line="6" start_column="0" end_line="6" end_column="0" />
            <range source_id="0" covered="yes" start_line="8" start_column="0" end_line="8" end_column="0" />
            <range source_id="0" covered="no" start_line="9" start_column="0" end_line="9" end_column="0" />
            <range source_id="0" covered="no" start_line="10" start_column="0" end_line="10" end_column="0" />
            <range source_id="0" covered="yes" start_line="13" start_column="0" end_line="13" end_column="0" />
            <range source_id="0" covered="yes" start_line="15" start_column="0" end_line="15" end_column="0" />
            <range source_id="0" covered="yes" start_line="16" start_column="0" end_line="16" end_column="0" />
            <range source_id="0" covered="partial" start_line="21" start_column="0" end_line="21" end_column="0" />
            <range source_id="0" covered="no" start_line="22" start_column="0" end_line="22" end_column="0" />
            <range source_id="0" covered="no" start_line="23" start_column="0" end_line="23" end_column="0" />
            <range source_id="0" covered="yes" start_line="25" start_column="0" end_line="25" end_column="0" />
          </ranges>
        </function>
      </functions>
      <source_files>
        <source_file id="0" path="x:\coveragetest\source.cpp" checksum_type="MD5" checksum="8A4EC27BE0FE2BCE77401F836581836D">
        </source_file>
      </source_files>
    </module>
  </modules>
</results>

Adding line coverage is no problem. Information for condition coverage is not available, only block coverage on function level. My idea is to add block coverage as condition coverage for the myFunc line:
line 3, 5+3 blocks, 5 blocks covered

Any other ideas?

from sonar-cxx.

What a pity... I hoped that we can get both, line and branch coverage. As
for other ideas: I think, missing branch coverage is actually better that
one thats only partually correct and potentially misleading. Things like
"if you use MS coverage, have in mind that branch coverage is actually
, in case of bullseye do not pay attention to <this and that

, bla" are very annoying, IMO. If the core allows it (not sure about
that), I would just go with a fully present (and hopefully correct) line
coverage and skip branch coverage altogether.

2014-04-24 8:19 GMT+02:00 Günter Wirth [email protected]:

Like to add Visual Studio coverage support. See below an example for
coverage result calling with myFunc(0, 1):

#include "stdafx.h"
void myFunc(int a, int b){
int x = 0;
int y = 0;

if( a ) {
    x = 1;
    y = 1;
}

if( b )
{
    x = 2;
    y = 2;
}

if( a &&
    b
    ) {
        x = 3;
        y = 3;
}}

Resulting Visual Studio XML coverage file:

Adding line coverage is no problem. Information for condition coverage is
not available, only block coverage on function level. My idea is to add
block coverage as condition coverage for the myFunc line:
line 3, 5+3 blocks, 5 blocks covered

Any other ideas?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/79#issuecomment-41246121
.

from sonar-cxx.

What a pity... I hoped that we can get both, line and branch coverage.

You get both. SonarQube is calculating the branch coverage in the file header by accumulating the condition coverage. With the API you can set only setHits for line coverage and setConditions for condition coverage. By setting the condition coverage to the first line of the function SonarQube shows an accurate branch coverage.

Things like "if you use MS coverage, have in mind that branch coverage is actually , in case of bullseye do not pay attention to <this and that, bla" are very annoying, IMO.

Because the tools are measuring different coverage types the result is different. I think it is more important that the native usage of the coverage tools and the results in SonarQube are the same. The coverage tools itself can't be compared with one another.

from sonar-cxx.

Docu: https://github.com/wenns/sonar-cxx/wiki/Get-code-coverage-metrics

from sonar-cxx.