Idea: show code coverage information in tree view about atom-phpunit-integrator HOT 16 CLOSED

0.0.19 enables serializing the tree view coverage. I opted out of serializing the other components since the data there is mostly useful only after running a test(s).

Same rule applies, 'Clear All' will remove the coverage in the tree. I can imagine clicking it in a large project would be annoying, but it's by design.

from atom-phpunit-integrator.

Good point about the interfaces, that was something I complained about in netbeans. In the coverage panel I already mark those in orange. I think I'll just omit them from the tree entirely.

Regarding the merging of reports, I should be able to implement idea 2 above, again, since it appears anonymous functions will be removed from the report in sebastianbergmann/php-code-coverage#606. A downside to this is that running a version of phpunit without this fix will yield slightly wrong coverage statistics for classes and methods.

For the percentages in the tree, I'll implement it similar to GitLab and base it on the covered lines, I'll also exclude anything with 0 lines.

from atom-phpunit-integrator.

0.0.22 Should be fully working now.

I have based the coverage of the lines covered rather than the elements, any file with zero lines are excluded. The same has been applied to the coverage view.

Point to note, the console will log a warning message if class/element metrics differ between those counted and those stated in the clover.xml report. This is an issue within php-code-coverage itself and should be fixed in version 6.0.4.

from atom-phpunit-integrator.

I wanted to add something to this, that I believe I've also seen in PhpStorm, but am even less sure about how feasible it would be: I believe PhpStorm can also occasionally run the test suite in the background and update the code coverage information in the tree automatically.

Though, to be fair, for larger projects, this will be hard to pull off in a way that is not noticeable. Even in a project such as PHP Integrator Core, the entire suite takes around 13 seconds with PHPUnit to run. This is without xdebug even loaded and on fast hardware. If xdebug is loaded - but code coverage is not generated -, this already increases to around 20 seconds. Generating with code coverage can take a minute or longer. Tools such as paratest may help here, but spawning processes every so often that hog your computer for more than a few seconds seems like a bad idea.

Still, I wanted to mention it since the feature in itself is nice and handy and it motivates one to keep code coverage up.

In any case, if the information in the Atom tree view would be populated just by running the test suite with code coverage enabled - the same way the code coverage pane is now populated -, I think that would be a major feat already. That is, assuming that this information doesn't disappear if you rerun the tests afterwards without code coverage. It may also be advisable to restore this information between Atom sessions.

from atom-phpunit-integrator.

The test reports are written to the OS's temp directory (though I could easily change this), I had the intention of caching the coverage report for later usage. But, as I've since discovered, any changes to the PHP code break the report (line numbers become offset). It would require a far more advanced caching system to load and use an old report, especially between sessions. For this reason I decided to abandon the idea of loading previous coverage reports. However, keeping track of the percentage number for unmodified files should be a non issue, it's only the specific covered lines (shown as highlights in the source) which are a problem.

The idea of having constant up to date coverage is intriguing. I suppose the easiest method would be to use a crontab. IE the OS would run phpunit every say 5 minutes, then php-unit-integrator could watch a directory for file changes. That shouldn't be too difficult to implement. It could be done in nodejs I suppose, there is not much overhead to spawning a process on a timer. I'll keep this in mind when I code the feature.

from atom-phpunit-integrator.

0.0.17 contains the initial tree view integration. I opted to display a simple percentage and use a single 'info' colour. I don't want the tree view looking like an xmas tree.

I've tried to keep a full tally of the coverage between running single suites/cases. It does this by always selecting the max coverage of the last and current test. This can be problematic if a files true coverage actually decreases. There is no way to avoid it though, not if you want to keep full coverage vs single test coverage in the tree view. Actually, running a full test on a timer in the background would keep it up to date. Not everybody would want that though.

I now need to implement a method of clearing/resetting this component.

from atom-phpunit-integrator.

Neat!

I tried generating some code coverage for PHP Integrator's core and afterwards I saw some percentages pop up in the tree view, but only for some files, which were coincidentally all 0%. I didn't get any error notifications, but did notice this in the developer tools console:

/home/user/.atom/packages/php-unit-integrator/lib/views/php-unit-tree-view.js:244 Uncaught (in promise) Error: A file element for '/home/user/Projects/php-integrator/core/src/Analysis/SourceCodeReading/FileSourceCodeReaderInterface.php' could not be found!
    at DirectoryStats.createFileStats (/home/user/.atom/packages/php-unit-integrator/lib/views/php-unit-tree-view.js:207:10)
    at PhpUnitTreeView.updateCoverageStatistics (/home/user/.atom/packages/php-unit-integrator/lib/views/php-unit-tree-view.js:336:13)
    at /home/user/.atom/packages/php-unit-integrator/lib/views/php-unit-tree-view.js:293:10
    at Function.module.exports.Emitter.simpleDispatch (/usr/lib/atom/node_modules/event-kit/lib/emitter.js:25:14)
    at Emitter.<anonymous> (/usr/lib/atom/node_modules/event-kit/lib/emitter.js:154:44)
    at Emitter.module.exports.Emitter.emitAsync (/usr/lib/atom/node_modules/event-kit/lib/emitter.js:157:12)
    at /home/user/.atom/packages/php-unit-integrator/lib/tester/php-unit-test-runner.js:565:25
    at Generator.next (<anonymous>)
    at step (/home/user/.atom/packages/php-unit-integrator/lib/tester/php-unit-test-runner.js:9:273)
    at <anonymous>

I assume this is the reason the remaining items didn't get populated. The code coverage pane does show all files properly.

Apart from this, I think the current color is fine; in my theme as well as One Dark it's noticeable, but not disturbing.

Might using float: right be a better idea than just padding? That seems to align them all into a column. Your mileage may vary, though. If you decide to keep the padding, I'd lower it to around 5px instead of 10px, or use an equivalent font-based size like 0.3em.

Finally, it may be interesting to store/restore the code coverage percentages in the tree view - and perhaps also the code coverage pane - on shutdown/startup. That way, if you have to wait a long time, at least you can see it again on subsequent sessions without having to rerun it. (Atom has functionalities for this, I believe the tester package also restores its state this way). You could also do the same for the tests tree, if you want. If you want, I can create a separate ticket for that.

from atom-phpunit-integrator.

Hmm, that error indicates a discrepancy between paths listed in the coverage report and paths available in the tree. Are there any symlinks in the project? I'll have to clone and test it myself.

I thought about floating right, the problem is they'll wrap when not enough room. I had a similar problem while displaying the test times in my own tree view. the 10px comes from @component-padding in the theme. I'll play around with it.

from atom-phpunit-integrator.

I do use symlinks sometimes, but the file in this case wasn't a symlink.

FYI, I can trigger a slightly different error in the PHP Integrator Core with just the PHP Integrator Unit Test Suite and code coverage enabled. This hits the throw new Error(A file element for '${currPath}' could not be found!) exception instead, which is probably related.

For some reason, the Autocomplete folder does not seem to get any list of subfolders, whilst the other folders do.

The unit test suite is the fastest one to run, which should make debugging a bit easier.

from atom-phpunit-integrator.

The tree view removes the child elements of collapsed directories. This complicates things.

from atom-phpunit-integrator.

Fixed in 0.0.18

I can't display the coverage span on the right without drastically changing the tree views default styles. Applying float right causes the spans to wrap when the view is too narrow. An ideal solution would be how I implemented my own results tree, where the text is clipped allowing the time to always be visible.

Once this package hits a major version, I'll contact the tree view devs and see if they can't provide some way to do this via a service, much like they already do for the icons.

I'll leave this open and see about caching the results to disk between atom sessions. As mentioned above though, the percentages in the tree are not guaranteed to be accurate. It's merely an accumulation of all test run while coverage is enabled. To get a true result, the "Clear All" button needs clicking, which deletes ALL cache, then a full test run with coverage.

from atom-phpunit-integrator.

Looks really neat. Now that it's serialized, a rerun on every session becomes unnecessary, which is a large improvement in my opinion.

Now to figure out why GitLab claims my coverage is ~87% and the tree 62% 😄.

from atom-phpunit-integrator.

Hmm, There could be an error in my calculations. The only thing that stands out right now is using Math.round for each file rather than returning a float. But, that wouldn't explain a 25% discrepancy.

from atom-phpunit-integrator.

The coverage report returned by phpunit contains metrics for each file. this metrics states the count of covered methods, statements and elements (a combination of the two). For a method to be covered, ALL lines within that method also need to be covered. The problem is, the report does not state which lines belong to which method. This causes a big problem when merging reports. The first test may cover half a method, the second test may cover the other half. Each report will state the method as uncovered, but when combined it should be fully covered.

This is only a problem in the edge case where PHP code is written after a class declaration. In all other instances, I can check if a line is marked as a method (type="method" in the report) then hit-test each line following until the next method. The problem lies with the final method, there is no way of knowing if all following lines are within the method body or actually code written after the class.

One solution is to call php-integrator-base to determine on which line the class ends. This comes with the cost of an async call on each file. It also binds the package to a third party. Also, I don't know how php-integrator-base is going to change in 4.X when the service is removed and if the required methods will still be available. Saying that though, all skelgen files and running from the cursor is already pretty much bound to the service already.

Another solution would be to hit-test all the final lines regardless of if they belong to the method. This would cause a slight discrepancy in the coverage percentage only for this edge case, and only when code after the class is also not fully covered. (I may opt to implement this solution)

A third, is to use only the count of covered lines. This would prevent displaying statistics like 'covered classes', 'covered methods' and 'covered statements' should I want to implement that in the future.

A fourth, but less optimal, would be to exclusively use the metrics returned by phpunit. This would disable the ability to run progressive tests, with the coverage statistics always being that of the last test run.

Previously I only included statement counts in the coverage calculation. While this still does not explain the 25% discrepancy above, it does account for the 3% discrepancy I noticed in my own testbed (symfony-framework-bundle). This discrepancy is only between the ratios of statements and elements. I'm not sure which GitLab uses (I did have a quick look at the source, but it's not obvious where it would be).

from atom-phpunit-integrator.

The second solution above will not work since the report contains a 'method' for each anonymous function declared within the body of a method. These anonymous functions are counted as class methods. While parsing the lines, there is no way to know which line belongs to the inner vs outer function.

Back to square one.

from atom-phpunit-integrator.

Hmm. It also appears that files that are untestable, such as interfaces or classes without methods, are listed as 0/0, which then results in 0%. Perhaps 100%, N/A or simply not showing coverage would be more appropriate for these?

With 0% it appears as if you missed adding tests for a class, whilst there is nothing to test. In the code coverage list this distinction is clearer, since there are no percentages and these items get a yellow background color, in the tree it is not so clear, however.

Also, in case you were wondering: with "GitLab" I meant code coverage being run via GitLab's CI. On the background this is just PHPUnit generating code coverage by running the entire suite. If PHPUnit is run with the --coverage-text option, it then just proceeds to look for the text matching the regular expression ^\s*Lines:\s*\d+.\d+\%, which is then turned into a percentage.

from atom-phpunit-integrator.