r-lib / covr Goto Github PK

1. Load all functions with traced calls
2. Run
   1. tests (like CRAN would, should support at least testthat and RUnit, preferably everything)
   2. examples
3. Return test coverage per line for each file

Is it possible to get coverage on .onLoad() and similar functions? I've tried, but get zero coverage for all those.

EDIT: I eventually got .onUnload() covered (https://coveralls.io/files/438500374), but I do get zero coverage on .onAttach().

This might very well be operator error, but ...

ff = function(x) identity(x)
function_coverage("ff", ff(1))

Results in

Error in aggregate.data.frame(mf[1L], mf[-1L], FUN = FUN, ...) : 
  no rows to aggregate

What am I missing? Thanks

Cool package! I noticed that if you run the package_coverage function, it gives you a ton of output, that I'm guessing gets uploaded to the coveralls service to give you a badge. However, when I'm trying to improve test coverage, I would want to be able to just re-run package_coverage as I add tests and see how the coverage changes.

Is there a way to generate a % coverage value from the package_coverage output? I'd be willing to write a summarize method if I knew how to interpret the values that are being returned.

I think covr should have the option of not reporting test coverage if any test has been skipped by testthat::skip.

I tend to use the skip function only when a test depends on some external web resource that might not be available. testthat:skip allows intermittent failures to reach that external web resource without the red warning signs of a failed test. If the tests are being skipped on a particular build because the resource was unavailable, it does not seem reasonable for the coverage to drop instead of just remaining at what it's last value was.

Not actually sure how to go about this. Would have to dig into @hadley's testthat code to see if skip is giving any other programmatic indication that a test has been skipped, other than the visual printing S instead of . where the test has would have been run in the testthat summary.

Hi,
I am about to release quickcheck 3.1.0. It's a testing package and has covr as an optional dep. For lack of releases, I am pointing my users to commit b181831 because I tested on it, but I am sure you are not looking forward to bugs being reported referencing a random commit in the past. So if you planned to start doing releases, here is a vote to get that going sooner rather than later. Thanks

• https://metacpan.org/pod/Devel::Cover
• https://github.com/z4r/python-coveralls
• https://github.com/eddyxu/cpp-coveralls
  • might be able to use this for the Rcpp/C code in R packages
• https://gcc.gnu.org/onlinedocs/gcc/Gcov.html
• https://github.com/integrum/SublimeRubyCoverage

I've just faced the below strange error:

$ git clone https://github.com/Rapporter/pander.git
$ R 
> library(covr)
> package_coverage('pander')

Error in seq.default(df[i, "first_line"], df[i, "last_line"]) : 
  'from' cannot be NA, NaN or infinite
> traceback()
9: stop("'from' cannot be NA, NaN or infinite")
8: seq.default(df[i, "first_line"], df[i, "last_line"])
7: seq(df[i, "first_line"], df[i, "last_line"])
6: match(x, table, nomatch = 0L)
5: seq(df[i, "first_line"], df[i, "last_line"]) %in% excl[[file]]
4: FUN(1:1553[[1552L]], ...)
3: vapply(seq_len(NROW(df)), function(i) {
       file <- df[i, "filename"]
       file %in% names(excl) && all(seq(df[i, "first_line"], df[i, 
           "last_line"]) %in% excl[[file]])
   }, logical(1))
2: exclude(coverage, exclusions = exclusions, exclude_pattern = exclude_pattern, 
       exclude_start = exclude_start, exclude_end = exclude_end)
1: package_coverage()

I will investigate further later, just wanted to make a note of it for the time being.

Related session info:

> sessionInfo()
R version 3.1.2 (2014-10-31)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=hu_HU.utf8       LC_NUMERIC=C             
 [3] LC_TIME=en_US.UTF-8       LC_COLLATE=en_US.UTF-8   
 [5] LC_MONETARY=en_US.UTF-8   LC_MESSAGES=hu_HU.utf8   
 [7] LC_PAPER=hu_HU.utf8       LC_NAME=C                
 [9] LC_ADDRESS=C              LC_TELEPHONE=C           
[11] LC_MEASUREMENT=hu_HU.utf8 LC_IDENTIFICATION=C      

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] covr_0.2.0.9000 pander_0.5.2   

loaded via a namespace (and not attached):
[1] devtools_1.7.0  digest_0.6.8    lazyeval_0.1.10 magrittr_1.5   
[5] Rcpp_0.11.4     rex_0.2.0       tools_3.1.2  

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: /build/gcc-multilib/src/gcc-4.9-20141224/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-cloog-backend=isl --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-multilib --disable-werror --enable-checking=release
Thread model: posix
gcc version 4.9.2 20141224 (prerelease) (GCC) 

I am not sure if the issue is in travis not running gcov properly or an issue with the json to coveralls, but digest contains mostly c code and it is not being reported. https://coveralls.io/jobs/3980390

Actually there is a bug locally as well, the source coverage only works the second time you run it.

Seems it doesn't pick up on stuff in inst/tests. All of my packages have tests there per an old comment from Hadley, but this convention might have been deprecated. Should I move all my tests for all of my packages to tests off the package root?

It differs from testthat::test_check in that the tests are run from the installed directory always.

I need to ensure that the tests are calling the new functions, or modify the installed namespace functions at runtime.

Ala r-lib/lintr#37

Would be nice to report only the blank lines

Along with #51, it would be handy to report of coverage of vignettes separately as well.

It would be a more complete solution because it actually changes the function pointer and would hopefully fix some edge cases.

If multiple covr sessions are used in parallel (which may not be unlikely on server farms), there is a risk that '~/.R/Makevars' is being rewritten while another R session is reading it leading to a potentially corrupt read.

One way to lower the risk for this is to have set_makevars() only modify '~/.R/Makevars' iff the content actually changes, i.e. only call writeLines(..., lines) if lines have changed from the originally read one.

This solution does not fully cover the case when a pre-existing '~/.R/Makevars' is changed, because then reset_makevars() in one session will undo what another session might have written. However, by only rewriting/undoing the file if something changes the risk for incorrect content is lowered.

The long-term solution is probably for R to provide a command-line option/system environment variable where one can specify a session-specific 'Makevars' file path.

Ideal features would be

Must haves

• select multiple files
• highlight lines that are not covered

Like to haves

• Select files using file system tree
• Count of line coverage for each covered line
• Toggle between test/example/vignette coverage

Does not need to track coverage over time as that is what coveralls.io is mainly for.

Does the package actually depend on having R 3.1.2? I was able to clone, change the Depends field in DESCRIPTION to R 3.1.0, and install on my system with R 3.1.1, and run it without difficulty.

Many packages don't have any tests, but often still have examples. Would be useful to see what code is covered by examples at least.

Love this tool, but when I have a single line function without braces, e.g.

f <- function(x)
  x

the coverage tool neither detects it is covered nor uncovered by tests. (Or at least coveralls.io doesn't display it.)

Is there any chance you might be able to combine covr with an existing C++ code coverage tool in order to also give C++ code coverage stats?

function(x) {
  x
}

These lines do not have a srcref apart from their enclosing function. Distinguishing them from symbols in a regular call would be somewhat tricky as well, and what increase the number of trace calls with little gain (IMHO).

However this situation should be documented (probably in the README)

as of #47 these are being skipped, but it would be better track these functions as well.

To handle weird things like data.table's testing situation.

Mostly an issue with testing, you probably wouldn't call it with any other argument in practice.

Ex. https://coveralls.io/files/410126873#L68

Need to tweak the logic to check if a given line is blank.

I am working on a tools for creating test cases for different VM implementations of R and I was thinking if it's possible to measure coverage of functions from default packages with covr.

For example,

function_coverage("colMeans", colSums(UCBAdmissions), env=getNamespace("base"))

or

ctl <- c(4.17,5.58,5.18,6.11,4.50,4.61,5.17,4.53,5.33,5.14)
trt <- c(4.81,4.17,4.41,3.59,5.87,3.83,6.03,4.89,4.32,4.69)
group <- gl(2, 10, 20, labels = c("Ctl","Trt"))
weight <- c(ctl, trt)
lm.D9 <- lm(weight ~ group)
function_coverage("lm", lm(weight ~ group), env=getNamespace("stats"))

But now, it seems that trace_calls does not insert any trace points into those functions. How can it be changed?

You may have mentioned this before but it is quite disturbing that when writing ifs without optional brackets coverage is overestimated. Is this a bug? Thanks

> ff
function(x = T) {
    if(x) {
      1}
    else{
      0}
  }
> cov = function_coverage("ff", ff())
> names(cov) <- paste0("<file>", names(cov))
> cov
Package Coverage: 66.67%
<file>: 66.67%
> ff = 
+   function(x = T) {
+     if(x) 
+       1
+     else
+       0
+   }
> cov = function_coverage("ff", ff())
> names(cov) <- paste0("<file>", names(cov))
> cov
Package Coverage: 100.00%
<file>: 100.00%

Right now if any of your tests fail you don't get any test coverage reported.

Maybe it would be better to wrap in a try block.

@wrathematics mentioned that covr fails with clang 3.4 and icc (due to gcov issues). This should be noted, and if possible a minimum version of gcc (which one TBD) should be added as well.

At present, if any of the tests produce an error, covr seems to simply silently ignore it, and report coverage without the test in question, or any subsequent ones. It seems like it would be preferable to at least indicate that an error occurred, rather than leaving the user confused as to why the coverage is so low.

I'm working on a package with compiled code. It builds fine, but fails when I run:

> Rscript -e "library(covr);package_coverage()"
Loading required package: methods
rfsrc tests : 

Unexpected version: *904.
Invalid .gcno File!
.
.
.
Unexpected version: *904.
Invalid .gcno File!
Error in names(res) <- file.path(src_path, names(res)) : 
  attempt to set an attribute on NULL
Calls: package_coverage -> run_gcov
Execution halted

This happens locally on OS X.

It fails on the TravisCI Linux server as well:

Error in seq.default(df[i, "first_line"], df[i, "last_line"]) : 
  'from' cannot be NA, NaN or infinite
Calls: coveralls -> to_coveralls -> per_line -> seq -> seq.default
Execution halted

From my package, it seems that package functions constructed by an internal meta-function is not traced. For example,

function_builder <- function(x) {
  function(a) {
    x + a
  }
}}

#' @export
useful_function1 <- function_builder(1)
#' @export
useful_function2 <- function_builder(2)

Even if useful_function1 and useful_function2 are both called in tests, the closure function(a) { x + a } is always reported not covered.

Example is here:
The code in red https://coveralls.io/builds/1833742/source?filename=.%2FR%2Fpipe.R#L183 is actually called test https://github.com/renkun-ken/pipeR/blob/master/tests/testthat/test2-Pipe.R#L64

Why is do_something() shown as covered in the following example:

if (FALSE)
  do_something();

Real-world example: For the linked file there is only one test for which I'm pretty sure that is.matrix(X) is TRUE and is.complex(X) is FALSE. Also, the function doesn't end prematurely -- however, lines 33, 36 and 38 are indicated as covered.

It is probably beneficial for a package developer to exclude lines from their coverage. I could see two interfaces for this

1. as a parameter to the given coverage function
   • named vector with filenames c(src/test.R = c(2, 3, 5, 6, 10:20))
2. as an annotation within the source file
   • # no_covr
   • # no_covr_start
   • # no_covr_end

They both have positive and negatives in my mind, perhaps the best solution is to use both...

Add a test package that has some zero lines of coverage to properly test the percent_coverage and zero_coverage functions

We get the following error after successful build with travis:

Error: No description at /home/travis/build/stnava/DESCRIPTION

The complete travis log file, associated with this error is:

* help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (ANTsR)

The command "R CMD INSTALL ANTsR" exited with 0.
$ Rscript -e 'library(covr);coveralls()'
Loading required package: methods
Error: No description at /home/travis/build/stnava/DESCRIPTION
Execution halted

Done. Your build exited with 0.

This is our .travis.yml here. I would appreciate any comments on this.

PS The coverage after this error remains "unknown"

Seems to be a handful of issues

• paths passed to subproccess need to use forward slashes
• does the gcc version with RTools support gcov?
• if it does support gcov are the output files in a different place than on linux/osx?

Hi,
I was wondering what's the meaning of repeating rows

The package is RevolutionAnalytics/quickcheck/pkg

pc = package_coverage("pkg/")
zero_coverage(pc)
1  R/quickcheck.R         39        39     0
2  R/quickcheck.R         40        41     0
3  R/quickcheck.R         41        41     0
4  R/quickcheck.R         41        41     0
5  R/quickcheck.R         41        41     0
6  R/quickcheck.R         49        50     0
7  R/quickcheck.R         53        54     0
8  R/quickcheck.R         83        83     0
9  R/quickcheck.R         83        83     0

thanks

If a package uses tools(*) that "dynamically" create functions based on eval(parse(text=...)) during build, then such functions will have a srcfile attribute that is "<text>". For example:

> eval(parse(text='foo <- function(...) cat("foo()\n")'))
> attr(attr(foo, "srcref"), "srcfile")
<text>

This will cause covr::coveralls() to throw an error:

> covr::coveralls()
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") : cannot open file '<text>': Invalid argument

6: file(con, "r")
5: FUN(c("<text>")[[1L]], ...)
4: lapply(filenames, readLines)
3: per_line(x)
2: to_coveralls(package_coverage(path, relative_path = TRUE, ...))
1: covr::coveralls()

One quick fix is to have covr::key() do gsub("<text>", ".text", ...) such that they file name, which base on this key, is a valid file name.

Footnotes:
(*) Such tools are setGenericS3() and setMethodS3() in R.methodsS3. Yep, that's my package; I could maybe update those functions to set the srcfile attribute such that it points to some kind of source file. Regardless, there may be other code out there that relies on eval(parse(text=...)) for creating functions.

Hi Jim,

I've tried out the latest version of covr on my ore package, and I see

> package_coverage()
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
  cannot open file '/Users/jon/Development/Git/ore/src/onig/enc/ascii.c.gcov': No such file or directory
> traceback()
7: file(con, "r")
6: readLines(file)
5: parse_gcov(paste0(file, ".gcov"))
4: FUN(c("/Users/jon/Development/Git/ore/src/main.c", 
<snip>
   "/Users/jon/Development/Git/ore/src/onig/regversion.c", "/Users/jon/Development/Git/ore/src/onig/st.c"
   )[[2L]], ...)
3: lapply(sources, run_gcov)
2: unlist(lapply(sources, run_gcov))
1: package_coverage()

I don't think this is related to #24, even though the error is superficially similar. I see .gcda and .gcno files in the hierarchy, but no .gcov files. This is OS X 10.10.1, with self-built R 3.1.2.

Apologies if I've missed something obvious.

After issue #33 was fixed, covr correctly worked with source files in subdirectories of src, but there seems to be another problem in the current version. After updating I see this when I try to check coverage in package ore:

> library(covr)
> package_coverage()  # from "ore" package root directory
Regular expression compilation : .....
Matching and related operations : .................
Compatibility with rex : .

Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
  cannot open file './src/iso8859_1.c': No such file or directory

The correct path is actually ./src/onig/enc/iso8859_1.c. If it's relevant, Makevars says

OBJECTS_ENC = onig/enc/ascii.o onig/enc/iso8859_1.o onig/enc/unicode.o onig/enc/utf8.o

OBJECTS_ONIG = onig/regcomp.o onig/regenc.o onig/regerror.o onig/regexec.o onig/regext.o onig/reggnu.o onig/regparse.o onig/regposerr.o onig/regposix.o onig/regsyntax.o onig/regtrav.o onig/regversion.o onig/st.o

OBJECTS = main.o $(OBJECTS_ONIG) $(OBJECTS_ENC)

PKG_CPPFLAGS = -Ionig

If '/.R/Makevars' exists, then set_makevars() rewrites the file and reset_makevars() undoes this rewrite. Although package_coverage() uses on.exit() to lower the risk for failing to undo the rewrite, there is still a risk that the R session is aborted corrupting the preexisting '/.R/Makevars' file. Also, it will mess with the file time stamp in either case.

SUGGESTION:
I would like to propose to have set_makevars() to rename any existing file to, say, '~/.R/Makevars.original.covr' and then have reset_makevars() to rename it back. This way the original file is still there in case of a core dump or similar, and its time stamp should be preserved.

See the comments by Coveralls in https://github.com/grake/grake/pull/4. The increase in coverage for c59c498 has been reported twice, while that for 3480d2c, the previous commit, hasn't been reported at all. I haven't investigated this further.