r-lib / covr Goto Github PK
View Code? Open in Web Editor NEWTest coverage reports for R
Home Page: https://covr.r-lib.org
License: Other
Test coverage reports for R
Home Page: https://covr.r-lib.org
License: Other
Is it possible to get coverage on .onLoad()
and similar functions? I've tried, but get zero coverage for all those.
EDIT: I eventually got .onUnload()
covered (https://coveralls.io/files/438500374), but I do get zero coverage on .onAttach()
.
This might very well be operator error, but ...
ff = function(x) identity(x)
function_coverage("ff", ff(1))
Results in
Error in aggregate.data.frame(mf[1L], mf[-1L], FUN = FUN, ...) :
no rows to aggregate
What am I missing? Thanks
Cool package! I noticed that if you run the package_coverage
function, it gives you a ton of output, that I'm guessing gets uploaded to the coveralls service to give you a badge. However, when I'm trying to improve test coverage, I would want to be able to just re-run package_coverage
as I add tests and see how the coverage changes.
Is there a way to generate a % coverage value from the package_coverage
output? I'd be willing to write a summarize
method if I knew how to interpret the values that are being returned.
I think covr
should have the option of not reporting test coverage if any test has been skipped by testthat::skip
.
I tend to use the skip function only when a test depends on some external web resource that might not be available. testthat:skip
allows intermittent failures to reach that external web resource without the red warning signs of a failed test. If the tests are being skipped on a particular build because the resource was unavailable, it does not seem reasonable for the coverage to drop instead of just remaining at what it's last value was.
Not actually sure how to go about this. Would have to dig into @hadley's testthat code to see if skip
is giving any other programmatic indication that a test has been skipped, other than the visual printing S
instead of .
where the test has would have been run in the testthat
summary.
Hi,
I am about to release quickcheck 3.1.0. It's a testing package and has covr as an optional dep. For lack of releases, I am pointing my users to commit b181831 because I tested on it, but I am sure you are not looking forward to bugs being reported referencing a random commit in the past. So if you planned to start doing releases, here is a vote to get that going sooner rather than later. Thanks
I've just faced the below strange error:
$ git clone https://github.com/Rapporter/pander.git
$ R
> library(covr)
> package_coverage('pander')
Error in seq.default(df[i, "first_line"], df[i, "last_line"]) :
'from' cannot be NA, NaN or infinite
> traceback()
9: stop("'from' cannot be NA, NaN or infinite")
8: seq.default(df[i, "first_line"], df[i, "last_line"])
7: seq(df[i, "first_line"], df[i, "last_line"])
6: match(x, table, nomatch = 0L)
5: seq(df[i, "first_line"], df[i, "last_line"]) %in% excl[[file]]
4: FUN(1:1553[[1552L]], ...)
3: vapply(seq_len(NROW(df)), function(i) {
file <- df[i, "filename"]
file %in% names(excl) && all(seq(df[i, "first_line"], df[i,
"last_line"]) %in% excl[[file]])
}, logical(1))
2: exclude(coverage, exclusions = exclusions, exclude_pattern = exclude_pattern,
exclude_start = exclude_start, exclude_end = exclude_end)
1: package_coverage()
I will investigate further later, just wanted to make a note of it for the time being.
Related session info:
> sessionInfo()
R version 3.1.2 (2014-10-31)
Platform: x86_64-unknown-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=hu_HU.utf8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=hu_HU.utf8
[7] LC_PAPER=hu_HU.utf8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=hu_HU.utf8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] covr_0.2.0.9000 pander_0.5.2
loaded via a namespace (and not attached):
[1] devtools_1.7.0 digest_0.6.8 lazyeval_0.1.10 magrittr_1.5
[5] Rcpp_0.11.4 rex_0.2.0 tools_3.1.2
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: /build/gcc-multilib/src/gcc-4.9-20141224/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-cloog-backend=isl --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-multilib --disable-werror --enable-checking=release
Thread model: posix
gcc version 4.9.2 20141224 (prerelease) (GCC)
I am not sure if the issue is in travis not running gcov
properly or an issue with the json to coveralls, but digest contains mostly c code and it is not being reported. https://coveralls.io/jobs/3980390
Actually there is a bug locally as well, the source coverage only works the second time you run it.
Seems it doesn't pick up on stuff in inst/tests
. All of my packages have tests there per an old comment from Hadley, but this convention might have been deprecated. Should I move all my tests for all of my packages to tests
off the package root?
It differs from testthat::test_check in that the tests are run from the installed directory always.
I need to ensure that the tests are calling the new functions, or modify the installed namespace functions at runtime.
Ala r-lib/lintr#37
Would be nice to report only the blank lines
Along with #51, it would be handy to report of coverage of vignettes separately as well.
It would be a more complete solution because it actually changes the function pointer and would hopefully fix some edge cases.
If multiple covr
sessions are used in parallel (which may not be unlikely on server farms), there is a risk that '~/.R/Makevars' is being rewritten while another R session is reading it leading to a potentially corrupt read.
One way to lower the risk for this is to have set_makevars()
only modify '~/.R/Makevars' iff the content actually changes, i.e. only call writeLines(..., lines)
if lines
have changed from the originally read one.
This solution does not fully cover the case when a pre-existing '~/.R/Makevars' is changed, because then reset_makevars()
in one session will undo what another session might have written. However, by only rewriting/undoing the file if something changes the risk for incorrect content is lowered.
The long-term solution is probably for R to provide a command-line option/system environment variable where one can specify a session-specific 'Makevars' file path.
Ideal features would be
Must haves
Like to haves
Does not need to track coverage over time as that is what coveralls.io is mainly for.
Does the package actually depend on having R 3.1.2? I was able to clone, change the Depends field in DESCRIPTION to R 3.1.0, and install on my system with R 3.1.1, and run it without difficulty.
Many packages don't have any tests, but often still have examples. Would be useful to see what code is covered by examples at least.
Love this tool, but when I have a single line function without braces, e.g.
f <- function(x)
x
the coverage tool neither detects it is covered nor uncovered by tests. (Or at least coveralls.io doesn't display it.)
Is there any chance you might be able to combine covr with an existing C++ code coverage tool in order to also give C++ code coverage stats?
function(x) {
x
}
These lines do not have a srcref
apart from their enclosing function. Distinguishing them from symbols in a regular call would be somewhat tricky as well, and what increase the number of trace calls with little gain (IMHO).
However this situation should be documented (probably in the README)
as of #47 these are being skipped, but it would be better track these functions as well.
To handle weird things like data.table
's testing situation.
Mostly an issue with testing, you probably wouldn't call it with any other argument in practice.
Ex. https://coveralls.io/files/410126873#L68
Need to tweak the logic to check if a given line is blank.
I am working on a tools for creating test cases for different VM implementations of R and I was thinking if it's possible to measure coverage of functions from default packages with covr
.
For example,
function_coverage("colMeans", colSums(UCBAdmissions), env=getNamespace("base"))
or
ctl <- c(4.17,5.58,5.18,6.11,4.50,4.61,5.17,4.53,5.33,5.14)
trt <- c(4.81,4.17,4.41,3.59,5.87,3.83,6.03,4.89,4.32,4.69)
group <- gl(2, 10, 20, labels = c("Ctl","Trt"))
weight <- c(ctl, trt)
lm.D9 <- lm(weight ~ group)
function_coverage("lm", lm(weight ~ group), env=getNamespace("stats"))
But now, it seems that trace_calls
does not insert any trace points into those functions. How can it be changed?
You may have mentioned this before but it is quite disturbing that when writing ifs without optional brackets coverage is overestimated. Is this a bug? Thanks
> ff
function(x = T) {
if(x) {
1}
else{
0}
}
> cov = function_coverage("ff", ff())
> names(cov) <- paste0("<file>", names(cov))
> cov
Package Coverage: 66.67%
<file>: 66.67%
> ff =
+ function(x = T) {
+ if(x)
+ 1
+ else
+ 0
+ }
> cov = function_coverage("ff", ff())
> names(cov) <- paste0("<file>", names(cov))
> cov
Package Coverage: 100.00%
<file>: 100.00%
Right now if any of your tests fail you don't get any test coverage reported.
Maybe it would be better to wrap in a try block.
@wrathematics mentioned that covr
fails with clang 3.4 and icc (due to gcov issues). This should be noted, and if possible a minimum version of gcc (which one TBD) should be added as well.
At present, if any of the tests produce an error, covr
seems to simply silently ignore it, and report coverage without the test in question, or any subsequent ones. It seems like it would be preferable to at least indicate that an error occurred, rather than leaving the user confused as to why the coverage is so low.
I'm working on a package with compiled code. It builds fine, but fails when I run:
> Rscript -e "library(covr);package_coverage()"
Loading required package: methods
rfsrc tests :
Unexpected version: *904.
Invalid .gcno File!
.
.
.
Unexpected version: *904.
Invalid .gcno File!
Error in names(res) <- file.path(src_path, names(res)) :
attempt to set an attribute on NULL
Calls: package_coverage -> run_gcov
Execution halted
This happens locally on OS X.
It fails on the TravisCI Linux server as well:
Error in seq.default(df[i, "first_line"], df[i, "last_line"]) :
'from' cannot be NA, NaN or infinite
Calls: coveralls -> to_coveralls -> per_line -> seq -> seq.default
Execution halted
From my package, it seems that package functions constructed by an internal meta-function is not traced. For example,
function_builder <- function(x) {
function(a) {
x + a
}
}}
#' @export
useful_function1 <- function_builder(1)
#' @export
useful_function2 <- function_builder(2)
Even if useful_function1
and useful_function2
are both called in tests, the closure function(a) { x + a }
is always reported not covered.
Example is here:
The code in red https://coveralls.io/builds/1833742/source?filename=.%2FR%2Fpipe.R#L183 is actually called test https://github.com/renkun-ken/pipeR/blob/master/tests/testthat/test2-Pipe.R#L64
Why is do_something()
shown as covered in the following example:
if (FALSE)
do_something();
Real-world example: For the linked file there is only one test for which I'm pretty sure that is.matrix(X)
is TRUE
and is.complex(X)
is FALSE
. Also, the function doesn't end prematurely -- however, lines 33, 36 and 38 are indicated as covered.
It is probably beneficial for a package developer to exclude lines from their coverage. I could see two interfaces for this
c(src/test.R = c(2, 3, 5, 6, 10:20))
# no_covr
# no_covr_start
# no_covr_end
They both have positive and negatives in my mind, perhaps the best solution is to use both...
Add a test package that has some zero lines of coverage to properly test the percent_coverage and zero_coverage functions
We get the following error after successful build with travis:
Error: No description at /home/travis/build/stnava/DESCRIPTION
The complete travis log file, associated with this error is:
* help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (ANTsR)
The command "R CMD INSTALL ANTsR" exited with 0.
$ Rscript -e 'library(covr);coveralls()'
Loading required package: methods
Error: No description at /home/travis/build/stnava/DESCRIPTION
Execution halted
Done. Your build exited with 0.
This is our .travis.yml
here. I would appreciate any comments on this.
PS The coverage after this error remains "unknown"
Seems to be a handful of issues
Hi,
I was wondering what's the meaning of repeating rows
The package is RevolutionAnalytics/quickcheck/pkg
pc = package_coverage("pkg/")
zero_coverage(pc)
1 R/quickcheck.R 39 39 0
2 R/quickcheck.R 40 41 0
3 R/quickcheck.R 41 41 0
4 R/quickcheck.R 41 41 0
5 R/quickcheck.R 41 41 0
6 R/quickcheck.R 49 50 0
7 R/quickcheck.R 53 54 0
8 R/quickcheck.R 83 83 0
9 R/quickcheck.R 83 83 0
thanks
If a package uses tools(*) that "dynamically" create functions based on eval(parse(text=...))
during build, then such functions will have a srcfile
attribute that is "<text>"
. For example:
> eval(parse(text='foo <- function(...) cat("foo()\n")'))
> attr(attr(foo, "srcref"), "srcfile")
<text>
This will cause covr::coveralls()
to throw an error:
> covr::coveralls()
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") : cannot open file '<text>': Invalid argument
6: file(con, "r")
5: FUN(c("<text>")[[1L]], ...)
4: lapply(filenames, readLines)
3: per_line(x)
2: to_coveralls(package_coverage(path, relative_path = TRUE, ...))
1: covr::coveralls()
One quick fix is to have covr::key()
do gsub("<text>", ".text", ...)
such that they file name, which base on this key, is a valid file name.
Footnotes:
(*) Such tools are setGenericS3()
and setMethodS3()
in R.methodsS3. Yep, that's my package; I could maybe update those functions to set the srcfile
attribute such that it points to some kind of source file. Regardless, there may be other code out there that relies on eval(parse(text=...))
for creating functions.
Hi Jim,
I've tried out the latest version of covr
on my ore
package, and I see
> package_coverage()
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
cannot open file '/Users/jon/Development/Git/ore/src/onig/enc/ascii.c.gcov': No such file or directory
> traceback()
7: file(con, "r")
6: readLines(file)
5: parse_gcov(paste0(file, ".gcov"))
4: FUN(c("/Users/jon/Development/Git/ore/src/main.c",
<snip>
"/Users/jon/Development/Git/ore/src/onig/regversion.c", "/Users/jon/Development/Git/ore/src/onig/st.c"
)[[2L]], ...)
3: lapply(sources, run_gcov)
2: unlist(lapply(sources, run_gcov))
1: package_coverage()
I don't think this is related to #24, even though the error is superficially similar. I see .gcda
and .gcno
files in the hierarchy, but no .gcov
files. This is OS X 10.10.1, with self-built R 3.1.2.
Apologies if I've missed something obvious.
After issue #33 was fixed, covr
correctly worked with source files in subdirectories of src
, but there seems to be another problem in the current version. After updating I see this when I try to check coverage in package ore
:
> library(covr)
> package_coverage() # from "ore" package root directory
Regular expression compilation : .....
Matching and related operations : .................
Compatibility with rex : .
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
cannot open file './src/iso8859_1.c': No such file or directory
The correct path is actually ./src/onig/enc/iso8859_1.c
. If it's relevant, Makevars
says
OBJECTS_ENC = onig/enc/ascii.o onig/enc/iso8859_1.o onig/enc/unicode.o onig/enc/utf8.o
OBJECTS_ONIG = onig/regcomp.o onig/regenc.o onig/regerror.o onig/regexec.o onig/regext.o onig/reggnu.o onig/regparse.o onig/regposerr.o onig/regposix.o onig/regsyntax.o onig/regtrav.o onig/regversion.o onig/st.o
OBJECTS = main.o $(OBJECTS_ONIG) $(OBJECTS_ENC)
PKG_CPPFLAGS = -Ionig
If '/.R/Makevars' exists, then /.R/Makevars' file. Also, it will mess with the file time stamp in either case.set_makevars()
rewrites the file and reset_makevars()
undoes this rewrite. Although package_coverage()
uses on.exit()
to lower the risk for failing to undo the rewrite, there is still a risk that the R session is aborted corrupting the preexisting '
SUGGESTION:
I would like to propose to have set_makevars()
to rename any existing file to, say, '~/.R/Makevars.original.covr' and then have reset_makevars()
to rename it back. This way the original file is still there in case of a core dump or similar, and its time stamp should be preserved.
See the comments by Coveralls in https://github.com/grake/grake/pull/4. The increase in coverage for c59c498 has been reported twice, while that for 3480d2c, the previous commit, hasn't been reported at all. I haven't investigated this further.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.