grailbio / rules_r Goto Github PK
View Code? Open in Web Editor NEWR rules for Bazel
License: Apache License 2.0
R rules for Bazel
License: Apache License 2.0
Reproducible mode has been the default at GRAIL for ~8 months and has been working without any issues. We should write more unit tests for it and make it the default and only mode.
This repository can then be overridden in the WORKSPACE file, and all built R packages can use these files.
Let's test that we can support ARM processors, specifically on the new AWS instance types. We may have to wait for R 4.1.0 for full support.
For macOS, CRAN recommendation is to use Rosetta for R versions less than 4.1.0.
here is part of my BUILD file
PKG_NAME = "flowWorkspace"
PKG_CHECK_DEPS = [
"@R_knitr//knitr:knitr",
"@R_ggcyto//:ggcyto",
]
r_pkg(
name = PKG_NAME,
srcs = glob(
["**"],
exclude = [
"BUILD",
"src/Makevars",
],
),
cc_deps = [
":flowWorkspace_hdrs",
],
config_override = ":empty",
makevars = ":flowworkspace_makevars",
pkg_name = "flowWorkspace",
deps = [
"@R_Biobase//Biobase",
"@R_DelayedArray//DelayedArray",
"@R_RBGL//RBGL",
"@R_Rcpp//Rcpp",
"@R_RcppParallel//RcppParallel",
],
)
r_library(
name="library",
pkgs = [PKG_NAME]
)
r_pkg_test(
name = "check",
timeout = "short",
pkg = PKG_NAME,
check_args = ["--no-codoc", "--no-manual", "--no-build-vignettes"],
suggested_deps = PKG_CHECK_DEPS,
)
I have encountered a test error during R CMD check (invoked by bazel run packages/flowWorkspace:check
)
I'd like to reproduce and troubleshoot the test error interactively at regular R console with devtools:test()
I know all the r packages and its deps can be copied to system R library path by running the r_library
target (i.e. bazel run packages/flowWorkspace:library), but it will miss the packages defined in PKG_CHECK_DEPS.
I can't paste these to deps list of r_pkg rule since that will fail the bazel build due to some circular dependencies from suggests
deps of r package.
I wonder if there is a convenient way to copy the packages listed in PKG_CHECK_DEPS (i.e. bazel-out/k8-fastbuild/bin/packages/flowWorkspace/check.runfiles/
I guess) to system R path instead of manually doing it.
In build.sh, we use 200 as the constant for the file descriptor that we lock.
This is most likely causing unwanted waits for the lock to be released when multiple packages are being built simultaneously.
This is just a hunch though that needs to be validated.
We rely on system provided R and are not controlling the version of R being used, or the packages installed at .Library
search path.
A toolchain to have better control will make the build more hermetic.
Maybe not the workspace macro, but the BUILD file generation can be done entirely in Go.
The test target should get this information from the package provider, and make the Makevars, headers and library archives available as runfiles. The paths, etc. should also change to represent short paths.
I have a WORKSPACE which works on my dev machine but breaks on my team's jenkins box.
My dev laptop is Fedora, and the jenkins box is ubuntu. I don't think that should make a difference but it's the one difference in the environment I can think of.
I had to apt-get install r-base
because prior to this there was no R on the jenkins box.
Any help appreciated.
WORKSPACE snippet:
http_archive(
name = "com_grail_rules_r",
strip_prefix = "rules_r-master",
urls = ["https://github.com/grailbio/rules_r/archive/master.tar.gz"],
)
load("@com_grail_rules_r//R:dependencies.bzl", "r_register_toolchains", "r_rules_dependencies")
r_rules_dependencies()
r_register_toolchains()
load("@com_grail_rules_r//R:repositories.bzl", "r_repository_list", "r_repository")
r_repository(
name = "R_data_table",
sha256 = "f5b2b7d44ef5d8cb3505b4e6b4c4539e7a2132dffc5516da6f717fa51ebe9d3b",
strip_prefix = "data.table-c0052964694a4c618ab182aa474f924d40576d94",
urls = [
"https://github.com/Rdatatable/data.table/archive/c0052964694a4c618ab182aa474f924d40576d94.tar.gz",
],
)
r_repository_list(
name = "r_repositories_bzl",
package_list = "//sample_r:packages.csv",
remote_repos = {
"CRAB": "https://cloud.r-project.org",
},
)
load("@r_repositories_bzl//:r_repositories.bzl", "r_repositories")
r_repositories()
load("@com_grail_rules_r//R:dependencies.bzl", "r_coverage_dependencies")
r_coverage_dependencies()
bash log:
$ bazel build //sample_r:sample
INFO: Writing tracer profile to '/home/jenkinsadmin/.cache/bazel/_bazel_jenkinsadmin/b33a4b3c4e6f0c61d58e78df9100fe07/command.profile.gz'
INFO: SHA256 (https://github.com/grailbio/rules_r/archive/master.tar.gz) = bacdb1d81e81db7a95d9e364e9eea73c9cfea43f0901c115bc7b8d3f5eec1ae2
DEBUG: Rule 'com_grail_rules_r' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "bacdb1d81e81db7a95d9e364e9eea73c9cfea43f0901c115bc7b8d3f5eec1ae2"
DEBUG: Call stack for the definition of repository 'com_grail_rules_r' which is a http_archive (rule definition at /home/jenkinsadmin/.cache/bazel/_bazel_jenkinsadmin/b33a4b3c4e6f0c61d58e78df9100fe07/external/bazel_tools/tools/build_defs/repo/http.bzl:292:16):
- /home/jenkinsadmin/repo/mob/WORKSPACE:391:1
INFO: Call stack for the definition of repository 'r_repositories_bzl' which is a r_repository_list (rule definition at /home/jenkinsadmin/.cache/bazel/_bazel_jenkinsadmin/b33a4b3c4e6f0c61d58e78df9100fe07/external/com_grail_rules_r/R/repositories.bzl:139:21):
- /home/jenkinsadmin/repo/mob/WORKSPACE:414:1
ERROR: An error occurred during the fetch of repository 'r_repositories_bzl':
Failed to generate bzl file:
Error in available.packages(repos = repos, type = type) :
unused argument (repos = repos)
Calls: generateWorkspaceMacro ... mergeWithRemote -> as.data.frame -> available.packages
Execution halted
ERROR: no such package '@r_repositories_bzl//': Failed to generate bzl file:
Error in available.packages(repos = repos, type = type) :
unused argument (repos = repos)
Calls: generateWorkspaceMacro ... mergeWithRemote -> as.data.frame -> available.packages
Execution halted
ERROR: no such package '@r_repositories_bzl//': Failed to generate bzl file:
Error in available.packages(repos = repos, type = type) :
unused argument (repos = repos)
Calls: generateWorkspaceMacro ... mergeWithRemote -> as.data.frame -> available.packages
Execution halted
INFO: Elapsed time: 1.323s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
We should test that our cache state is invalidated whenever the system R is updated, or some other related toolchain like CC, etc. is updated.
They are not specified as runfiles for r_unit_test
, and the PATH
variable is not modified. r_pkg_test
seems to have them set up correctly.
See bazelbuild/bazel#4570 for progress on the API.
It is not quite ready yet, but this issue will track work on using the API to solve our long standing problem of obtaining PIC.
Noticed that dependencies aren't being downloaded from the bazel cache, but rather are redownloaded every time.
With a default value of '.', this attribute can specify the path in which the packages are installed inside the tarball. Running the rule will also copy the packages from this path into the library directory.
We currently don't test that our rules result in a reproducible build on execution on the same machine. These tests have been performed manually before.
Question about having an r_pkg depend on a cc_library through cc_deps.
cc_library has a includes
argument that tells downstream dependencies how to search for and use its header files (according to my understanding).
But it seems that this is not passed on R CMD BUILD via r_pkg?
For example I am only seeing the includes related to pkg_deps not cc_deps.
The consequence for my build is that I can't compile against a pure C++ library dependency.
Normally for pure R, I would pass stuff in via configure_args I think, but I don't know the sandbox path ahead of time.
Here's my c++ dependency:compiles fine
cc_library(
name = "cytolib",
srcs = glob([
"src/*.cpp"
]),
hdrs = glob([
"include/cytolib/*.hpp",
"include/cytolib/*.h",
]),
copts = [
"-I.",
"-std=c++17",
"-fopenmp",
],
linkopts = ["-lhdf5_serial"],
include_prefix="cytolib",
includes=["."],
strip_include_prefix="include/cytolib",
deps = [
"@boost_hdrs//:boost",
"@hdf5_hdrs//:hdf5",
"@hdf5_lib//:libhdf5",
],
)
and here's my R package that depends on it.
r_pkg(
name = "flowCore",
srcs = glob(["**"], exclude=["BUILD"]),
pkg_name = "flowCore",
deps = [
'@R_Biobase//Biobase:Biobase',
'@R_BiocGenerics//BiocGenerics:BiocGenerics',
'@R_Rcpp//Rcpp:Rcpp',
'@R_matrixStats//matrixStats:matrixStats',
'@R_RcppArmadillo//RcppArmadillo:RcppArmadillo',
'@R_S4Vectors//S4Vectors:S4Vectors',
],
cc_deps = ["@cytolib_repo//packages/cytolib:cytolib"],
)
The compilation command I see is:
g++ -std=gnu++11 -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/packages/cytolib/cytolib' -I"/usr/local/lib/R/include" -DNDEBUG -DROUT -pthread -I/usr/local/include -I/usr/local/cytolib/include -I/usr/local/include/tiledb -DBOOST_NO_AUTO_PTR -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/external/R_Rcpp/Rcpp/lib/Rcpp/include' -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/external/R_RcppArmadillo/RcppArmadillo/lib/RcppArmadillo/include' -I/usr/local/include -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fdebug-prefix-map="/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/=" -fpic -g -O2 -c RcppExports.cpp -o RcppExports.o
with the error:
pairVectorRcppWrap.h:10:10: fatal error: cytolib/compensation.hpp: No such file or directory
10 | #include <cytolib/compensation.hpp>
So it's not finding the header files for cytolib, since they don't seem to be passed on.
And what I would like / expect to see is:
g++ -std=gnu++11 -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/packages/cytolib/_virtual_includes/cytolib' -I"/usr/local/lib/R/include" -DNDEBUG -DROUT -pthread -I/usr/local/include -I/usr/local/cytolib/include -I/usr/local/include/tiledb -DBOOST_NO_AUTO_PTR -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/external/R_Rcpp/Rcpp/lib/Rcpp/include' -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/external/R_RcppArmadillo/RcppArmadillo/lib/RcppArmadillo/include' -I/usr/local/include -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fdebug-prefix-map="/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/=" -fpic -g -O2 -c RcppExports.cpp -o RcppExports.o
With the difference being in the first -I
flag:
-I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/packages/cytolib/_virtual_includes/cytolib'
Would love some insights if anyone has some.
To reproduce:
cd tests
bazel --output_user_root=/tmp/a\\b sync
...
ERROR: An error occurred during the fetch of repository 'r_repositories_bzl':
Failed to generate bzl file:
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
Calls: source -> file
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '/private/tmp//fbcd335144c4da9c79038f9ce88e9f6d/external/com_grail_rules_r/scripts/razel.R': No such file or directory
Execution halted
We rely on a package's configure script to locate the system dependency. We should be able to specify some system requirements through Bazel and provide configure flags to use the Bazel built requirement.
This rule can be like an http_archive, and generate a default BUILD file for the package.
Modalities to obtain the R package can be, either exclusive or in order of preference,
r_pkg can take a list of strings as a "roclets" attribute to run roxygen on package source code before building it. This will automate a manual step that developers do frequently before running a build or test.
r_pkg_test builds the source archive of the package independently of the r_pkg rule.
This means all the package build configuration and dependencies have to be specified twice.
Have the ability for users to specify a metadata dictionary with values allowing substitutions from the volatile and stable status files. Have a boolean attribute stamp
to allow users to specify if stable status file should be a dependency.
r_pkg_test
rules do not seem to work correctly when the target r_pkg
has a different name
than pkg_name
.
Given the following BUILD
file for a package:
load("@com_grail_rules_r//R:defs.bzl", "r_pkg", "r_pkg_test")
r_pkg(
name = "dummy",
pkg_name = "<pkgname>",
srcs = glob(
["**"],
exclude = [
"BUILD",
],
)
)
r_pkg_test(
name = "check",
pkg = ":dummy",
)
I see the following when trying to run the check
target (lightly edited for privacy):
bazel run //<pkgname>:check
Loading:
Loading: 0 packages loaded
Analyzing: target //<pkgname>:check (1 packages loaded, 0 targets configured)
INFO: Analyzed target //<pkgname>:check (27 packages loaded, 375 targets configured).
INFO: Found 1 target...
bazel: Entering directory `<snip>/execroot/<pkgname>/'
[0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt
bazel: Leaving directory `<snip>/execroot/<pkgname>/'
Target //<pkgname>:check up-to-date:
bazel-bin/check
INFO: Elapsed time: 1.024s, Critical Path: 0.66s
INFO: 10 processes: 7 internal, 3 linux-sandbox.
INFO: Build completed successfully, 10 total actions
INFO: Running command line: external/bazel_tools/tools/test/test-setup.sh ./check
INFO: Build completed successfully, 10 total actions
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:check
-----------------------------------------------------------------------------
* using log directory ‘<snip>/execroot/<pkgname>/bazel-out/k8-fastbuild/bin/check.runfiles/<pkgname>/dummy.Rcheck’
* using R version 3.6.3 (2020-02-29)
* using platform: x86_64-pc-linux-gnu (64-bit)
* using session charset: UTF-8
* checking package directory ... ERROR
package directory `<snip>/execroot/<pkgname>/bazel-out/k8-fastbuild/bin/check.runfiles/<pkgname>/dummy.Rcheck/00_pkg_src/dummy’ does not exist
* DONE
Status: 1 ERROR
See
‘<snip>/execroot/<pkgname>/bazel-out/k8-fastbuild/bin/check.runfiles/<pkgname>/dummy.Rcheck/00check.log’
for details.
This is almost certainly because R CMD check
expects the tarball name to match the package name, but when using pkg_name
this will not be the case.
I'm building custom images containing R libs (among other things) using GoogleContainerTools/distroless/package_manager. I have worked though collecting and adding R deps to packages.csv so all of my R world dependencies are fine.
But those R libraries have system library dependencies. When you load the R libraries in R code, they look for the system libraries they use, and throw at the first one they can't find.
I am not a native R person, so please forgive me if this is a stupid question: is there any way to find the native deps of R libraries? Iterating over this is really time consuming at scale.
Any help appreciated.
We have been using a covr dependency with some of our own modifications. Would be nice to get these changes upstreamed and then depend on a CRAN release directly.
The PR is r-lib/covr#468.
It would be possible to get code coverage for r_pkg as follows:
It is possible to wrap an r_binary invocation in a similar way.
The LCOV reports can then be consumed in many ways, for instance using the genhtml command-line utility.
Code coverage can be transitive if --instrumentation_filter is properly set, and covr integrates nicely with native code, either generated by rules_r or handled through cc_deps.
It seems a package will only be properly created from files that are under an 'R' subdirectory.
This seems to go against the notion of Bazel; is there no way to enable a more flexible source structure?
The implementation of these rules expect a minimum of R 3.3. We should check for it on bootstrap and message the user accordingly.
Is there a deployment story native to bazel? Or is the out of scope for this?
We assume a brew install prefix of /usr/local in Makevars.darwin for finding gfortran.
Provide a default Makevars file target that is dynamically generated and uses the right brew prefix for the user.
I wanted to use my R code to call a python code that is also built with bazel. Currently, I have to build them as two separate targets (a py_binary and a r_binary). But is there a way that I can put the py_binary as a deps
for the r_binary, so that when I build the r_binary, it will automatically build the py_binary?
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Data-in-packages
A workaround right now is to use post_install files, but this can be determined automatically and the files added to the output files set.
The packages built have stamped information about the built timestamp, the source directory and the library directory for the installation. This is especially bothersome with docker images as different layers are created with each build.
The build timestamp can be fixed to an empty string with the --built_timestamp
flag to R CMD INSTALL
. For the rest, we need to build and install in a constant directory, which means fixing a /tmp path for a package, and acquiring a lock on that path so that builds in other workspaces do not interfere with this build.
Technically, r_pkg is supposed to collect *.so's and bundle them into the package, which is what it does at the preliminary build step. However, this doesn't really work because R CMD build
throws out those *.so's, not recognizing them as a valid part of the package, so by the time build.sh
needs to run R CMD install
they're no longer there.
It would seem that, instead, they should be copied after build
archive is expanded. I'm not sure if install
won't throw them out, but cursory search in install.R
seems to suggest that it shouldn't.
Hello,
It's looks like r_pkg
rule is failing when WORKSPACE file is in the root of the package. Simple R package repo with such issue: https://github.com/knightdave/simpleR .
rules_r version: 0.5.9.tar.gz
❯ bazel --version
bazel 3.5.0
❯ bazel build --verbose_failures //...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 164d6861-16b7-4791-92fc-7e51dbb3f51b
INFO: Analyzed target //:simpleR (18 packages loaded, 115 targets configured).
INFO: Found 1 target...
ERROR: /hidden/simpleR/BUILD.bazel:18:6: Building R package simpleR failed (Exit 1): build.sh failed: error executing command
(cd /hidden/.cache/bazel/_bazel_rycerzd/ff5bd6ed62b50ccee0dfeb7260c1c601/sandbox/linux-sandbox/4/execroot/__main__ && \
exec env - \
BAZEL_R_DEBUG=false \
BAZEL_R_VERBOSE=false \
BUILD_ARGS=''\''--no-build-vignettes'\'' '\''--no-manual'\''' \
BUILD_TOOLS_EXPORT_CMD='export PATH' \
CONFIG_OVERRIDE='' \
C_CPP_FLAGS='' \
C_LIBS_FLAGS='' \
C_SO_FILES='' \
EXPORT_ENV_VARS_CMD='' \
FLOCK_PATH=bazel-out/host/bin/external/com_grail_rules_r/R/scripts/flock \
INSTALL_ARGS='' \
INSTRUMENTED=false \
INSTRUMENT_SCRIPT=external/com_grail_rules_r/R/scripts/instrument.R \
INST_FILES_MAP='' \
METADATA_MAP='' \
PKG_BIN_ARCHIVE=bazel-out/k8-fastbuild/bin/simpleR.bin.tar.gz \
PKG_LIB_PATH=bazel-out/k8-fastbuild/bin/lib \
PKG_NAME=simpleR \
PKG_SRC_ARCHIVE=bazel-out/k8-fastbuild/bin/simpleR.tar.gz \
PKG_SRC_DIR='' \
R='/usr/bin/R --slave --no-restore --no-init-file --no-save --no-site-file --no-environ' \
REQUIRED_VERSION='' \
ROCLETS='' \
RSCRIPT='/usr/bin/Rscript --no-init-file --no-save --no-site-file --no-environ' \
R_LIBS_DEPS='' \
R_LIBS_ROCLETS='' \
R_MAKEVARS_SITE='' \
R_MAKEVARS_USER='' \
STATUS_FILES=bazel-out/volatile-status.txt \
external/com_grail_rules_r/R/scripts/build.sh)
Execution platform: @local_config_platform//:host
Use --sandbox_debug to see verbose messages from the sandbox build.sh failed: error executing command
(cd /hidden/.cache/bazel/_bazel_rycerzd/ff5bd6ed62b50ccee0dfeb7260c1c601/sandbox/linux-sandbox/4/execroot/__main__ && \
exec env - \
BAZEL_R_DEBUG=false \
BAZEL_R_VERBOSE=false \
BUILD_ARGS=''\''--no-build-vignettes'\'' '\''--no-manual'\''' \
BUILD_TOOLS_EXPORT_CMD='export PATH' \
CONFIG_OVERRIDE='' \
C_CPP_FLAGS='' \
C_LIBS_FLAGS='' \
C_SO_FILES='' \
EXPORT_ENV_VARS_CMD='' \
FLOCK_PATH=bazel-out/host/bin/external/com_grail_rules_r/R/scripts/flock \
INSTALL_ARGS='' \
INSTRUMENTED=false \
INSTRUMENT_SCRIPT=external/com_grail_rules_r/R/scripts/instrument.R \
INST_FILES_MAP='' \
METADATA_MAP='' \
PKG_BIN_ARCHIVE=bazel-out/k8-fastbuild/bin/simpleR.bin.tar.gz \
PKG_LIB_PATH=bazel-out/k8-fastbuild/bin/lib \
PKG_NAME=simpleR \
PKG_SRC_ARCHIVE=bazel-out/k8-fastbuild/bin/simpleR.tar.gz \
PKG_SRC_DIR='' \
R='/usr/bin/R --slave --no-restore --no-init-file --no-save --no-site-file --no-environ' \
REQUIRED_VERSION='' \
ROCLETS='' \
RSCRIPT='/usr/bin/Rscript --no-init-file --no-save --no-site-file --no-environ' \
R_LIBS_DEPS='' \
R_LIBS_ROCLETS='' \
R_MAKEVARS_SITE='' \
R_MAKEVARS_USER='' \
STATUS_FILES=bazel-out/volatile-status.txt \
external/com_grail_rules_r/R/scripts/build.sh)
Execution platform: @local_config_platform//:host
Use --sandbox_debug to see verbose messages from the sandbox
cp: cannot stat '/DESCRIPTION': No such file or directory
Target //:simpleR failed to build
INFO: Elapsed time: 8.045s, Critical Path: 0.27s
INFO: 3 processes: 3 remote cache hit.
FAILED: Build did NOT complete successfully
The same package is working when I'll put it to some subfolder.
Just got bit trying to set stamp
& metadata
on an r_package
, but all it passes through is the srcs & deps.
I'm not sure if there's any particular reason not to just let the r_package
take **kwargs
& directly pass it along to its underlying r_pkg
?
Though honestly, looking at it, I probably would've written r_package
purely in terms of kwargs. Then it's a 100% transparent drop-in replacement, plus you don't need to worry at all about updating the convenience wrapper when the real thing adds new options.
E.g.
def r_package(name, **kwargs):
r_pkg(
name = name,
**kwargs,
)
r_library(
name = "library",
pkgs = [":%s" % name],
tags = ["manual"],
)
I think it'd be cool if r_unit_test
could define data dependencies so that Bazel could download and symlink testdata for unit tests in an atomic, pipelined, and checksummed fashion. Currently the only deps for r_unit_test
can be other packages. While that might be appropriate for r_pkg_test
, seems like for r_unit_test
that's too restrictive.
In particular, this could be useful during development if the authors are not planning to release the package, or if package will be modified before release to be able to do a package level test instead. Indeed, the package could dynamically choose either Bazel managed or pre-installed data source depending on how it's being tested.
Rule analogous to sh_binary but for the Rscript interpreter. Will have attributes srcs, data and r_pkg_deps.
Running r_library targets should clean up the destination packages before copying new files there. Otherwise, previously installed files from inst/, etc. will continue to exist in the new install.
Currently, the libraries to link are aggregated as such:
c_libs_flags = l.path for d in cc_deps for l in d.cc.libs
As a consequence, in the following case, the linker complains that some symbols in B are not found, because the dependencies are not given to the linker in topological order:
cc_library A transitively depends on cc_library B
cc_deps = [":B", ":A"]
Command: g++ ... -lB -lA
The solution is to aggregate the d.cc.libs using the depset mechanism instead of simply iterating through the list (I suppose for d.cc.libs the order is "topological").
See RcppCore/RcppArmadillo#331 for discussion.
Note that covr does use the +=
assignment operator in the personal Makevars file to set the flags.
Currently, r_pkg
leans pretty heavily on R to build *.cpp code, which is pretty slow.
What if, instead, C++ code was built with Bazel (which is fast), and then passed to r_pkg
through cc_dep
, already prebuilt? I've hacked up something like this and it seems to work fine, and it's much faster. I had to use an empty *.cpp file to get r_pkg
to link the *.so, and I had to put libRcpp.so into a location where it could easily be found, but other than that this required no changes to the C++ code. The resulting package seems to work OK under r_unit_test
. Not sure what r_pkg_test
would think of it, but it'll probably be fine as well, since it works OK with other cc_deps
and this is no different.
So basically the suggestion here is to modify the rule such that it compiles all C++ source with cc_library
(and lines up the *.so's and headers for that to work, so symlinking hack is not necessary), and then feeds the result into build.sh
much like it would feed cc_deps
.
Having a symlinked, read-only reference to the source files can be problematic for some configure
scripts, e.g., for the XML or Rgraphviz packages (they overwrite source files, they create temporary files within the source tree, ...). One quick fix is to make R CMD install work, for those packages, on a full, writable recursive copy of the source files.
I propose a new copy_srcs
bool attribute to r_pkg
to enable that. This would reuse the machinery of REPRODUCIBLE_BUILD and copy sources with cp -LpR ...
instead of cp -a
(-L dereferences symlinks).
I am building an R package via r_pkg
that depends on a cc_library
A. Library A depends on two system libraries B and C that I've imported as external repositories via new_local_repository
and then created a rule for these using cc_library.
Library A is built fine but my R package is having problems linking against A, and what's more it seems to try and include B and C in the linking statement.
The error I see depends on whether I'm running sandboxed or not. If sandboxed, I see:
cp: cannot create regular file 'external/R_flowCore/flowCore/src/libhdf5.so': Read-only file system
cp: cannot create regular file 'external/R_flowCore/flowCore/src/libarmadillo.so': Read-only file system
those are the system dependencies B and C.
If I don't run sandboxed I see:
g++: error: libhdf5.so: No such file or directory
g++: error: libarmadillo.so: No such file or directory
I don't fullly understand what directories bazel is doing its work in, so I'm having trouble understanding why I see these particular errors, especially the latter non-sandboxed one. Poking around I do see an R package source directory where these .so files have been copied into, but it's doesn't contain the compiled .o files for the rest of the cpp code, so presumably the build directory is elsewhere.
I've looked at rules_r/R/scripts/build.sh
and there's a lot of complexity there that I'm having difficulty unravelling.
Finally,thinking this may be related to the bundles configure script, but I see this same behavior irrespective of whether I use the package's bundled configure
file or if I remove the configure
script.
I'd like to understand:
When the list of dependencies gets really long, R CMD INSTALL no longer sees the installed dependencies when building a package.
I can confirm this by shortening R_LIBS_DEPS, or by installing the packages that aren't found into the R/site-library, then R CMD INSTALL sees them properly.
Two questions:
We currently use new_http_archive for external R packages, and explicitly list CRAN and BioC repo URLs for the package. A new repository rule for R remote repositories should help.
Creating a tar copies all dependencies in the tar. This can be an expensive operation if done as part of an automated build of all library targets.
The current recommendation is to use the tag manual to skip building library targets.
Using the fact that in sandboxed builds, the runfiles are symlinks, we can let the runfiles be the package files, and on running the library target, copy the library directories into the destination, following symlinks.
To separately package the system dependency tools that R packages expect. Useful for R library docker images.
The package layout and the symlink structure currently does not find package deps provided through the layers attribute. See image_test.sh.
I think it was not available in the past, but now it is possible to run travis on OSX for Open Source projects.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.