Setting RENV_PATHS_ROOT seems to work

Sys.setenv(RENV_PATHS_ROOT="~/.renv")
renv:::renv_paths_cache()
#> [1] "~/.renv/R-3.5/cache/v2"

but only trying to change the cache does not

Sys.setenv(RENV_PATHS_CACHE="~/.renv/mycache")
renv:::renv_paths_cache()
#> [1] "C:\\Users\\chris\\AppData\\Local/renv/R-3.5/cache/v2"

I think there is something wrong here read here

Sys.setenv(`RENV_PATHS_CACHE/V2`="~/.renv/mycache")
renv:::renv_paths_cache()
#> [1] "~/.renv/mycache"

I am not sure this is expected. Am I right ?

If you want I can PR something and do other check like these.

E.g.

> renv::install("reticulate")
Error: can't retrieve package 'reticulate': 'unknown' is an unrecognized source 

This occurs because I had installed reticulate from source earlier. Note that renv warns about this case when snapshot() is called:

> renv::snapshot()
The following package(s) were installed from an unknown source:

	reticulate

Consider re-installing these packages from a known source (e.g. CRAN).
Do you want to proceed? [Y/n]: 

But it would be nice to encode some explicit way of handling unknown sources, e.g.

• Hard error (as we currently do)
• Attempt to just install the latest version from CRAN
• Something else?

When would I use hydrate? When would I use restore? What is the difference between the two?

• .Rprofile
• .gitignore
• .Rbuildignore

1: Multiple hashed copies of httr [1.4.0] detected.
The version with hash '3fcffbd765a89d1eb9bedf19bc0eaa01' will be used.
2: Multiple hashed copies of mime [0.6] detected.
The version with hash '93beba1d766060a3f12a27f54ae67664' will be used.
3: Multiple hashed copies of openssl [1.2.2] detected.
The version with hash '2d036d89f158a04cc026b7520e5bd868' will be used.
4: Multiple hashed copies of askpass [1.1] detected.
The version with hash '51493f8310516c4c29a440df5badbe20' will be used.
5: Multiple hashed copies of memoise [1.1.0] detected.
The version with hash '665a0e91043f65582db2bdf8a02f65ba' will be used.
6: Multiple hashed copies of pkgload [1.0.2] detected.

What am I supposed to do when I see these warnings?

This is useful for each reference page to link to the R file, and have the github button inside the pkgdown website

desc::desc_add_urls("https://github.com/rstudio/renv/")

A small but useful improvement

For a new project, if a package is not already cached, it will be installed in the cache but not linked to the project library. It will be copied from temp library where it was installed.

This is because, when a package is not in cache renv_install_package_local is used. I understand that

• Package is downloaded in cache (binary folder on windows)
• Package is installed in a temporary library (templib <- tempfile("renv-templib-"))
• Cache is synchronize with renv_cache_synchronize: Temp installation is copy into cache.
  Not move and linked because library == renv_paths_library() is FALSE (one is temp lib the other is project lib)
• Then the package is moved from temp lib to project lib
  

  renv/R/install.R

  Lines 69 to 73 in 4c29ee4

  	# migrate packages into true library
  	sources <- list.files(templib, full.names = TRUE)
  	targets <- file.path(library, basename(sources))
  	names(targets) <- sources
  	enumerate(targets, renv_file_move, overwrite = TRUE)

At the end here, the package is not a linked folder to cache library but installed in the project.
I would have expected the package to be linked from the cache to the project library.

Seems like what this does

Perhaps this could be outsourced to remotes or the pak package, but I think it'd be really nice if the restore was atomic; either the environment/snapshot gets restored completely or nothing changes.

One of packrat's most frustrating attributes was semi-restored environments

Surface an option in init to turn on Python tracking, and add a function that turns on Python support in an existing environment.

Today this is done through renv::settings$python() which is an awkward interface that can be tricky to discover

In multi-tenant servers (i.e. RStudio pro products), it might be beneficial to have "build awareness" in relation to #3 (when using a shared cache).

I.e. 5 users request to build the same package version at the same time. It would be nice to see if it's possible for renv to have hooks or some type of awareness into that build pipeline so that redundancy can be reduced.

Honestly, part of this may be just awareness / extensibility and potential dependence on an arbitrary package installation client (which might provide this build awareness). I just wanted to document the idea in case it affects the development process at all.

For instance - on Connect (packrat), if 5 users attempt to install the same package at the same time, they will all build from source concurrently. Each one, as it completes, will update the cache with its completed version. This (1) is redundant and (2) burns up the CPU. Binary packages could resolve much of the issue, but in any case where binaries are not available, some additional client awareness would be useful.

To highlight init(), snapshot() and restore()

When restoring package from unknown source with renv.verbose=TRUE there is error with following message:
Message:
Error in renv_verbose() && renv_pretty_print_records(unknown, "The following package(s) were installed from an unknown source:", : invalid 'y' type in 'x && y'.

Bug was probably introduced with commit:
327eabf

In function renv_restore_preflight_unknown_source:
renv_verbose() && renv_pretty_print(...)
probably should be:
if (renv_verbose()) { renv_pretty_print(...) }

There might be similar issue in function renv_restore_preflight_permissions.

What information about the selected version of Python do we need to capture? Some options:

• The path to the Python executable,
• The version of the Python interpreter used,
• The name, or path, to a virtual environment / Conda environment.

Where do we want to capture this? Within the project options? Or as part of the lockfile? Or both?

The primary benefit of capturing these things in the lockfile is that an renv::restore() using just a project lockfile could then attempt to find the path to an appropriate Python executable, and then use that (+ a requirements.txt) to restore the Python packages in e.g. a virtual environment.

However, capturing paths may not be useful since there's no guarantee that /usr/local/bin/python on one system and /usr/local/bin/python on another system are the same.

We could enforce (or at least encourage) the use of virtual environments by name, since it would be easier to orchestrate -- e.g. "virtual environment 'tensorflow-1.13' uses Python x.y.z and these tensorflow packages", and that concept would be uniform across different machines.

The 'reproducible' solution would involve enforcing the use of virtual environments, and capturing Python by version. Then we could (in theory) attempt to find or download that particular version of Python, create a virtual environment using that Python, and then restore any packages recorded in requirements.txt.

Ultimately, I think we want to make sure it's easy for users to collaborate on the same repository, which does imply being able to easily reproduce the environment on different machines.

We should also have some provisioning for automatically selecting a project-local virtual environment called r-reticulate.

Some API proposals:

# init could take a 'python' parameter
renv::init(python = "/usr/bin/python")

# renv could activate a version of Python post-hoc
renv::settings$python("/usr/bin/python")

I would argue no, because it's actually more important for a CRAN package to be compatible with current versions of packages. (Although in the short-term, I can certainly see the advantages of dog-fooding)

vignette("renv") will also work (whereas other packages might have vignettes called introduction). pkgdown also handles specially (putting in the top-level nav, rather than under articles)

Warning message:
Project requested R version '3.5.3' but '3.5.2' is currently being used

It feels like this is going to be annoying until we make it easy to run different versions of R

I just upgraded my version of renv to test last commits

> renv::upgrade()
A new version of the renv package will be installed:

	[0.2.0-59] -> [0.2.0-68]

This project will use the newly-installed version of renv.

Do you want to proceed? [y/N]: Y
Retrieving 'https://api.github.com/repos/rstudio/renv/tarball/18b32a57463100e49c16246a5ee5aece32dc9029' ...
	OK [downloaded 121.9 Kb in 1.5 secs]
Installing renv [0.2.0-68] from GitHub ...
	OK (built from source)

Restarting R session...

Using libraries at paths:
- C:/Users/chris/Documents/R/win-library/3.5
- C:/Program Files/R/R-3.5.3/library
* Project '~/DEV_R/#Blogging/cderv' loaded. [renv 0.2.0-68]

I believe this should upgrade my renv version for the project. It seems the 📦 is not a the project library but in the cache

> find.package("renv")
[1] "C:/Users/chris/AppData/Local/renv/R-3.5/bootstrap/x86_64-w64-mingw32/3.5/renv/0.2.0-68/renv"

I understand it modifies the infrastucture of renv inside the project

However, the lock file is not updated unless I run

> renv::snapshot()
The following lockfile fields will be updated:

  [renv]
  Version: [0.2.0-59] => [0.2.0-68]
  
The lockfile will be written to '~/DEV_R/#Blogging/cderv/renv.lock'.
Do you want to proceed? [y/N]: y
* Lockfile written to '~/DEV_R/#Blogging/cderv/renv.lock'.

I would expect upgrade to update the lock file or at least ask user to do so. What do you think ?

It does not seem my project can work with another version of renv even if the lockfile says so...

I modify the version to previous one (with renv::modify())

> readLines("renv.lock", n = 2)
[1] "[renv]"           "Version=0.2.0-59"

and restarted the session.

The renv version loaded is [renv 0.2.0-68], not like the renv.lock, but determine according to activate. However, this says that everything is ok

> renv::status()
* The project is already synchronized with the lockfile.

> renv::restore()
* The project is already synchronized with the lockfile.

What do you think ?

I think some more helpers to manage renvs that are created elsewhere in the file system (or in the home directory) would be helpful. I have long been annoyed / desiring similar helpers in virtualenv, but maybe I just do not know the incantation. rm -r has been my go-to.

Currently, renv installations live in their own private library, and renv gets loaded with some magic initialization code that gets written to renv/activate.R. We should consider the pros and cons of this approach, versus just having renv live in the private library.

If I want to commit a record of my environment in GH, what do I need to commit? The most recent manifest? The whole manifest directory? Everything?

Related question:

What is the relationship between the env file and the manifest file? Do we need both?

I'm so happy to see renv out in the open now! I'm just getting started and I noticed that after I run renv::init() within a newly created RStudio project that the packages pane still lists all of the available R packages in the global R library. I'm hoping that in a future update that the IDE will only show the packages in the renv custom library, like how packrat currently operates in the IDE?

For a brand new user of renv, it would be really nice if we renv would pre-populate the cache with the system library. That way, creating your first new project environment would be really fast, since the cache would already be warm with the packages you use most!

• renv::install will update the lockfile
• renv::remove will not update the lockfile

It may be intented but as I find it not consistent I wanted to raise a comment. I would have expected it to be the opposite of each other.

The doc says

his implies that you can call (for example) renv::upgrade(remote = "0.1.0-75) to request the installation of that particular version of renv if so required.

But

> renv::upgrade(remote = "0.2.0-59")
Error in unclass(numeric_version(rhs))[[1]] : subscript out of bounds
Call `rlang::last_error()` to see a backtrace

This is because there is no version in records

renv:::renv_remotes_parse("0.2.0-59")
#> $Package
#> [1] "0.2.0-59"
#> 
#> $Version
#> NULL
#> 
#> $Source
#> [1] "CRAN"

so request is NULL and version_compare(current, request) == 0 throws the error

It is like Package and Version field are not correct. I would expect

#> $Package
#> [1] "renv"
#> 
#> $Version
#> [1] "0.2.0-59"
#> 
#> $Source
#> [1] "Github"

Currently from the code, I don't think it is possible to install a specific renv version like documented. I understand that the call should be something like that rstudio/[email protected] to be understand as a github remote

 renv:::renv_remotes_parse("rstudio/[email protected]")

But even like that it is not ok because 0.2.0-59 is a tag name and renv_remotes_parse_github only knows how to handle branch reference (git/refs/heads/)

• https://api.github.com/repos/rstudio/renv/git/refs/heads/0.2.0-59 does not exist
• https://api.github.com/repos/rstudio/renv/git/refs/tags/0.2.0-59 does exist

Github API makes a difference here between commit / branch and tag. You would need to test both or use another approach like download the tarball and read DESCRIPTION inside
https://api.github.com/repos/rstudio/renv/tarball/<any git ref>

Currently, renv::upgrade only works to upgrade from master or branch but not from a version number.

─ Session info --------
 version  R version 3.3.3 (2017-03-06)
 os       macOS  10.14.2           

I am using the latest renv build from master, 49588f3

When I activate a project virtual environment, I get the following libraries:

.libPaths()
[1] "/Users/sean/Documents/Sandbox/renv-test/renv/library/x86_64-apple-darwin13.4.0/3.3/renv-test"
[2] "/Library/Frameworks/R.framework/Versions/3.3/Resources/library"   

Here is the environment file:

[Environment]
Environment=renv-test

[R]
Version=3.3.3
Library=renv-test
Overlay=FALSE

[R/Repositories]
CRAN=https://demo.rstudiopm.com/cran/latest/
dev=https://demo.rstudiopm.com/dev/latest

I would expect only the single library path in this case, not both.

I was playing with renv a bit and I find it pleasant to use - specially I like that renv::install() automatically updates renv.lock, since one can easily forget to call snapshot.

One cool feature that could be added would be a check of installed package versions and versions in renv.lock when R session is started. So if versions for some package are different, then user would be asked to run renv::restore() (or there could be [y/N] prompt to call restore).

Use case:
For example there are multiple people working on the same project and then someone updates/downgrades some library and pushes changes to git. Then other guys might pull changes but they would not notice package changes and project would break for them. With this check everyone would know right away that some package version was changed and that they should update their local packages.

The snapshots are human readable, but it'd be nice to have a "preview" method that shows what would change if you were to restore a snapshot compared to the active snapshot.

Today, if you run renv::modify in a project that doesn't have a project environment, it simply opens an empty "edit" screen....

currently I get an error after closing usethis::file.edit() window

> utils::file.edit("renv.lock")
Error in editor(file = file, title = title) : 
  Error in parse(targetFile) : renv.lock:1:1: unexpected '['
1: [
    ^
You can attempt to correct the error using renv.lock = edit()
Call `rlang::last_error()` to see a backtrace

Got the same with file.edit

> utils::file.edit("renv.lock")
Error in editor(file = file, title = title) : 
  Error in parse(targetFile) : renv.lock:1:1: unexpected '['
1: [
    ^
You can attempt to correct the error using renv.lock = edit()
Call `rlang::last_error()` to see a backtrace

don't know if it will stay or not (seems like it would have been removed after #15)...

Do you want to proceed? [Y/n]: 
Operation aborted.

i.e. the default actually appears to be N

i.e. it should capture the dependency information from the system library, filtered by dependencies()

Along the lines of what is mentioned in #1 , is it possible to share a package cache across users?

For documentation, this was requested in Packrat:

rstudio/packrat#357

Minor comment but when I initialized renv in a Shiny app project directory (in this case a very simple example with shinydashboard) it took a bit of time for the packages to be copied to the cache. It would be nice to have an optional flag to show a progress bar in this step.

The doc says that

But it is not the case... It took me some digging to find it.
by default we have

renv:::renv_paths_cache()
#> [1] "C:\\Users\\chris\\AppData\\Local/renv/R-3.5/cache/v2"

which is expected according to the code in path.R

I think the doc should be updated

It would be great if the renv.lock file can also accommodate a library that is installed either from a local .tar.gz file or via devtools::install_github(, auth_token = ...). I am not sure if this is easy to facilitate though.

Great work on this library, incredibly useful!

I feel like I am seeing some type of weirdness with renvs that are tried with a path.

library(reprex)
renv::init()
#> * Virtual environment 'reprex548277d89f6' activated -- please restart the R session.
renv::activate()
#> * Virtual environment 'reprex548277d89f6' activated -- please restart the R session.
text <- "blahblah2"
renv::create(paste0("/", text))
system("ls ~/.renv/environment", intern = TRUE)
#> [1] "blah" "tmp"

system("ls ./", intern = TRUE)
#> [1] "renv"                   "reprex_reprex.R"       
#> [3] "reprex_reprex.spin.R"   "reprex_reprex.spin.Rmd"

renv::deactivate()
#> * Virtual environment deactivated -- please restart the R session.
renv::create(paste0("/tmp/", text))
system("ls ~/.renv/environment", intern = TRUE)
#> [1] "blah" "tmp"
system("ls ~/.renv/environment/tmp/", intern = TRUE)
#> [1] "blah"
renv::create(paste0("./", text))
#> Error: Local virtual environment './blahblah2' already exists.
system("ls ~/.renv/environment", intern = TRUE)
#> [1] "blah" "tmp"

system("ls ./", intern = TRUE)
#> [1] "renv"                   "reprex_reprex.R"       
#> [3] "reprex_reprex.spin.R"   "reprex_reprex.spin.Rmd"

^{Created on 2019-01-03 by the reprex package (v0.2.1)}

I think this relates to my preferences in virtualenv 😄

Not sure if renv::init() is the only way you can get a renv within a project, and whether any project would ever be justified in having more than one renv locally.

The other use case to wonder about here is whether you could put renvs or the cache at a shared location. This is something that I know some users were interested in for packrat.

See e.g.

rstudio/packrat#181
rstudio/packrat#454

Then all renv related files would be in the same directory.

Right now, we suppress output from install.packages(); we should instead redirect output to a file and show that output on failure.

The motivation for this: for a project that is "finished", one might want to collect the R packages the project depends on and ensure they're installed directly in the project library (rather than linked from the cache). This is primarily just to make sure the project library could outlive the cache, or (perhaps) be more easily transplanted to another system.

This is in the same vein as packrat::bundle().

This might not be common use case to have project root in $HOME. But when running project in docker one might have it there.

The issue seems to be, that source('activate.R') is called recursively when one have project root in $HOME or in other words when .Rprofile is in $HOME (~/.Rprofile).

The issue seems to be in 'activate.R' in lines:

# source the user profile if any, respecting R_PROFILE_USER
profile <- Sys.getenv("R_PROFILE_USER", unset = "~/.Rprofile")
if (file.exists(profile)) source(profile)

ref:

It'd be nice to have a function that sets the repo option for the environment

The alternative is teaching users to set the repo option in their .Rprofile (which is a PITA)

The current default, to record whatever repo is in use at the time of snapshot makes sense, but it'd be nice if there was an explicit way to update the repo setting through renv and have it apply to the session

Don't know if it is a side effect of #46 but currently the cache is not the same

packageVersion("renv")
#> [1] '0.2.0.68'
renv:::renv_paths_cache()
#> [1] "C:\\Users\\chris\\AppData\\Local/renv/cache/R-3.5/v2"
renv:::renv_cache_list()
#> character(0)
library(magrittr)
fs::dir_ls(renv:::renv_paths_common(""))  %>%
  fs::dir_ls()
#> C:/Users/chris/AppData/Local/renv/R-3.5/binary
#> C:/Users/chris/AppData/Local/renv/R-3.5/bootstrap
#> C:/Users/chris/AppData/Local/renv/R-3.5/cache
#> C:/Users/chris/AppData/Local/renv/R-3.5/repos
#> C:/Users/chris/AppData/Local/renv/source/rcmdcheck
#> C:/Users/chris/AppData/Local/renv/source/remotes
#> C:/Users/chris/AppData/Local/renv/source/renv
#> C:/Users/chris/AppData/Local/renv/source/skeleton

^{Created on 2019-04-06 by the reprex package (v0.2.1.9000)}

As you see above renv:::renv_paths_cache() path does not exist. I have renv/R-3.5 and renv/source and not renv/cache

Something has changed I guess...

This concerns the use of renv::init() for a brand new project.

Suppose you have a project that depends on dplyr; however, for some reason your user library is missing the dependency tibble. In this case, we see output like the following:

> deps <- renv:::dependencies()
> deps
                            Source Package Require Version
1 /Users/kevin/scratch/oops/deps.R   dplyr

> str(renv:::renv_dependencies(".", deps$Package))
List of 19
 $ tidyselect: chr "~/Library/R/3.5/library/tidyselect"
 $ R6        : chr "~/Library/R/3.5/library/R6"
 $ magrittr  : chr "~/Library/R/3.5/library/magrittr"
 $ assertthat: chr "~/Library/R/3.5/library/assertthat"
 $ graphics  : chr "/Library/Frameworks/R.framework/Resources/library/graphics"
 $ tools     : chr "/Library/Frameworks/R.framework/Resources/library/tools"
 $ plogr     : chr "~/Library/R/3.5/library/plogr"
 $ glue      : chr "~/Library/R/3.5/library/glue"
 $ utils     : chr "/Library/Frameworks/R.framework/Resources/library/utils"
 $ dplyr     : chr "~/Library/R/3.5/library/dplyr"
 $ tibble    : logi NA
 $ grDevices : chr "/Library/Frameworks/R.framework/Resources/library/grDevices"
 $ Rcpp      : chr "~/Library/R/3.5/library/Rcpp"
 $ stats     : chr "/Library/Frameworks/R.framework/Resources/library/stats"
 $ methods   : chr "/Library/Frameworks/R.framework/Resources/library/methods"
 $ rlang     : chr "~/Library/R/3.5/library/rlang"
 $ pkgconfig : chr "~/Library/R/3.5/library/pkgconfig"
 $ BH        : chr "~/Library/R/3.5/library/BH"
 $ purrr     : chr "~/Library/R/3.5/library/purrr"

> renv::init()
* Discovering package dependencies ... Done!
* Copying packages into the cache ... Done!
* Resolving missing dependencies  ...
Retrieving 'https://cran.rstudio.com/bin/macosx/el-capitan/contrib/3.5/tibble_2.1.1.tgz' ...
        OK [downloaded 310.6 Kb in 0.1 secs]
Installing crayon [1.3.4] ...
        OK (linked cache)
Installing cli [1.1.0] ...
        OK (linked cache)
Installing fansi [0.4.0] ...
        OK (linked cache)
Installing utf8 [1.1.4] ...
        OK (linked cache)
Installing pillar [1.3.1] ...
        OK (linked cache)
Installing tibble [2.1.1] from CRAN ...
        OK (installed binary)
* Lockfile written to '~/scratch/oops/renv.lock'.
* renv activated -- please restart the R session.

So, renv is able to recover and automatically download and install tibble, which is helpful. Some questions:

Suppose a version of tibble was installed in the cache. In what situations can we use it? We could:

• Just use the newest version of tibble;
• Use a version from the cache if it matches the latest version available on CRAN;
• Use a version from the cache if it's equal to, or newer, than the version on CRAN;
• Ask the user?

See: rstudio/packrat#535

Report if the user doesn't have write access to their project library (or some packages within) for some reason, as future attempts to update those packages could fail.

hadley [9:37 AM]

1. No lockfile + no package library -> discover package dependencies, install them, reload
2. Lockfile + no package library -> restore the project from the lockfile, reload
3. Anything else - ask whether you want 1 or 2

I think a reminder message after the function completes would be adequate. Something like: "don't forget to snapshot when you're confident the packages work"