PolyFun can't find R dependencies about echofinemap HOT 10 OPEN

I think a clue is at this part:

[INFO]  R home found: /Library/Frameworks/R.framework/Resources
[DEBUG]  Looking for LD_LIBRARY_PATH with: /Library/Frameworks/R.framework/Resources/bin/Rscript -e cat(Sys.getenv("LD_LIBRARY_PATH"))
[INFO]  R library path: 
[INFO]  LD_LIBRARY_PATH: 

If I can tell rpy2 to look for R or my R library at a specific location, then maybe it will find the packages.

From within Rstudio (outside conda env):

> .libPaths()
[1] "/Library/Frameworks/R.framework/Versions/4.2/Resources/library"

From within conda env:

> .libPaths()
[1] "/Users/schilder/Library/Caches/org.R-project.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/lib/R/library"

from echofinemap.

@omerwe let me know if you have any ideas about this, or if it's something you've encountered before. Thanks!

from echofinemap.

@jdblischak I think your input here would also be super valuable if you wouldn't mind! Thanks!

from echofinemap.

via command line (within conda env)

From command line, with the conda env activated, this command works fine and is able to load.

import rpy2
import rpy2.robjects.numpy2ri as numpy2ri
from rpy2.robjects.packages import importr
importr('Ckmeans.1d.dp')
median_seg_func = ro.r('Ckmedian.1d.dp')
mean_seg_func = ro.r('Ckmeans.1d.dp')

via Rstudio python interpreter (outside conda env)

However, when I run the same command through a system() wrapper of the polyfun.py python script, I get:

import rpy2
import rpy2.robjects.numpy2ri as numpy2ri

R[write to console]: Error in .Primitive("as.environment")("package:utils") : 
  no item called "package:utils" on the search list

RRuntimeError: Error in .Primitive("as.environment")("package:utils") : 
  no item called "package:utils" on the search list

The above error message persists even when I explicitly set the R_HOME global:

import os
os.environ['R_HOME'] = '/Users/schilder/Library/Caches/org.Rproject.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/lib/R'

from echofinemap.

This isn't a PolyFun-specific issue. I run PolyFun inside of a conda env all the time. The official PolyFun conda env spec includes R and its dependencies, and Python/rpy2 thus have no trouble finding them.

run the same command through a system() wrapper of the polyfun.py python script

This is where I suspect the awkwardness is arising. Are you somehow activating the conda env in the system() call prior to running the Python script? The Python parts are working because you are pointing directly to the python executable, so python "knows" where to find its installed packages. However, it has no idea that it is inside a conda env, nor that R is available inside that same conda env.

I think it's probably the same situation with the command-line example from the original post. You need to activate the conda env first.

# Replace this
/Users/schilder/Library/Caches/org.R-project.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/bin/python /Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/polyfun.py
# with
conda activate echoR_mini
python /Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/polyfun.py

from echofinemap.

Yes indeed @jdblischak, this is not specific to PolyFun. PolyFun as controlled by echofinemap via Rstudio is just an example of a scenario in which this occurs. The reason I'm doing it this way is bc my main focus is getting echofinemap (and all its dependencies) to fully work from within Rstudio.

Running PolyFun directly in the conda env from the terminal is no problem, it's interfacing with the dependencies inside the conda env from outside it that I'm having trouble with. This occurs with both the "polyfun" and "echoR_mini" (which is partially based on "polyfun") conda envs.

Activating the conda env via basilisk or reticulate doesn't seem to help finding the R packages within the env. This might be due to the disconnect between using either of these to activate the env from within R, and the system function I'm using to execute the python script. Or it might simply be that both basilisk and reticulate only handle making python packages available to R, not R packages that are also installed in the same env.

Despite all these constraints, I do think there should be a way to get polyfun (or any python script) to find R packages in a given coda env, even when it's being called from outside it (e.g. from Rstudio). Setting os.environ['R_HOME'] seems like it should work, but nevertheless gives the error I mentioned above.

from echofinemap.

@LTLA do you happen to know if there's a way to activate a conda environment from within Rstudio such that the R packages within that conda env become available to rpy2 scripts?

I realize this phrasing is rather confusing.
If it helps, the control hierarchy would look something like this:

- Rstudio 
-> Basilisk/reticulate 
--> python 
---> rpy2 
----> R packages installed in conda env

from echofinemap.

Not to my knowledge. basilisk.utils::activateEnvironment should set the appropriate environment variables inside the current R session, which should then be inherited by any R subprocess that is kicked off somewhere. I'm guessing that the activation process would try to set R_LIBS, R_LIBS_SITE or R_LIBS_USER; I would further speculate that these variables only have an effect during R start-up, so one would have to make sure that they are set before the subprocess starts.

from echofinemap.

Thanks for the feedback @LTLA ! These are some helpful clues. I'll keep trying and keep everyone posted if I figure out a solution.

from echofinemap.

I put together a minimal example to see if I could figure this out. Unfortunately I couldn't get it to work. rpy2 always identifies the outer R process (in the original post, this would be R running in RStudio), instead of the intended R installed in a conda env (the env to run PolyFun).

# Outer env to run R
mamba create --yes -n outer r-base=4.2 r-reticulate

# Inner env that calls R from Python via rpy2 (simulate PolyFun)
mamba create --yes -n inner python rpy2 r-base=4.0 r-susier

# Start from outer R process
mamba activate outer

# Python script that uses rpy2 to find R
cat > rpy2-code.py << EOL
import sys
print(f"Python executable: {sys.executable}")
import rpy2.situation
print(f"R_HOME: {rpy2.situation.get_r_home()}")
EOL

# R script that uses reticulate to activate conda env
cat > run-code-in-conda-env.R << EOL
library("reticulate")
use_condaenv("inner")
reticulate::source_python("rpy2-code.py")
EOL

# reticulate finds the inner env python, but rpy2 still detects outer R
Rscript run-code-in-conda-env.R
## Python executable: ~/.conda/envs/inner/bin/python
## R_HOME: ~/.conda/envs/outer/lib/R

from echofinemap.