scicloj / clojisr Goto Github PK

View Code? Open in Web Editor NEW

147.0 7.0 10.0 3.07 MB

Clojure speaks statistics - a bridge between Clojure to R

Home Page: https://scicloj.github.io/clojisr/

License: Eclipse Public License 2.0

Clojure 99.49% Dockerfile 0.37% SCSS 0.12% Shell 0.02%

clojure r rlang

clojisr's Introduction

ClojisR

Clojure speaks statistics - a jisr between Clojure and R

Documentation

How to pronounce it?

The beginning of the pronunciation is the same as Clojure, but then it rhymes with 'kisser'. Actually, the last vowel is nonexistent, so you may try to pronounce it with less movement between s and r, like 'yesssr!'.

Status

still evolving
have been tested and used for few years

Clojurists Together

Hurray!

We are happy to announce that ClojisR is selected by Clojurists Together in Q4 2020! Expect more information soon.

Scope of the project

Libraries for Clojure-R interop are not new - see this list.

This project suggests yet another way to use R from Clojure.

Currently we target only JVM Clojure, but we are interested in generalizing the work to Clojurescript.

The related problem, of calling Clojure from R, may be addressed too in the future. We are experimenting with that.

Meta Goals

Realize what is essential for Clojure to become a beginner-friendly solution for data science.
Expose the Clojure ecosystem to a different culture and to more diverse groups of users/programmers.

Technical Goals

A Function-centric API, where the default mode of usage is calling R functions on R objects, from Clojure (Status: supported)
"R code as Clojure data", inspired by the EDN-based syntax inroducted in gg4clj and used in huri (Status: supported)
Interop with minimal copying of data (Status: supported)
Compatibility with common data abstractions such as tech.ml.dataset datasets (Status: partially supported)
Convenient wrappers for common use cases, such as visualization (Status: basic support for plots and Rmarkdown)
Abstraction over different runtimes (GNUR R, Renjin, FastR) (Status: GNU R is supported through Rserve; Renjin has some basic support, moved to a separate library ClojisRenjin)
Convenient multi-session support (Status: basic support with some known issues)

Usage requirements

Linux, MacOS or WSL (Windows Subsystem for Linux)
JDK 1.8 or later
Clojure 1.9.0 or later
R
The Rserve R package (install.packages("Rserve",,"http://rforge.net")) Tested with Rserve version 1.8.6. Earlier versions are known to have a bug.
This library:

MacOS installation

Installing R with Rserve on MacOS can be problematic due to issues related to openssl installation. Please apply following steps (thanks to @ezmiller):

Download the lastest R for mac from here: https://cloud.r-project.org/
Install openssl: brew install openssl.
Make sure that the openssl library is linked. Try in order:
- brew link --force openssl
- If that doesn't work, follow directions in brew info openssl for setting environment variables. Set the LIBRARY_PATH environment variable to the location of the library, e.g. export LIBRARY_PATH=/usr/local/opt/[email protected]/lib.

Setting up the logging

clojisr library uses clojure/tools.logging for logging. tools.logging doesn't force any logging backend and users have to configure it on their side. To force specific backend you can set it using JVM options, for example in lein profile.clj or project.clj:

:jvm-opts ["-Dclojure.tools.logging.factory=clojure.tools.logging.impl/jul-factory"]

Docker image

Thanks to Carsten Behring we have a Docker template prepared

https://github.com/behrica/clj-py-r-template

The Dockerfile of the template adds as well python + libpython-clj for completeness.

So it has in a single place all dependencies and they do work together and no further setup is required.

There is as well a devcontainer setup in this GitHub template. It provides as well an out-of-the box working environment and template for projects using ClojisR (and libpythion-clj

Checking if it works

This should work for you (assuming you have the clj tool):

$ clj -Sdeps '{:deps {scicloj/clojisr {:mvn/version "1.0.0-BETA20"}}}'
Clojure 1.10.1
user=> (require '[clojisr.v1.r :refer [r]])

user=> (r '(+ 1 2))
[1] 3

Known issues

clojisr can behave in a strange way when abandoned R (with Rserve) processes are running. Please kill such processes before creating an Rserve session.
Nextjournal can hang due to problems with logging, please add org.slf4j/slf4j-nop {:mvn/version "1.7.30"} to the deps to disable logger.

Video presentations

The main ideas were discussed at Scicloj Web meeting #7 and ClojuTRE 2019.

Note however that:

The API has changed since then (code generation mechanism, data visualization support, printing - see the Tutorials below).
On the meeting, there is some careless use of the term 'zero copy'. Actually, what is usually meant by this term is not supported at the moment.

Tutorials

The tutorials are now organized in a book.
More examples -- see the clojisr-examples repo

Background

Choices of the current project

Here are the current priorities of the project in some central design and implementation questions.

Future opportunities

Here are some possible future developments we are considering.

Discussion

Please share your comments, thoughts, ideas and questions at the Issues Page of this project and at the r-interop stream of the Clojurians Zulip.

Also we run a stream for developers or people interested in contributing.

Testing

The tests are regular clojure.test tests, but are auto-genedated from the tutorials.

Tools used

Working on this project, we enjoyed the following tools (partial list):

In early versions, hara.test was used for automated docstrings by tests. We may come back to using it.
clj-kondo for code quality control
Clay for documentation and test generation

License

This program and the accompanying materials are made available under the terms of the Eclipse Public License 2.0 which is available at http://www.eclipse.org/legal/epl-2.0.

This Source Code may also be made available under the following Secondary Licenses when the conditions for such availability set forth in the Eclipse Public License, v. 2.0 are satisfied: GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version, with the GNU Classpath Exception which is available at https://www.gnu.org/software/classpath/license.html.

clojisr's People

Contributors

Stargazers

Watchers

Forkers

behrica genmeblog niharikam333 pink-junkjard tuh8888 ashimapanjwani randomactsofsoftware chagasvinicius spartakos87 jansuran03

clojisr's Issues

depending on an additional Maven repository

Currently, Clojuress depends on Renjin.
https://mvnrepository.com/artifact/org.renjin/renjin-script-engine/3.5-beta65

This requires to add an additional Maven repository on any project that depends on Clojuress.

In other words, this does not work:

clj -Sdeps '{:deps {scicloj/clojuress {:mvn/version "1.0.0-BETA4"}}}}'

but this does:

clj -Sdeps '{:mvn/repos {"bedatadriven" {:url "https://nexus.bedatadriven.com/content/groups/public/"}} :deps {scicloj/clojuress {:mvn/version "1.0.0-BETA4"}}}}'

One solution to avoid this would be to load the Renjin jar and add it to the classpath on runtime, when needed, and not through the Leiningen project.

Another one would be to make sure that the Renjin jars live in a more standard place.

unary minus on colon

Trying to translate

-(3:7)

This doesn't work

(r '(- (colon 3 7)))

workaround

(r '(- [(colon 3 7)]))

Unary minus should wrap next form into parantheses

Save chart to other types of formats and BufferedImage

Enhancement

Save any rendered chart to supported image formats (png, bmp, jpg, tiff) also render to BufferedImage (saving a file and then loading into RGBA BufferedImage)

Add p_load as part of the usual workflow

p_load can install missing packages if they are missing.

Following a suggestion of Nick Stares, we may wish to make it an almost-transparent behaviour in our interop story.

Should it be part of the require-r default behaviour?

to consider: rmarkdown function should return string instead of file

Enabling the use of both backends (Rserve and Renjin) in the same project but different namespaces

Explicit data loading into clj namespace

Sometimes data from the package should be loaded on demand using data() function on the R side.

I propose creating data function which:

loads data on the R side
creates symbol on the clj side which returns loaded data

Option: list all data from given package by calling data(package='any_r_package')$results[,'Item']"

empty-symbol in functional call

Looks like there are cases when creation of empty-symbol fails.

I found a case when (bra data.table nil something) throws an exception about missing value.

The root of a problem lays with the fact that empty symbol cannot be assigned to a value (http://adv-r.had.co.nz/Expressions.html#names). When we evaluate something it's always assigned to a value.

What seems to work perfectly is artificial RObject:

(def empty-symbol (clojisr.v1.robject/->RObject "" (:session (r 1)) nil ["name"]))

That means to we have to have special case for RObject recreation during session change. When we set code to nil they should stay untouched.

add `help`

Help can be retrieved using following R call: capture.output(tools:::Rd2txt(utils:::.getHelpFile(as.character(help(mean))), options=list(underline_titles=FALSE)))

Rd2HTML can be used to return HTML.

We can implement:

help function which returns just help string, or a section
help-map function which parses help and returns map of sections
print-help print help (or section) to stdout

Help can be used on R name (string or symbol) or on RObject (if created via require-r should contain original name in meta)

Attaching doc string to required R symbol can be slow (need to test it first) - maybe we can do it in future task?

(something) form

Sometimes form should be wrapped into parantheses. We need to have a ability to emit (something).

RServe - named list processing is too narrow

Sometimes REXPList is returned by R

(r/r->java (r "formals(mean)"))
;; => #object[org.rosuda.REngine.REXPList 0x2316ca25 "org.rosuda.REngine.REXPList@2316ca25[2]named"]
(.isNamed (.asList (r/r->java (r "formals(mean)"))))
;; => true

I attempted to use the same strategy as REXTGenericVector to create Clojure object, unfortunately REXPList doesn't have named parameter. Options:

(quick and dirty), wrap into REXTGenericVector and process

(org.rosuda.REngine.REXPGenericVector. (.asList (r/r->java (r "formals(mean)"))))

Refactor to operate on RList via keys() and at() functions.

(seq (.keys (.asList (r/r->java (r "formals(mean)")))));; => ("x" "...")

side note: formals() returns list where keys are important, values are empty symbols

Memory management issues

using-sessions/eval-code sets garbage collector for RObject. It seems not to work properly. Objects created by require-r and bound to vars are set to NULL very quickly. They are no longer available.

Discussion: https://clojurians.zulipchat.com/#narrow/stream/204621-r-interop/topic/releasing.20memory

org.clojure/tools.logging dependency

I just added clojisr beta 11 to a new project.clj
On compiling I got the error that a namespace from org.clojure/tools.logging is missing.
I fixed it by adding [org.clojure/tools.logging "1.0.0"] to my project.clj

I then did lein deps :tree and found this dependency of clojisr:
[commons-logging "1.2"]

Unfortunately since you dont have ci tool, I cannot really look into the source related to this
version. And I am too lazy to extract the source code from the jar that is in:
~/.m2/repository/scicloj/clojisr/1.0.0-BETA11

Could it be that you changed logging system and forgot to update a dependency?

Loading dynamic dependencies fails with Java 11. The problem may be related to Alembic.

We get this error message:

Syntax error (ExceptionInfo) compiling at (alembic/still.clj:65:1).
Alembic can not manipulate specified ClassLoader.

when trying to add a dependency on runtime here:
https://github.com/scicloj/clojisr/blob/master/src/clojisr/v1/renjin.clj#L5

Discussion and possible solution by @joinr:
https://clojurians.zulipchat.com/#narrow/stream/204621-r-interop/topic/problems.20with.20alembic

tech.ml.dataset do not recognize `NA` in logical columns

Rserve has a maximal socket buffer size, and that limits the amount of data we can pass through it.

Our solution can be to wrap the r->clj and clj->r functions in a way that will transparently break the data into chunks before passing it over the socket.

Rethink and rewrite session management

The current notion of sessions (Rserve sessions in the current implementation) requires some rethinking to allow for a more robust experience.

See some discussion here.

seqable

Dataset column is seqable but not sequential. I think we want to add seqable at the end of check in form->code and treat is as a sequence.

Abandoned R processes

When Java is killed or stopped without closing session, R process is still running which can cause strange error for another sessions (see: https://clojurians.zulipchat.com/#narrow/stream/151924-data-science/topic/r-interop)

We should check the R processes and safely run fresh on not used port informing use that R processes exist. Or force somehow to shutdown R when there is no connection.

Create `defr`

Create defr or def-r which creates a symbol on the R side too.

See discussion: https://clojurians.zulipchat.com/#narrow/stream/204621-r-interop/topic/formulas

Add a Renjin backend

This library has begun as an experiment with Renjin, and it is time to come back to the old code and adapt it the current protocols.

This will give us access to a huge collection of pure-JVM functions for statistics, IO (e.g., read.csv) and data processing (e.g., dataframes and their transformations).

simpler API for Renjin

This issue suggests to add to the clojisr.v1.renjin namespace versions of the main API functions from clojisr.v1.renjin , that make sure to run on a :renjin session-type.

Related to #35.

Change logger

Change logger to tools.logger

Separate session from Rserve server

We need to separate rserve server process from it's connections. Such decoupling will allow to create multiple connections/sessions to the same server without need of creating separate processes. Each session can be run on different thread or future (I can imagine 'session on demand': open, calculate, close).

When requiring ns, get "Connection refused"

Following the steps in the README for testing if the installation works, upon requiring the ns

$ clj -Sdeps '{:deps {scicloj/clojisr {:mvn/version "1.0.0-BETA10"}}}'

user=> (require '[clojisr.v1.r :refer [r]])

I am told that the "Connection refused" (see 1. in stacktrace). From what I can tell, clojisr is attempting to start Rserve via the command

$ /usr/bin/R --no-save --slave -e "library(Rserve); run.Rserve(port=44735);"

By requiring the rserve ns first, I can see that this is the default port.

(require '[clojisr.v1.rserve :as rserve])
(rserve/set-as-default!)
;; {:port 44735, :session-type :rserve}

I have verified that my Rserve package is the recommended version.

> sessionInfo()
R version 4.0.0 (2020-04-24)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Manjaro Linux

...

other attached packages:
[1] Rserve_1.8-6

loaded via a namespace (and not attached):
[1] compiler_4.0.0  grid_4.0.0      lattice_0.20-41

And I am using scicloj/clojisr {:mvn/version "1.0.0-BETA10"} in my deps.edn.

What am I missing?

Stacktrace:

Unhandled clojure.lang.Compiler$CompilerException
Error compiling r.clj at (118:10)

   #:clojure.error{:phase :execution, :line 118, :column 10, :source "r.clj"}
             Compiler.java: 3707  clojure.lang.Compiler$InvokeExpr/eval
             Compiler.java:  457  clojure.lang.Compiler$DefExpr/eval
             Compiler.java: 7182  clojure.lang.Compiler/eval
             Compiler.java: 7636  clojure.lang.Compiler/load
                   RT.java:  381  clojure.lang.RT/loadResourceScript
                   RT.java:  372  clojure.lang.RT/loadResourceScript
                   RT.java:  459  clojure.lang.RT/load
                   RT.java:  424  clojure.lang.RT/load
                  core.clj: 6126  clojure.core/load/fn
                  core.clj: 6125  clojure.core/load
                  core.clj: 6109  clojure.core/load
               RestFn.java:  408  clojure.lang.RestFn/invoke
                  core.clj: 5908  clojure.core/load-one
                  core.clj: 5903  clojure.core/load-one
                  core.clj: 5948  clojure.core/load-lib/fn
                  core.clj: 5947  clojure.core/load-lib
                  core.clj: 5928  clojure.core/load-lib
               RestFn.java:  142  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 5985  clojure.core/load-libs
                  core.clj: 5969  clojure.core/load-libs
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 6007  clojure.core/require
                  core.clj: 6007  clojure.core/require
               RestFn.java:  408  clojure.lang.RestFn/invoke
                      REPL:   61  user/eval57512
                      REPL:   61  user/eval57512
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
    interruptible_eval.clj:   91  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   20  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  155  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  190  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  189  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  830  java.lang.Thread/run

Caused by org.rosuda.REngine.Rserve.RserveException
Cannot connect: Connection refused

          RConnection.java:   90  org.rosuda.REngine.Rserve.RConnection/<init>
          RConnection.java:   60  org.rosuda.REngine.Rserve.RConnection/<init>
               session.clj:  115  clojisr.v1.impl.rserve.session/make
               session.clj:  105  clojisr.v1.impl.rserve.session/make
               session.clj:   46  clojisr.v1.session/make
               session.clj:   37  clojisr.v1.session/make
               session.clj:   98  clojisr.v1.session/make-and-init
               session.clj:   97  clojisr.v1.session/make-and-init
               session.clj:  104  clojisr.v1.session/fetch-or-make
               session.clj:  102  clojisr.v1.session/fetch-or-make
                     r.clj:   23  clojisr.v1.r/r
                     r.clj:   22  clojisr.v1.r/r
               RestFn.java:  410  clojure.lang.RestFn/invoke
                  AFn.java:  154  clojure.lang.AFn/applyToHelper
               RestFn.java:  132  clojure.lang.RestFn/applyTo
             Compiler.java: 3702  clojure.lang.Compiler$InvokeExpr/eval
             Compiler.java:  457  clojure.lang.Compiler$DefExpr/eval
             Compiler.java: 7182  clojure.lang.Compiler/eval
             Compiler.java: 7636  clojure.lang.Compiler/load
                   RT.java:  381  clojure.lang.RT/loadResourceScript
                   RT.java:  372  clojure.lang.RT/loadResourceScript
                   RT.java:  459  clojure.lang.RT/load
                   RT.java:  424  clojure.lang.RT/load
                  core.clj: 6126  clojure.core/load/fn
                  core.clj: 6125  clojure.core/load
                  core.clj: 6109  clojure.core/load
               RestFn.java:  408  clojure.lang.RestFn/invoke
                  core.clj: 5908  clojure.core/load-one
                  core.clj: 5903  clojure.core/load-one
                  core.clj: 5948  clojure.core/load-lib/fn
                  core.clj: 5947  clojure.core/load-lib
                  core.clj: 5928  clojure.core/load-lib
               RestFn.java:  142  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 5985  clojure.core/load-libs
                  core.clj: 5969  clojure.core/load-libs
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 6007  clojure.core/require
                  core.clj: 6007  clojure.core/require
               RestFn.java:  408  clojure.lang.RestFn/invoke
                      REPL:   61  user/eval57512
                      REPL:   61  user/eval57512
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
    interruptible_eval.clj:   91  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   20  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  155  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  190  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  189  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  830  java.lang.Thread/run

Caused by java.net.ConnectException
Connection refused


                  Net.java:   -2  sun.nio.ch.Net/connect0
                  Net.java:  493  sun.nio.ch.Net/connect
                  Net.java:  482  sun.nio.ch.Net/connect
        NioSocketImpl.java:  588  sun.nio.ch.NioSocketImpl/connect
      SocksSocketImpl.java:  339  java.net.SocksSocketImpl/connect
               Socket.java:  603  java.net.Socket/connect
               Socket.java:  552  java.net.Socket/connect
               Socket.java:  475  java.net.Socket/<init>
               Socket.java:  249  java.net.Socket/<init>
          RConnection.java:   85  org.rosuda.REngine.Rserve.RConnection/<init>
          RConnection.java:   60  org.rosuda.REngine.Rserve.RConnection/<init>
               session.clj:  115  clojisr.v1.impl.rserve.session/make
               session.clj:  105  clojisr.v1.impl.rserve.session/make
               session.clj:   46  clojisr.v1.session/make
               session.clj:   37  clojisr.v1.session/make
               session.clj:   98  clojisr.v1.session/make-and-init
               session.clj:   97  clojisr.v1.session/make-and-init
               session.clj:  104  clojisr.v1.session/fetch-or-make
               session.clj:  102  clojisr.v1.session/fetch-or-make
                     r.clj:   23  clojisr.v1.r/r
                     r.clj:   22  clojisr.v1.r/r
               RestFn.java:  410  clojure.lang.RestFn/invoke
                  AFn.java:  154  clojure.lang.AFn/applyToHelper
               RestFn.java:  132  clojure.lang.RestFn/applyTo
             Compiler.java: 3702  clojure.lang.Compiler$InvokeExpr/eval
             Compiler.java:  457  clojure.lang.Compiler$DefExpr/eval
             Compiler.java: 7182  clojure.lang.Compiler/eval
             Compiler.java: 7636  clojure.lang.Compiler/load
                   RT.java:  381  clojure.lang.RT/loadResourceScript
                   RT.java:  372  clojure.lang.RT/loadResourceScript
                   RT.java:  459  clojure.lang.RT/load
                   RT.java:  424  clojure.lang.RT/load
                  core.clj: 6126  clojure.core/load/fn
                  core.clj: 6125  clojure.core/load
                  core.clj: 6109  clojure.core/load
               RestFn.java:  408  clojure.lang.RestFn/invoke
                  core.clj: 5908  clojure.core/load-one
                  core.clj: 5903  clojure.core/load-one
                  core.clj: 5948  clojure.core/load-lib/fn
                  core.clj: 5947  clojure.core/load-lib
                  core.clj: 5928  clojure.core/load-lib
               RestFn.java:  142  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 5985  clojure.core/load-libs
                  core.clj: 5969  clojure.core/load-libs
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 6007  clojure.core/require
                  core.clj: 6007  clojure.core/require
               RestFn.java:  408  clojure.lang.RestFn/invoke
                      REPL:   61  user/eval57512
                      REPL:   61  user/eval57512
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
    interruptible_eval.clj:   91  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   20  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  155  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  190  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  189  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  830  java.lang.Thread/run

Move RObject IFn implementation to functions ns

I propose to move RObject IFn implementation to functions namespace.

This way:

We can avoid resolve each time function is called
Enhancing RObject to be a function perfectly matches functions namespace

Why wrapping TRUE -> [true]?

I think that simple TRUE, FALSE or NA values should be converted to true, false and nil without wrapping into a vector.

R datasets -> tech.ml.dataset conversion issues

As example: demonsnacks from LaplacesDemon has row names. They are not visible in dataset after conversion.

I think they should be added as the first column (additional one)

Recover session after killing R processes

When R processes are killed externally, session should be recovered (process respawned and client connected). Now exception is thrown.

error is thrown when calling a fn that requires user input

When you call the tidytext function get_sentiments and you have not yet downloaded the required dataset, you get a request for user input if you are working in R:

> get_sentiments("afinn")
Do you want to download:
 Name: AFINN-111
 URL: http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010
 License: Open Database License (ODbL) v1.0
 Size: 78 KB (cleaned 59 KB)
 Download mechanism: https

1: Yes
2: No

When you call this fn in clojisr, you get an error that doesn't really indicate what has happened. You do see the output in the console.

It would be nice to somehow allow for user input across the divide.

Primitive conversion in vector

Following check fails

(let [x1 (r->clj (r [109 65 22 3 1]))
      x2 (r->clj (r "c(109, 65, 22, 3, 1)"))]
  (check = x1 x2))

x1 contains ints
x2 contains doubles

RObject as map value is not translated to R

(clj->r (base/list [1 2 3]))
;; => [[1]]
;; [1] 1 2 3
(clj->r {:a 1 :b (base/list [1 2 3])})
;; ERROR (ClassCastException)
(clj->r {:a 1 :b (r->clj (base/list [1 2 3]))})
;; => $a
;; [1] 1
;;
;; $b
;; $b[[1]]
;; [1] 1 2 3

column-map in tech.ml.dataset is renamed starting from beta39

"Renamed column-map to column-name->column-map"

Be careful when upgrading to the newest tech.ml.dataset

ensure locking when talking with an Rserve

In the main place we use it, we use a lock:
https://github.com/scicloj/clojisr/blob/master/src/clojisr/v1/impl/rserve/java.clj#L41

But there are some parts of the Session implementation that do not go through that entry point:
https://github.com/scicloj/clojisr/blob/master/src/clojisr/v1/impl/rserve/session.clj#L23

Make R functions reify IFn

Currently, when given an R function, we need to explicitly convert it to a Clojure function using clojuress.v1.r/function.

Practice shows that the distinction between R functions and their Clojure wrappers is confusing and complicates the workflow.

We could make any RObject which is an R function reify the IFn interface, and thus be a Clojure function.

See the discussion of analogous work at Libpython-clj here.

release checks

After commit and before release we have to in the future apply:

clojure style checks (maybe)
clj-kondo linter
lein check
tests (how to apply to tests in notespace?)
run notespace docs generation

Make require-r compatible with the full require spec

At the moment, require-r supports only basic syntax analogous to Clojure's require and Libpython-clj's require-python.

We wish it to become fully compatible with require-like syntax.

Add proper function metadata

Function metadata will allow tooling such as ElDoc help the user know what arguments are expected.

Here again, the experience of Libpython-clj can teach us a lot. See this and related topics.

Printing loops never stop

Every time new session is created, printing loop is started but never stopped. We should refactor it to stop the loop when session is closed.

Fix R symbols/names -> clojure symbols/keywords conversion

We have two places where R symbol conversion is applied.

Named list keys are converted to keywords
R library symbols are converted to Clojure symbols.

Both keys and symbols can be invalid in various ways in Clojure so proper conversion should be made.

Strategy is:

Convert invalid names (keys) to a string
Skip invalid symbols (like "[[<-.factor") or wrap into dedicated function (like brabra)

For both cases we use #"[\Q[](){}#@;:,\/^'~"\E].*"` which catches only first letter as forbidden letter.

Two regexes should be prepared for each cases separately and tested against real names taken from R libraries (mainly base).

Last example in tutorial on different environment fails on R 3.4.4

On my environment deparse returns something different than presented in tutorial, this causes fail of last test.

> deparse(c(a=c(1,2),b="hi!"))
[1] "structure(c(\"1\", \"2\", \"hi!\"), .Names = c(\"a1\", \"a2\", \"b\"))"

Probably my R version is not fresh enough

> version
               _
platform       x86_64-pc-linux-gnu
arch           x86_64
os             linux-gnu
system         x86_64, linux-gnu
status
major          3
minor          4.4
year           2018
month          03
day            15
svn rev        74408
language       R
version.string R version 3.4.4 (2018-03-15)
nickname       Someone to Lean On

Write tests for Renjin backend

ensure path existance before saving chart to file

in applications/plotting namespace we have to add path creation when file is saved to local folder.

Add %>% to symbolic evaulation

Add threading to codegen.

(%>% a b c) -> "a %>% b %>% c"

Simpler behaviour of code generation with simple values

Currently, the ->code function for code generation converts almost all Clojure values (except for symbols and functions) to corresponding RObjects, and puts these objects' temporary names in the generated code.

See the

E.g.:

(r/->code 3.14)
;; => ".MEM$xf63d99caf6de4aa1"
(r/->code false)
;; => ".MEM$xc6df8e9af9d94817"
(r/->code [1 2 3])
;; => ".MEM$x55d70535c65c4328"

For simple primitive values, such as number, strings and booleans, we would rather put those values explicitly in the generated R code.

That is, what we want is this:

(r/->code 3.14)
;; => "3.14"
(r/->code false)
;; => "FALSE"
(r/->code [1 2 3])
;; => ".MEM$x55d70535c65c4328"

This is will require a rather small change at the codegen namespace, and some adapdtation of the tests.

Make logging user-friendly

Currently, the default logging behaviour is too verbose for typical data-analysis work.

The default behaviour should be relatively quiet.

Also, changing the behaviour should be beginner-friendly and clearly documented.

R functions arglist

I prepared POC for adding arglist for RObjects acting as function using formals. It can be helpful for REPL driven development. Arglists conform calling way in clojuress (associative destructuring on variadic position)

Examples:

(formals base/mean)
;; => ([x & {:keys [...]}])
(formals stats/arima0) 
;; => ([x & {:keys [order seasonal xreg include.mean delta transform.pars fixed init method n.cond optim.control]}])
(formals dev/dev-off)
;; => ([& {:keys [which]}])
(formals base/Sys-info)
;; => ([])

formals can be used to create :arglists meta tag.

(I checked and it works on CIDER)

Reify Iterable

Following the good experience of Libpython-clj, what RObject instances should reify Iterable, and how?

Fow example, it might be nice to iterate over the rows of a data frame as Clojure maps.

Support geospatial data conversion

Both R and the JVM have a good story for processing geospatial data.

R has sf, Java has JTS, etc.

Communicating between the two has a good potential for Clojure. For example, it will allow for easy, declarative geospatial visualization with R libraries such as Leaflet, the related Mapview and the new Mapdeck.

This issue suggests to make geospatial data conversion between R and the JVM officially supported by Clojuress.

For most use cases, simple Geojson represetntation is good enough as a way for communication. This could be used in a first version of the implementation.

For a first step of this task, it would be good to write an example workflow with explicit conversion using Geojson, even if it is not yet part of the r->clj and clj->r API functions.

Some experiments have shown this is a practical way, at least for non-huge datasets.

edit: wording

Add a proper API documentation

At the moment, most documentation and testing is done in the tutorials.

We need to see if and how this workflow could live in harmony with a systematic API documentation (e.g., function docstrings, list of API functions), which is now missing.

Preload default packages by default

Preload default packages by default, ie: stats, base, graphics, datasets, grDevices, probably more.