Support for Sparse Matrices about reticulate HOT 9 CLOSED

@jjallaire I see this is closed, but FYI indeed Matrix does have a C API -- it uses the SuiteSparse package by Tim Davis. You need to include at least "Matrix.h" and "Matrix_stubs.c". See https://github.com/bwlewis/irlba/blob/master/src/irlb.c for an example of a package linking to the Matrix package (and there are others that do this too). I think https://github.com/scikit-sparse/scikit-sparse uses the same cholmod subset of suite sparse that R (and Matlab) does. I'm not sure what the rest of scipy uses for sparse matrices.

See this ancient pre-reticulate fork of the tensorflow package for an example that supports the most common sparse matrix object between R/Python (dgCMatrix/scipy.sparse.csc.csc_matrix):

https://github.com/bwlewis/python/tree/1e92aec80fd29fdf8e3f38e894d3a600e25b5aae

A crude/old but working example of sparse martrices.

from reticulate.

@eddelbuettel yeah, in the case of Matrix we have available a curated subset of SuiteSparse that we can simply directly link to. No need to embed again, just link to Matrix.

from reticulate.

@bwlewis Do you know whether the Matrix package exposes a C API for sparse matrices?

from reticulate.

I don't think it does. The best short-term bet may be to get to the functions from Matrix via Rcpp.

A Google Summer of Code student is currently putting more sparse matrix support into RcppArmadillo.

from reticulate.

What we would need to do is somehow get our hands on the underlying memory and then create a SciPy sparse matrix from it. That said, I don't think Keras currently accepts SciPy sparse matrices (discussion I've seen about the issue seems to center on converting each batch to dense array before feeding it into Keras).

@gsimchoni Perhaps you could write your own training loop (via train_on_batch where you go from sparse to dense as required in each iteration?

from reticulate.

Thank you @jjallaire , this is exactly the advice I got from @dfalbel in the original issue.

from reticulate.

@bwlewis thanks for the SuiteSparse reminder. That requires external linking, or as Matrix does, embedding of SuiteSparse. May make for a good test case for RcppArmadillo too.

from reticulate.

@bwlewis Yep, which is what lme4 has done all those years -- that was always "exhibit one" of how to access code from another package which provides a cleanly exported API.

from reticulate.

Re-capping, it looks like Keras doesn't currently support Sparse matrices (they just suggest that you use train_on_batch. It also looks like Sparse matrix support is in SciPy rather than NumPy, meaning that even if we can get ahold of R's memory there isn't anywhere in NumPy to put it.

from reticulate.