eddelbuettel / rcppredis Goto Github PK

A daughter issue to #17.

There is no way currently to cleanly close a redis connection. Some combination of CLIENT SETNAME/GETNAME and CLIENT KILL could avoid this. In practice, it probably isn't very important to solve because we tend to build a redis connection and to keep using it.

Minimal, contrived, example:

system("redis-server --port 7777", wait=FALSE)
r <- new(RcppRedis::Redis, "localhost", 7777)
r$ping()
r$exec("SHUTDOWN")

will crash R with

> r$exec("SHUTDOWN")
[15435] 23 Nov 10:54:23.121 # User requested shutdown...
[15435] 23 Nov 10:54:23.122 # Redis is now ready to exit, bye bye...

 *** caught segfault ***
address (nil), cause 'memory not mapped'

Traceback:
 1: .External(list(name = "CppMethod__invoke_notvoid", address = <pointer: 0x2fab870>,     dll = list(name = "Rcpp", path = "/home/rich/lib/R/library/Rcpp/libs/Rcpp.so",         dynamicLookup = TRUE, handle = <pointer: 0x35d8390>,         info = <pointer: 0x7fc132594c40>), numParameters = -1L),     <pointer: 0x271b430>, <pointer: 0x362b4a0>, .pointer, ...)
 2: r$exec("SHUTDOWN")

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace

Seen on various flavours of OSX and Linux. The gdb backtrace is:

(gdb) bt
#0  Redis::extract_reply (this=0xef74a0, reply=0x0) at Redis.cpp:76
#1  0x00007ffff2d54884 in Redis::exec (this=0xef74a0, cmd=...) at Redis.cpp:157
#2  0x00007ffff2d531e3 in Rcpp::CppMethod1<Redis, SEXPREC*, std::string>::operator() (this=this@entry=0x1d6dcd0, object=0xef74a0, args=args@entry=0x7fffffffbde0)
    at /home/rich/lib/R/library/Rcpp/include/Rcpp/module/Module_generated_CppMethod.h:111
#3  0x00007ffff2d5b25f in Rcpp::class_<Redis>::invoke_notvoid (this=<optimised out>, method_xp=<optimised out>, object=0x2007a60, args=0x7fffffffbde0, 
    nargs=<optimised out>) at /home/rich/lib/R/library/Rcpp/include/Rcpp/module/class.h:234
#4  0x00007ffff348b1ec in CppMethod__invoke_notvoid (args=<optimised out>) at Module.cpp:220
#5  0x00007ffff78f1161 in ?? () from /usr/lib/R/lib/libR.so
#6  0x00007ffff79369db in Rf_eval () from /usr/lib/R/lib/libR.so
#7  0x00007ffff7938b60 in ?? () from /usr/lib/R/lib/libR.so
#8  0x00007ffff79367e3 in Rf_eval () from /usr/lib/R/lib/libR.so
#9  0x00007ffff7937b6f in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#10 0x00007ffff79365bf in Rf_eval () from /usr/lib/R/lib/libR.so
#11 0x00007ffff795da32 in Rf_ReplIteration () from /usr/lib/R/lib/libR.so
#12 0x00007ffff795dd81 in ?? () from /usr/lib/R/lib/libR.so
#13 0x00007ffff795de34 in run_Rmainloop () from /usr/lib/R/lib/libR.so
#14 0x00000000004007eb in main ()
#15 0x00007ffff7261ec5 in __libc_start_main (main=0x4007d0 <main>, argc=1, argv=0x7fffffffde78, init=<optimised out>, fini=<optimised out>, 
    rtld_fini=<optimised out>, stack_end=0x7fffffffde68) at libc-start.c:287
#16 0x000000000040081b in _start ()

A fix would be to add something like:

  if (reply == NULL) {
    error("Failure communicating with the Redis server");
  }

after every redisCommand / redisCommandArgv in src/Redis.cpp

While the above is through Redis server shutdown the same thing happens if the connection is lost (e.g. client and server on different machines on different hosts and a loss of the link between). I have confirmed this with toxiproxy.

Redis keeps adding new functions, e.g. PEXPIRE is only available since 2.6.0. We should probably check the version of the Redis server we connect to and disallow calls to functions that aren't available. Preferably, we would throw a warning. Ideally we'd simulate the desired effect. For example, a call to PEXPIRE could just round to the nearest full second and call EXPIRE with a warning.

redis$execv(c("LRANGE mylist 0 elephant")) returns the text error message from Redis but not an error in R. See also, ropensci-archive/RedisAPI#3

That should allow us to retire use of package BH.

The Status statement in README.md terminates abruptly mid-word.

Observed behavior: Sometimes RcppRedis does not obey the timeout and locks the R process.
Desired behavior: RcppRedis obeys timeout and produces error

Minimal example, something like:

library(RcppRedis)

options(redisRAMCache.host = "hostWithRunningRedisServer.com")
options(redisRAMCache.port=6379)
redisRAMCache <- new(Redis, getOption("redisRAMCache.host"), getOption("redisRAMCache.port"),"",2.0)

options(redisREcho.host = "hostWithoutRedisServer.com")
options(redisREcho.port=6379)
try(redisREcho <- new(Redis, getOption("redisREcho.host"),getOption("redisREcho.port"),"",0.5),silent=TRUE)

Except it seems like this edge case only gets hit sometimes and not in a way that is fully reproducable. This is a stub if anyone else encounters similar behavior perhaps they can narrow it down further.

It seems specific to the case where the 2nd server isn't there and may require that the first server have established a connection already.

When the timeout is above 1, I've never been able to replicate the issue.

ticket for imminent PR.

it would be better if redis connection could auto reconnect after fork when redis is used as shared memory for multicore style parallel computing.

It might be possible to process the Redis documentation JSON to create a full set of API functions. We could avoid collision with existing functionality by making all calls all capitals. However, because Cpp has strict typing, it might be a lot easier to write in a way that works by having the code that provides those functions written in R; that might involve encapsulating the RcppRedis constructor in some R code so that we can add functions to the resulting object. All such functions would probably dispatch to execv in the Cpp code.

Hi, I tried to use this package on MacOS. Installed hiredis via brew, but cannot connect.

#R> redis = new (RcppRedis::Redis)
Error in new_CppObject_xp(fields$.module, fields$.pointer, ...) : 
  Redis connection error: Connection refused

Hi,

Attempting a devtools install on Ubuntu 15.10 amd64. Seems to build fine and then barfs during package load test...

devtools::install_github('eddelbuettel/RcppRedis')
Downloading GitHub repo eddelbuettel/RcppRedis@master
Installing RcppRedis
'/usr/lib/R/bin/R' --no-site-file --no-environ --no-save --no-restore CMD INSTALL
'/tmp/RtmpxdQHRv/devtools3f174bdccb2e/eddelbuettel-rcppredis-e625370'
--library='/home/br00t/R/x86_64-pc-linux-gnu-library/3.2' --install-tests

• installing source package ‘RcppRedis’ ...
  checking for g++... g++
  checking whether the C++ compiler works... yes
  checking for C++ compiler default output file name... a.out
  checking for suffix of executables...
  checking whether we are cross compiling... no
  checking for suffix of object files... o
  checking whether we are using the GNU C++ compiler... yes
  checking whether g++ accepts -g... yes
  checking how to run the C++ preprocessor... g++ -E
  checking whether we are using the GNU C++ compiler... (cached) yes
  checking whether g++ accepts -g... (cached) yes
  checking for pkg-config... yes
  checking for grep that handles long lines and -e... /bin/grep
  checking for egrep... /bin/grep -E
  checking for ANSI C header files... yes
  checking for sys/types.h... yes
  checking for sys/stat.h... yes
  checking for stdlib.h... yes
  checking for string.h... yes
  checking for memory.h... yes
  checking for strings.h... yes
  checking for inttypes.h... yes
  checking for stdint.h... yes
  checking for unistd.h... yes
  checking hiredis/hiredis.h usability... yes
  checking hiredis/hiredis.h presence... yes
  checking for hiredis/hiredis.h... yes
  configure: creating ./config.status
  config.status: creating src/Makevars
  Completed configuration and ready to build.
  ** libs
  g++ -I/usr/share/R/include -DNDEBUG -I"/home/br00t/R/x86_64-pc-linux-gnu-library/3.2/Rcpp/include" -I"/home/br00t/R/x86_64-pc-linux-gnu-library/3.2/RApiSerialize/include" -D_FILE_OFFSET_BITS=64 -I/usr/include/hiredis -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g -c Redis.cpp -o Redis.o
  g++ -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o RcppRedis.so Redis.o -lhiredis -L/usr/lib/R/lib -lR
  installing to /home/br00t/R/x86_64-pc-linux-gnu-library/3.2/RcppRedis/libs
  ** R
  ** demo
  ** inst
  ** tests
  ** preparing package for lazy loading
  ** help
  *** installing help indices
  ** building package indices
  ** testing if installed package can be loaded

  *** caught segfault ***
  address 0x20, cause 'memory not mapped'

Traceback:
1: .Call(Module__classes_info, xp)
2: Module(module, mustStart = TRUE, where = env)
3: doTryCatch(return(expr), name, parentenv, handler)
4: tryCatchOne(expr, names, parentenv, handlers[[1L]])
5: tryCatchList(expr, classes, parentenv, handlers)
6: tryCatch(Module(module, mustStart = TRUE, where = env), error = function(e) e)
7: loadModule(module = "Redis", what = TRUE, env = ns, loadNow = TRUE)
8: (function (ns) loadModule(module = "Redis", what = TRUE, env = ns, loadNow = TRUE))()
9: doTryCatch(return(expr), name, parentenv, handler)
10: tryCatchOne(expr, names, parentenv, handlers[[1L]])
11: tryCatchList(expr, classes, parentenv, handlers)
12: tryCatch((function (ns) loadModule(module = "Redis", what = TRUE, env = ns, loadNow = TRUE))(), error = function(e) e)
13: eval(expr, envir, enclos)
14: eval(substitute(tryCatch(FUN(WHERE), error = function(e) e), list(FUN = f, WHERE = where)), where)
15: .doLoadActions(where, attach)
16: methods:::cacheMetaData(ns, TRUE, ns)
17: loadNamespace(package, c(which.lib.loc, lib.loc))
18: doTryCatch(return(expr), name, parentenv, handler)
19: tryCatchOne(expr, names, parentenv, handlers[[1L]])
20: tryCatchList(expr, classes, parentenv, handlers)
21: tryCatch(expr, error = function(e) { call <- conditionCall(e) if (!is.null(call)) { if (identical(call[[1L]], quote(doTryCatch))) call <- sys.call(-4L) dcall <- deparse(call)[1L] prefix <- paste("Error in", dcall, ": ") LONG <- 75L msg <- conditionMessage(e) sm <- strsplit(msg, "\n")[[1L]] w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w") if (is.na(w)) w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L], type = "b") if (w > LONG) prefix <- paste0(prefix, "\n ") } else prefix <- "Error : " msg <- paste0(prefix, conditionMessage(e), "\n") .Internal(seterrmessage(msg[1L])) if (!silent && identical(getOption("show.error.messages"), TRUE)) { cat(msg, file = stderr()) .Internal(printDeferredWarnings()) } invisible(structure(msg, class = "try-error", condition = e))})
22: try({ ns <- loadNamespace(package, c(which.lib.loc, lib.loc)) env <- attachNamespace(ns, pos = pos, deps)})
23: library(pkg_name, lib.loc = lib, character.only = TRUE, logical.return = TRUE)
24: withCallingHandlers(expr, packageStartupMessage = function(c) invokeRestart("muffleMessage"))
25: suppressPackageStartupMessages(library(pkg_name, lib.loc = lib, character.only = TRUE, logical.return = TRUE))
26: doTryCatch(return(expr), name, parentenv, handler)
27: tryCatchOne(expr, names, parentenv, handlers[[1L]])
28: tryCatchList(expr, classes, parentenv, handlers)
29: tryCatch(expr, error = function(e) { call <- conditionCall(e) if (!is.null(call)) { if (identical(call[[1L]], quote(doTryCatch))) call <- sys.call(-4L) dcall <- deparse(call)[1L] prefix <- paste("Error in", dcall, ": ") LONG <- 75L msg <- conditionMessage(e) sm <- strsplit(msg, "\n")[[1L]] w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w") if (is.na(w)) w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L], type = "b") if (w > LONG) prefix <- paste0(prefix, "\n ") } else prefix <- "Error : " msg <- paste0(prefix, conditionMessage(e), "\n") .Internal(seterrmessage(msg[1L])) if (!silent && identical(getOption("show.error.messages"), TRUE)) { cat(msg, file = stderr()) .Internal(printDeferredWarnings()) } invisible(structure(msg, class = "try-error", condition = e))})
30: try(suppressPackageStartupMessages(library(pkg_name, lib.loc = lib, character.only = TRUE, logical.return = TRUE)))
31: tools:::.test_load_package("RcppRedis", "/home/br00t/R/x86_64-pc-linux-gnu-library/3.2")
aborting ...
Segmentation fault (core dumped)
ERROR: loading failed

• removing ‘/home/br00t/R/x86_64-pc-linux-gnu-library/3.2/RcppRedis’
• restoring previous ‘/home/br00t/R/x86_64-pc-linux-gnu-library/3.2/RcppRedis’
  Error: Command failed (1)

sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 15.10

locale:
[1] LC_CTYPE=en_CA.UTF-8 LC_NUMERIC=C LC_TIME=en_CA.UTF-8
[4] LC_COLLATE=en_CA.UTF-8 LC_MONETARY=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
[7] LC_PAPER=en_CA.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

loaded via a namespace (and not attached):
[1] tools_3.2.2

I have coded a shiny app that runs perfectly locally but when I try to deploy it to shinyapps.io I get the following output:

Preparing to deploy application...DONE
Uploading bundle for application: 115430...DONE
Deploying bundle: 549719 for application: 115430 ...
Waiting for task: 242805727
  building: Parsing manifest
  building: Building image: 540950
  building: Fetching packages
  building: Building package: RcppRedis
################################ Begin Task Log ################################ 
[2016-09-06T09:42:13.614224848+0000] Installing R package: RApiSerialize (0.1.0)
* installing to library ‘/usr/local/lib/R/site-library’
* installing *binary* package ‘RApiSerialize’ ...
* DONE (RApiSerialize)
(repeated 22 times for all other required packages)
[2016-09-06T09:42:37.199855491+0000] Building R package: RcppRedis (0.1.7)
/mnt/packages/build /mnt
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘RcppRedis’ ...
** package ‘RcppRedis’ successfully unpacked and MD5 sums checked
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking how to run the C++ preprocessor... g++ -E
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether g++ accepts -g... (cached) yes
checking for pkg-config... yes
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking hiredis/hiredis.h usability... no
checking hiredis/hiredis.h presence... no
checking for hiredis/hiredis.h... no
configure: WARNING: Hiredis headers not found with via default CXXFLAGS and CPPFLAGS, trying /usr/local/include
configure: WARNING: Unsetting ac_cv_header_hiredis_hiredis_h
checking hiredis/hiredis.h usability... no
checking hiredis/hiredis.h presence... no
checking for hiredis/hiredis.h... no
configure: error: ERROR: hiredis headers required; use '-Iincludedir' in CXXFLAGS for unusual locations.
ERROR: configuration failed for package ‘RcppRedis’
* removing ‘/usr/local/lib/R/site-library/RcppRedis’
################################# End Task Log ################################# 
Error: Unhandled Exception: Child Task 242805728 failed: Error building image: Error building RcppRedis (0.1.7). Build exited with non-zero status: 1

Any ideas on what could be causing this and how to fix it? Any help would be much appreciated...

Hi,
I was trying to install it using devtools
library("devtools")
install()

but I get the following error message :
[...[
/usr/bin/ld: RcppExports.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
RcppExports.o: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make: *** [RcppRedis.so] Error 1
ERROR: compilation failed for package ‘RcppRedis’

• removing ‘/tmp/RtmpDEzN2n/devtools_install_20031bc9c839/RcppRedis’
  Erreur : Command failed (1)

Thanks

@bwlewis tells me that 0.1.4 can be subject to a seg.fault issue fixed in 1.0.0 upstream.

While that sounds promising, two caveats:

• one can usually trust Debian for update cadences and its hiredis is still at 0.1.4 everywhere but experimental
• we have not seen the segmentation fault, i.e. no minimally complete verifiable example to test pre / post

On the other hand, per #42 and thanks to @opencpu we have been using 1.0.0 for windows builds. So upgrade should be painless.

In other words make it .listen() and also drop the help page? Is it useful for users or just used inside redisMonitorChannels() ?

One problem I rediscovered with #11 is that when an auth provided password is provided, then is echoed to the console. In redisConnect I handled that by wrapping the constructor in base::capture.output(), but that isn't applicable now. Any ideas on how to suppress it?

... would be nice to have. Maybe even conditionally if a msgpack-c header is found.

The version packaged with Ubuntu seems woefully behind though. May need RcppMsgPack just to package the headers.