I was wondering whether a WIP repo shouldn't better live in ropenscilabs?

• its confusing knowing how to deal with expiry time. if you include expire in 1st request, but not in the 2nd exact same request, does the expiry setting from the 1st apply to the 2nd one?

• we need a way to simply wrap a package/script fxn in a webmiddens fxn instead of having to make the user modify the internals of their code , eg.,

# some fxn that does an http request
some_fxn <- function(x) {
  some_http_request(x)
}

# simply wrap it in a webmiddens fxn, with settings with a separte fxn perhaps
library(webmiddens)
web_settings(dir = "foo/bar", expire = 3)
cache(some_fxn(...))
# and can override session level settings
cache(some_fxn(...), expire = 5)

brought up by maëlle, ropensci/rcrossref#202 (comment)

Doing this with env vars might be tricky, but easier with a function call.

• in memory: an obvious choice needed
• on disk: should be default for the majority use case of using this in other pkgs where you want users to have responses cached ACROSS R sessions (in memory no good in this case)

via ropensci/rcrossref#185

• turn all caching on or off (WEBMIDDENS_TURN_OFF), takes logical
• set the expiry time via env var (WEBMIDDENS_EXPIRY_SEC), takes integer

...

the downside, vs. memoise, is that we're only caching the http response, so if there's still a lot of time spent processing the response, then the function will still be quite slow

But you could still memoise that response processing part if you make it a function. 😉

right now the only method we have is time based: number of seconds since the request was recorded.

perhaps we can use response headers, following https://github.com/sourcelevel/faraday-http-cache :

• cache-control: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
• age: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Age
• expires: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Expires
• last-modified: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified
  • if-modified-since: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since
  • if-unmodified-since: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Unmodified-Since
  • if-none-match: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match
• e-tag: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag

p.s.

• pragma: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Pragma (only recommended for backwards compatibility with http 1.0 clients)

Seems that the folowing are often found together:

• cache-control and age
• last-modified, if-modified-since, if-unmodified-since
• etag

• canonical ref perhaps: rfc2616 https://tools.ietf.org/html/rfc2616#page-108
• how httr gathers caching info https://github.com/r-lib/httr/blob/master/R/cache.R - not used in normal http requests in the pkg though

so that the user called function can be done more quickly

Right now, we need to wrap HTTP call in midden$call()

Maybe we keep that as an option, to wrap http call in a block, but easier to use would be a configuration setting function call - then just do http requests as normal. eg.,

library(webmiddens)
# 1 hr expiry
midden$new(path = "foobar", expire = 3600)

# then http requests are using webmiddens
crul::HttpClient$new("https://httpbin.org")$get("get", query = list(foo = "bar"))

Not sure this is possible. We'd have to have a specific hook within crul for this

Hi @sckott ,

Thanks for this great package, I started using and wanted to know if the following CRAN policy also apply to webmiddens or any other package to cache http requests:

Packages should not write in the user’s home filespace (including clipboards), nor anywhere else on the file system apart from the R session’s temporary directory (or during installation in the location pointed to by TMPDIR: and such usage should be cleaned up). Installing into the system’s R installation (e.g., scripts to its bin directory) is not allowed.

Limited exceptions may be allowed in interactive sessions if the package obtains confirmation from the user.

Thanks again

cc @hrbrmstr

see ropensci/rcrossref#185

right now, the user sets the cache path with wm_configuration("foo2") which sets the last part of the path, e.g., /home/jane/.cache/R/foo2

could we have wm_configuration() put the package name in the cache path too, so

/home/jane/.cache/R/rcrossref/foo2 instead of /home/jane/.cache/R/foo2

see ropensci/rcrossref#185 (comment)

perhaps something to do with where the mdenv environment is instantiated, perhaps put in onload, or somewhere else?

right now the only way expire time works is across all stubbed/cached requests, that is, if an expire time is set, then on the next http request using webmiddens, all stubs are deleted that are expired

i imagine more fine grained control may be useful:

• set expire time for all http requests matching a given http request pattern - i guess via any of the webmockr options, e.g., a url pattern, or a header pattern, or a body pattern
• a user of a package foobar may want to set different expire times for each function of a package they're using, e.g. 10 sec for fxn1(), 1 minute for fxn2(), and 1 week for fxn3()
• what else?