alltherooms / cached-request Goto Github PK

Whats about adding an Option for Custom Cache Keys?

 if(options.customkey) {
	  key = options.customkey;
  } else {
  key = JSON.stringify(this.normalizeOptions(options));
  key = this.hashKey(key);
  }

Using

// test.js
const request = require('request')
const cr = require('./lib/index')(request)
cr.setCacheDirectory('/tmp/cache');

const log = (i) => (err, response, body) => {
  if (err) throw err;
  const cached = !!response.headers["x-from-cache"];
  console.log(`${i}: ${response.statusCode}, ${cached}`);
};

cr({url: 'https://www.google.com', ttl: 10000}, log(1))
cr({url: 'https://www.google.com', ttl: 10000}, log(2))

If I execute it after a non-cached response, these executions sometimes will result in nothing printed in the screen, and only after the cache expires I'm able to see results back again.

$ node test.js
1: 200, false
2: 200, false
$ node test.js
$ node test.js
$ node test.js
$ node test.js
$ node test.js
$ node test.js
$ node test.js
$ node test.js
$ node test.js
$ node test.js
$ node test.js
1: 200, false
2: 200, false
$ node test.js
1: 200, true
2: 200, true

Unfortunately if I use this tool, I sometimes, randomly get this error (it does not happen always)

events.js:160
     throw er; // Unhandled 'error' event
     ^

Error: stream.push() after EOF
   at readableAddChunk (_stream_readable.js:156:17)
   at RequestMiddleware.Readable.push (_stream_readable.js:134:10)
   at IncomingMessage.<anonymous> (...\betrieb\places_zip_geo_mashup\node_modules\cached-request\lib\request-middleware.js:59:12)
   at emitOne (events.js:101:20)
   at IncomingMessage.emit (events.js:188:7)
   at IncomingMessage.Readable.read (_stream_readable.js:381:10)
   at flow (_stream_readable.js:761:34)
   at resume_ (_stream_readable.js:743:3)
   at _combinedTickCallback (internal/process/next_tick.js:74:11)
   at process._tickCallback (internal/process/next_tick.js:98:9)

I am using the library to query data (and cache it) from fixer.io. Fixer.io is rate limited. So I would like to throttle requests to the api but ONLY if the data is not read from cache. Is there any way on hooking up into the case when there is cache miss?

Yes, this is bad API design of theirs, but I just came across such situation an a foreign API that I am using. And this is actually an unexpected error on their side that comes and goes (actually comes quite often). This means that while using cachedRequest (or request too) sometimes I end up inside "success" case, even though the body says that there was an unexpected error.. and as the request becomes cached (haha, as it was 200 response), I cannot escape this error anymore, without deleting the entire cache directory.. not very handy..

Is there any workaround recommendation? Any feature of invalidating the cache for certain request? This would be really useful for me at the moment.. I know I could set ttl to lower value, but that would remove most of the benefits of using this library for me (now my ttl is 24 hours..).

var gRequest = cachedRequest({
            url: photoFetch.data.results,
            ttl: 1000 * 60 * 10 //1 minutes
        }).on("error", function(err) {
            console.log(err);
        });
        gRequest.pipe(res);

Cached file does not seem to be deleting automatically. I am doing it wrong?.

using 1.0.0 I can't seem to get my requests to cache.
I have the following at the start of my file:

var cachedRequest = require('cached-request')(request);
var cacheDirectory = '/tmp/cache';
cachedRequest.setCacheDirectory(cacheDirectory);
cachedRequest.set('ttl', 3000);

When starting my app, I get cachedRequest.set is not a function although this is in the docs. Looking at the object it doesn't indeed have a set method, but does have a setValue docs need to be updated?

Secondly I declare, as seen above, the cacheDirectory but keep getting the following error:

Error: ENOENT: no such file or directory, open '/tmp/cache/1275300535.json'

Looking at the code:
https://github.com/alltherooms/cached-request/blob/master/lib/cached-request.js#L164

It seems that there is no check in place to see if the folder should be created - a simple mkdir:

//Open headers file
  mkdirp(self.cacheDirectory);
  headersReader = fs.createReadStream(this.cacheDirectory + key + '.json');
...
//Open the response file
  mkdirp(self.cacheDirectory);
  responseReader = fs.createReadStream(self.cacheDirectory + key);

would prevent an anti-pattern and make sure that the dir exists.

Hi

I was writing some code to clean up the cached responses after a test and it occurred to me that a flag to 'not save' the response would be useful. Any thoughts?

npm WARN deprecated [email protected]: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible.

Hi!

Great idea, I'm thinking about taking on the missing helper methods (get, etc) but before I started I thought I'd ask - how come the cache isn't in memory? Would it be possible to have that as an option?

I like your module because it looks very simple and does exactly what I want. Though I found out one inconsistency, maybe I missed something.

When I perform the first request which is not cached yet, I get a plain text in the body property of the response object in the callback function.

body: 'Text\n'

Though when I perform the second request for the same url which is already cached, I get a Buffer object at the body property.

body: <Buffer 54 65 78 74 0a>

And same for the body argument of the callback function.

I perform the request I passing only the one parameter url:

cachedRequest({
    url: 'http://localhost',
  }, (err, response, body) => {
  console.log(err, response, body);
});

Occasionally the APIs give back HTML errors, yet I have no way of seemingly telling the cache to refresh. This would be a nice enhancement :)

[15:0:55:379] Message: "Checking min deposite - 863830190"
undefined:0

SyntaxError: Unexpected end of input
at Object.parse (native)
at ReadStream. (/bot2/node_modules/cached-request/lib/cached-request.js:179:35)
at ReadStream.emit (events.js:129:20)
at _stream_readable.js:908:16
at process._tickCallback (node.js:355:11)

Does anyone know what can cause this or how to fix it? It happens sometimes and crashes whole script. Thanks

According to the documentation to se the ttl you do:

cachedRequest.set('ttl', 1000);

This fails with cachedRequest.set is not a function. Looking at the source it would appear the right code would be:

cachedRequest.ttl = 1000;

Is this correct?

Hi,
I'm fetching data from external provider and everything works fine.
From time to time the program terminates with following print:

SyntaxError: Unexpected end of input
at Object.parse (native)
at ReadStream. (/Users/pd/node_modules/cached-request/lib/cached-request.js:189:35)
at emitNone (events.js:72:20)
at ReadStream.emit (events.js:166:7)
at endReadableNT (_stream_readable.js:921:12)
at nextTickCallbackWith2Args (node.js:442:9)
at process._tickCallback (node.js:356:17)

My program:
...

   cachedRequest(
            {
                method: 'POST',
                url: url,
                body: xmlReq,
                headers: {'Content-Type': 'text/xml'},
                ttl: 1200
            },
    function (error, response, body) {
            if (error) {
                console.error("getApiInfo: cachedRequest error: " + error);
                return;
            }
            if (!error && response.statusCode === 200) {
                var lastUpdated = new Date().toLocaleTimeString();
                var stop = Date.now();
                var processTime = (stop - start) / 1000;
                body = JSON.parse(body);
                body.processTime = processTime;
                body.lastUpdated = lastUpdated;
                callback(body, trainId);
            }
    });

...

There's this request-promise-native package, which wraps around request to deliver a Promise interface.
What would be the best approach delivre Promises with cached-request? Somehow, I see them both as mixins in a "class hiearchy", which could be mixed independently.

Thank you for that great module. I have one feature which I think would be usefull.

When the cache-folder does not exist, create it automatically so that the user does not have to do it.
This would especially be helpful when the cache-folder is in gitignore.

Hello, I don't know if it's implemented (at least I don't see it in docs) but I would want to delete the cache before ttl time ends and make a new cached request again.

Anyway to setup with LRU to limit the disk usage?