boblauer / cachegoose Goto Github PK

Is there a way for us to know if it retrieved the data from the database or from the cache?

This would be useful to know.

Also, i have some queries that use aggregate. I notice that if i specify different match criteria (for example query 1 is looking for name: bob and query 2 is looking for name: jim), I get the same results as the first query because they had the same cachekey. How can we get around this? I could make a long cachekey that contains all the attributes of the query, but that's not ideal. And then am i to invalidate the whole cache everytime a new item is added or removed?

Any insights would be appreciated.

Thanks,
Jas

If you are using Mongoose 4.x or below, you need to use version 4.1 of this library.

Hello!

For typescript projects, it would be nice if this library provides its official declaration files.

Here are some guides on how to write them

http://www.typescriptlang.org/docs/handbook/declaration-files/introduction.html

http://blog.wolksoftware.com/contributing-to-definitelytyped

I'm looking at tuning our setup wrt. caching of queries, but I'm missing insights into how cachegoose are affecting things.

Would it be possible to make it emit (or be queried for) hit/miss rates, possibly per-collection?

Ex:

var cachegoose = require('cachegoose');
var redisCacheOptions = { engine: 'redis', host: ..., port: ... };
// skip auth-setup for redis
var cg = cachegoose(mongoose, redisCacheOptions);

setInterval(function getCacheGooseStats() {
    var rates = cg.getHitRates();
    console.log("Cachegoose rates:", rates); // Cachegoose rates: { users: 0.9103, purchases: 0,001, ...}
    cg.resetHitRates(); // OR: Exponential moving averages (like UNIX load)
}, 10 * 1000);

Allow a cache key prefix for auto-generated keys. Probably going to submit a pull request myself for this one.

Is there a way to apply this directly on a mongodb collection?

  const result = await mongoose.connection.db.collection('websites')
    .findOne({
      name,
      _active: true
    })
    .cache(60, 'websiteByName-' + name)

I do this because this is a second, very simple, process to only retrieve data from mongo and I don't want to have the same schema in both projects. The main project does use schemas and caching works perfectly fine there.

Suppose a scenario in which a list is cached for 5 minutes, but before the list expires a new record is inserted. For this case I don't want to get the old list, but a new one with the record in it?
It would be nice to delete all keys belonging to the model where the record was inserted.

Hey,

Thanks for making cachegoose - looks to be (almost - I think) exactly what I'm after!

I wanted to double check I wasn't missing something - I'm attempting to cache the result of a count() operation - e.g. Document.count({ filter }).cache(60).exec()

I've had a skim through the code and I don't believe cachegoose caches count() at the moment? If I can get my head around it, would you be open to a PR that added support for it?

Thanks

Why is it that this needs to have a build when installing? It seems that the files can be rewritten such that a build is not needed at all. I imagine this would make it difficult to integrate in places like Heroku since the build script for this repo doesn't seem run automatically on installation.

Somewhat similar to #25.

Distinct returns an array of ObjectIDs rather than an array of Models, which appears to confuse the caching mechanism. Is there any way to fix it easily?

I wasn't able to run your module with ZEIT NOW 2.0 serverless ending up with the following error:

2019-08-08T18:55:32.063Z	undefined	ERROR	{ Error: Missing required npm module cacheman-memory
    at new CachemanError (/var/task/node_modules/cacheman/node/index.js:55:11)
    at Cacheman.engine (/var/task/node_modules/cacheman/node/index.js:188:19)
    at new Cacheman (/var/task/node_modules/cacheman/node/index.js:152:10)
    at new Cache (/var/task/node_modules/cachegoose/out/cache.js:7:17)
    at module.exports (/var/task/node_modules/cachegoose/out/cache.js:28:10)
    at Object.init [as default] (/var/task/node_modules/cachegoose/out/index.js:11:43)
    at Object.<anonymous> (/var/task/dist/connect/mongoose-connect.js:13:21)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Module.require (internal/modules/cjs/loader.js:690:17)
    at require (internal/modules/cjs/helpers.js:25:18)
    at Object.<anonymous> (/var/task/dist/connect/index.js:6:10)
    at Module._compile (internal/modules/cjs/loader.js:776:30) name: 'Error' }

Any ideas how to fix this?

npm ERR! Darwin 14.1.0
npm ERR! argv "node" "/usr/local/bin/npm" "run" "build"
npm ERR! node v1.0.3
npm ERR! npm v2.2.0
npm ERR! code ELIFECYCLE
npm ERR! [email protected] build: 6to5 lib-src --out-dir lib
npm ERR! Exit status 127
npm ERR!
npm ERR! Failed at the [email protected] build script '6to5 lib-src --out-dir lib'.
npm ERR! This is most likely a problem with the cachegoose package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! 6to5 lib-src --out-dir lib

It is caching result and when i call same query function with change in query still it is giving first cache query result. it is not checking is that current query matches with first query or not.
It should check query and give result as per query change.

.cache(0, 'unique-key-delete-later') is available with two params. I'm requesting for a new param to get un-cached data from request. Like a flag - Signify rather you want to get un-cached data or not.

.cache(0, 'boolean-params') // Ket-name -> isNotCache etc.

I need it a lot in my project.

I don't see this repo supporting findOne currently

Aggregate method requires callback parameter: .exec().then(results => {...}) type promise chain is unsupported.

How come that cached queries return proper data but changing ids ?

https://github.com/davidmarkclements/fast-safe-stringify
https://github.com/boblauer/cachegoose/blob/master/src/generate-key.js#L8

Hi,

Love this module. Works perfectly with Redis. However, I've found that it doesn't when using memory only. Example:

.find({}).cache(0, 'myawesomekey').exec()

Now. if I change any records and execute the above query again the changed records do show up. I was expecting that the "old" (cached) records show. Am I missing something?

Thanks.

Is there a way to pass redis connection string like redis://h:p0700fe183ea2f0fea05183962c0c71b74a72fa9f7@ec2-33-222-22-177.eu-west-1.compute.amazonaws.com:23456 ? cacheman redis could receive this as string instead of options object, but cachegoose is not

Sometimes, it creates the cache key as:
cachecache:cachegoose-cache:
instead of
cache:cachegoose-cache:

because when creating the Cacheman object, it the option.prefix is used for both option._prefix (i.e. cache:cachegoose-cache: ) and the engine (Redis) prefix.

so occasionally, in RedisClient, before issuing command, this.options.prefix is "cache", and it will append that to the agruments, which is "cache:cachegoose".

Not sure how to fix it in cachegoose or in cacheman.

If I dont' specify the .cache() method explicitely, will it be defaulted?

For example, if I have:

var mongoose = require('mongoose');
var cachegoose = require('cachegoose');

cachegoose(mongoose, {
  engine: 'redis',    /* If you don't specify the redis engine,      */
  port: 6379,         /* the query results will be cached in memory. */
  host: 'localhost'
});

Will the following query cache automatically?

Record
  .find({ some_condition: true })
  .exec(function(err, records) {
    ...
  });

It will be great to set time intervals in a human-readable format, ms.js can do it (https://github.com/rauchg/ms.js). Is it really to support it?

That is not an issue is only a question sorry for open that.

Imagine if I want to get and cache multiple querys from same collection lets says User, for one query i use id:10 and I want select only username and email, and for id:10 I want to select other fields address and name, for now I'm creating a custom key with filter id:10 concat with select fields and use a hash of that.

cache(10, 'user'+ id+ hash(criteria, select))

and redis keys are :
"cacheman:cachegoose-cache:user-10-3d7a7cc3c7c227b2dcc75ed62ccd0e166af5a186"
"cacheman:cachegoose-cache:user-10-3d7a7cc3c7c227b2dcc75ed62ccd0e166af5a186"

This two queries are used with differents parts of code and need this different selects, but now for invalidate this object for both objects (with to differents selects querys) but for same id, I want to delete all entries in redis. When I call findAndUpdate for id:10 I want to delete all two entris in redis w

There a way to doing this? like this cachegoose.clearCache('user-10-*');

Since #42 was closed (obviously looks like I failed to respond timely), I wanted to document issues I found with this project.

1. cachegoose uses JSON.stringify versus fast-safe-stringify
2. cacheman-redis uses redis as opposed to ioredis
3. cacheman-redis uses JSON.stringify versus fast-safe-stringify
4. cachegoose uses jsosort, which uses forEach for iterating over the keys of the object, which is incredibly slow compared to a for loop (ref: https://github.com/dg92/Performance-Analysis-JS, but it is widely known that forEach is harmful to performance)
5. The redis command DEL is used as opposed to UNLINK for clearing cache cachegoose.clearCache (ref: https://stackoverflow.com/questions/45818371/is-the-unlink-command-always-better-than-del-command)

Hi again :)

I noticed a bug in my application after implementing cachegoose and I believe it might be because the code to generate the cache key:

const key = {
      model: this.model.modelName,
      op: this.op,
      skip: this.options.skip,
      limit: this.options.limit,
      _options: this._mongooseOptions,
      _conditions: this._conditions,
      _fields: this._fields,
      _path: this._path,
      _distinct: this._distinct
    };

doesn't factor in sort()?

So two queries that are the same, but for the sort order, would generate the same cache key even though the result set would be different?

(async () => {
    const value = new Promise(function(resolve, reject) {
      cachegoose._cache.get("Test", (err, results) => {
        if (err) reject(err);
        resolve(results);
      });
    });
    try{
      console.log(await value);
    }catch (e) {
      console.log(e)
    }
  })(); 

Is there a way to access the cache methods? Basically get & set.

Does this module work with mongoose-q queries?

Would be nice to have the ability to disable caching if result set is empty. Either on the module or the individual query level. What are your thoughts?

What happens when you delete a document? does not specify in the documentation whether it must be deleted manually or does it automatically.

Since #44 was closed, I open this issue.
The data is OK except it generates a new Id for some reason.
Any simple example will show the bug.

Hi,

We use your tool all over. Thank you again for the work.

One question, could we potentially use MongoDB also as a cache engine in addition to Redis? Our experience with Redis cluster is rather sub-par and we would like to setup another MongoDB server for caching.

Hi,
I tried to cache my model on memory, as:
var query = Session.findOne({ Token: token, ... })
.cache(1000000000);
query.exec(function (err, session) {
console.log(query.getCacheKey());//test if cache is working well
....
});
but I get a different cachekey in each query.
is it the accepted behavior?

I was looking at this package to see if I can use it in my project. I would definitively do, however it might be useful to add support for other caching targets, like Memcached or even DynamoDB. This is because I already use the Apollo DynamoDB Cache plugin to cache my GrahQL queries on DynamoDB and it would be nice to have also the MongoDB queries cached there, so I get full access of everything in one place (not mentioning the fact that introducing a Redis server will also add a cost to my existing architecture. Not a big one, but still measurable)

With the latest version of mongoose, they'll tell you to switch to another promise library ( the one they use "mpromise" is decrapricateted ).

It seems Cachehoose doesn' support this currently and when I switched to the ES6 varriant I got ("Can't use ES6 promise with mpromise style constructor").

The mongoose version compatibility check compares strings by type coercion (i.e. by implicitly converting them to decimal numbers).

String-to-Number type coercion shouldn't be used (or at least never for comparing versions) since it fails for double-digit minor versions, e.g.

// Source
if (mongoose.version < '3.7') {
  throw new Error('Cachegoose is only compatible with mongoose 3.7+');
}
// Tests
if ('3.10' < '3.7') // false even though 3.10 is more recent than 3.7
if ('3.7.1' < '3.7') // false even though 3.7.1 is more recent than 3.7

Proposed fix

Use compare-versions

e.g.

const compareVersions = require('compare-versions');

if (compareVersions(mongoose.version, '3.7') < 0) {
  throw new Error('Cachegoose is only compatible with mongoose 3.7+');
}

The docs say passing a null key should clear the cache, it doesn't.

In index.js:
module.exports.clearCache = function(customKey, cb = () => { }) {
// if (!customKey) return cb(); // Change this line to the following to fix.
if (!customKey) return cache.clear(cb());
cache.del(customKey, cb);
};

instead of TCP?

Do you currently invalidate cache automatically when a model changes? For example:

const object = await SomeModel.findOne({ foo: 'bar' }).cache(3600); // query 1
...
object.foo = 'baz';
await object.save();
// when I run query 1 again here, will I get the stale version (i.e. cached)
// or is the cache automatically invalidated?

If not; do you know any easy way to do it by tweaking our models/cachegoose?

Do you guys have any idea why with cachegoose i got the inverse result ?

with cachegoose my result is in 1.2s (always)
without cachegoose my result is in 0.6s

on redis-cli keys *
127.0.0.1:6379> keys *

1. "cacheman:cachegoose-cache:2d2c760092875c462a88cd491ece39e3e6651773"
2. "cacheman:cachegoose-cache:categories"

model.find({}).cache(0, 'categories').exec(function (err, resp) {
    res.send(resp);
});

Thanks

cachegoose(mongoose, { engine: 'redis' });
Leaves test in unterminated state, tests go through but never finish

cachegoose(mongoose);
Working fine

Is there anything to be done about this?

If using .populate(), you must set .lean(), or fields will not be populated.

Without .lean():

Model
  .findOne(id)
  .populate('field')
  .cache(30)
  .exec((err, item) => {
    // item.field is just ObjectId
  })

With .lean():

Model
  .findOne(id)
  .populate('field')
  .lean()
  .cache(30)
  .exec((err, item) => {
    // item.field is { key: value }
  })

Is there any reason why there is no support for caching populated fields without lean?

I'm having an issue with aggregate and this library. The error I get is:

TypeError: Cannot read property 'constructor' of undefined

However, the aggregate return records and with this library all works good. Mind you, I did not cache anything or applied any caching to any finds yet. I simply installed this library. It looks like it hooks into aggregate and expects certain results.

The fsevents dependency was introduced in f2ef792. It breaks Linux and Windows installations.

I have inconsistent results when using $regex in query

Please add @types/cachegoose

We could build it directly into the clearCache method exposed by cachegoose in order for it to work properly.

https://github.com/caiogondim/fast-memoize.js