Code Monkey home page Code Monkey logo

redis-oplog's Introduction

Introducing BlueLibs

Welcome to Redis Oplog

LICENSE: MIT

Backers on Open Collective Sponsors on Open Collective

RedisOplog

A full re-implementation of the Meteor's MongoDB oplog tailing. This time, reactivity is controlled by the app, opening a new world into building reactive applications, highly scalable chat apps, games, and added reactivity for non-persistent data.

Incrementally adoptable & works with your current Meteor project.

Installation

meteor add cultofcoders:redis-oplog
meteor add disable-oplog

Configure it via Meteor settings:

// settings.json
{
    ...
    "redisOplog": {}
}

// default full configuration
{
  ...
  "redisOplog": {
    "redis": {
      "port": 6379, // Redis port
      "host": "127.0.0.1" // Redis host
    },
    "retryIntervalMs": 10000, // Retries in 10 seconds to reconnect to redis if the connection failed
    "mutationDefaults": {
        "optimistic": true, // Does not do a sync processing on the diffs. But it works by default with client-side mutations.
        "pushToRedis": true // Pushes to redis the changes by default
    },
    "debug": false, // Will show timestamp and activity of redis-oplog.
  }
}

To see what you can configure under "redis": {} take a look here: https://www.npmjs.com/package/redis#options-object-properties

meteor run --settings settings.json

Notes

RedisOplog is fully backwards compatible, so there won't be any change in how you use Meteor, unless you want to fine-tune your application for absolute performance.

To make sure it is compatible with other packages which extend the Mongo.Collection methods, make sure you go to .meteor/packages and put cultofcoders:redis-oplog as the first option.

RedisOplog does not work with insecure package, which is used for bootstrapping your app.

Stats

If you are interested in viewing how many observers are registered or memory consumption:

meteor shell
import { RedisOplog } from 'meteor/cultofcoders:redis-oplog';

// works only server-side
RedisOplog.stats()

The levels of scaling reactivity

  1. Just add RedisOplog, you will already see big performance improvements
  2. Fine-tune your reactivity by using custom namespaces and channels
  3. Implement your own custom reactivity by using Redis Vent

Events for Meteor (+ Redis Oplog, Grapher and GraphQL/Apollo)

  • Meteor Night 2018 Slide: Arguments for Meteor - Theodor Diaconu, CEO of Cult of Coders: β€œRedis Oplog, Grapher, and Apollo Live.

If you are using Optimistic UI (Latency Compensation) in your application, you should give this a read.

Find out what RedisOplog does behind the scenes

Find out how you can use the advantages of Redis Oplog to make your app very performant.

Find out how you can hook into redis events to customize, when it fails.

Find out how you can customize your reactivity and enable it across multiple languages/microservices with ease.

If you have different workers/services that perform updates to mongo and they exist outside Meteor, you can still trigger reactivity for the Meteor instances with a few lines of code.

Premium Support

If you are looking to scale your business using this package and you need to have your back covered. We are here to help. Feel free to contact us at [email protected].

Contributors

This project exists thanks to all the people who contribute. [Contribute].

Backers

Thank you to all our backers! πŸ™ [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

redis-oplog's People

Contributors

asood123 avatar benweissmann avatar bohan0 avatar brianlukoff avatar claudiuroman avatar donstephan avatar emaciel10 avatar ericbirdsall avatar evolross avatar fantostisch avatar floriferous avatar guncebektas avatar jamesgibson14 avatar jankapunkt avatar martineboh avatar maxnowack avatar maxpain avatar megawebmaster avatar mfen avatar nathan-muir avatar ramezrafla avatar simonsimcity avatar storytellercz avatar theodordiaconu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

redis-oplog's Issues

Using fields in a publication and making an update to a nested item on it does not properly update the client-side store.

Can I get clarification on this part in the readme?

Warning! If your publication contains "fields" options.

{
    fields: { text: 1, typing: 1 }
}
Even if the fields don't actually exist in the db. The reason we do it like this, is to allow control over access in the synthetic events. For some people you may want to see them, others you do not, depending on their role.

I have a publication with a cursor such as:

Meteor.publish('user.roles', function() {
  if (!this.userId) {
    throw new Meteor.Error('notLoggedIn')
  }
  return Meteor.users.find(this.userId, {fields: {roles: 1}})
})

When I replace my Meteor.publish with Meteor.publishWithRedis, I seem to be getting errors in my app.

Client-side updates are not pushed to Redis.

Calls to insert, update and remove methods on client-side collections doesn't seem to be published reactively...

As good practise, we've been wrapping the calls to client-side collection methods in Meteor methods for some time now. I guess (still not tested it) in those cases everything would work fine because the collection method would eventually be called during the server-side execution of the Meteor method (I wish they would have found another name for that, so confusing).

But we have a very big and old codebase and we still have a lot of calls to client-side collection methods directly in inappropriate places like template event handlers (sorry I feel sick just saying that, I hope your ears are not bleeding) and refactoring the whole thing is not necessarily realistic at this point...

@theodorDiaconu Any thoughts on that?

Expiration date for oplog data

We don't want to clutter our database with lots of oplog data that remains in our history forever, thus costing us money for storage and slowing other queries down.

It's not a priority right now, but it would be great if we could add an expiration to oplog data (i.e. 24 hours, a week, or a month)

Publish Composite replacement with redis oplog observer

  1. Create an observerCollection for the main element
  2. On added/changed/remove recompute the subsequent publications of the next elements in order to update them

Offer ability to an ObservableCollection to "modifyCursor" & reprocess the query. Make performant diffing.

Implement namespace reactivity, and ability to disable reactivity

I want to fine-tune or even disable reactivity for my db updates:

Collection.insert(data, { namespace: 'xxx' }) // same for .update, .upsert, .remove
Meteor.publish('xxx', function () {
       return cursor;
}, { namespace: 'xxx' }) // will only listen for changes on the collection on that namespace

{ pushToRedis: false } // will disable reactivity

Critical - Cache subscribers listening to the same channels.

Given 10 users, that subscribe to users in different manners.
We would have 10 redis listeners, and for each modification on users, we would get that modification 10 times, then we need to parse it 10 times. Parsing may be heavy since we use EJSON to serialize/deserialize messages pushed to redis. And also, we will have 9 unecessary redis subscriptions, which occupy network bandwidth and CPU.

Create a subscription cacher similar to the one for publications.

RedisSubscriptionStore - "channel", apply same principles, with the observers, if the last one is leaving stop listening.

RedisListener - will do something like

RedisSubscriptionStore.attach(this, channel, function(channel, data) {
})
RedisSubscription.detach(this)

Changed parameter order on update method

As mentioned in #41, the default parameter order of the update method has changed from ( ... , options, callback) to ( ... , callback, options). In the default order, the callback is always the last parameter. I haven't tested it yet, but passing custom options (like multi: true) to an update method shouldn't work at this moment

Client collection field incorrect after $pull operation in server method

Say we have a very basic publication:

Meteor.publish('tasks', function tasksPublication() {
    return Tasks.find({});
  });

The client subscribes to it:

Meteor.subscribe('tasks');

We perform an insert into a collection within a server method (in imports/server/ so not run on the client at all):

Tasks.insert({ testField: ['A', 'B', 'C', 'D'] });

We look at the collection on the client and the testField contains ['A', 'B', 'C', 'D'] which is exactly as expected.

Then in another method (again, on the server only) we use the $pull operator as follows:

Tasks.update(taskId, { $pull: { testField: 'A' } });

The client collection testField then contains ['B', 'C', 'D', 'D'] which is incorrect. However the Redis log for the operation shows the correct fields...

1479471336.454444 [5 127.0.0.1:53646] "publish" "tasks" "{\"e\":\"u\",\"f\":[\"testField\"],\"d\":{\"_id\":\"4iKFxnhjCSWxTKjKg\",\"testField\":[\"B\",\"C\",\"D\"]}}"
1479471336.454481 [5 127.0.0.1:53646] "publish" "tasks::4iKFxnhjCSWxTKjKg" "{\"e\":\"u\",\"f\":[\"testField\"],\"d\":{\"_id\":\"4iKFxnhjCSWxTKjKg\",\"testField\":[\"B\",\"C\",\"D\"]}}"

Let me know if you need any more tests or info.

Looks like error with matb33:collection-hooks in 1.0.14

I20161129-07:14:00.155(-7)? Exception in defer callback: TypeError: Cannot convert undefined or null to object
I20161129-07:14:00.158(-7)? at keys (native)
I20161129-07:14:00.160(-7)? at Function.keys (/home/james/.meteor/packages/es5-shim/.4.6.15.5j45wu++os+web.browser+w
eb.cordova/npm/node_modules/es5-shim/es5-shim.js:1103:24)
I20161129-07:14:00.161(-7)? at DotObject.dot (/home/james/.meteor/packages/cultofcoders_redis-oplog/.1.0.14.s1f2xl++
os+web.browser+web.cordova/npm/node_modules/dot-object/index.js:440:10)
I20161129-07:14:00.162(-7)? at Function.dot (/home/james/.meteor/packages/cultofcoders_redis-oplog/.1.0.14.s1f2xl++o
s+web.browser+web.cordova/npm/node_modules/dot-object/index.js:62:31)
I20161129-07:14:00.164(-7)? at SmartObject.getDotObject (packages/cultofcoders:redis-oplog/lib/utils/SmartObject.js:
39:34)
I20161129-07:14:00.165(-7)? at packages/cultofcoders:redis-oplog/lib/mongo/collection.extension.js:55:77
I20161129-07:14:00.166(-7)? at [object Object]._.extend.withValue (packages/meteor.js:1122:17)
I20161129-07:14:00.172(-7)? at packages/meteor.js:445:45
I20161129-07:14:00.173(-7)? at runWithEnvironment (packages/meteor.js:1176:24)
I20161129-07:14:00.178(-7)? Exception in callback of async function: TypeError: callback is not a function
I20161129-07:14:00.178(-7)? at packages/mongo/collection.js:651:5
I20161129-07:14:00.179(-7)? at args.(anonymous function) (packages/matb33_collection-hooks.js:391:23)
I20161129-07:14:00.180(-7)? at runWithEnvironment (packages/meteor.js:1176:24)
I20161129-07:14:00.180(-7)? at packages/meteor.js:1189:14
I20161129-07:14:00.181(-7)? at packages/mongo/mongo_driver.js:328:7
I20161129-07:14:00.181(-7)? at runWithEnvironment (packages/meteor.js:1176:24)

Clear Fields Function

Given an object and an array of fields. Return a filtered object containing only those fields MongoDB like.
Treat cases for fields like "profile.firstName" and deep nesting.

Not all fields being pushed to client after updating a nested document

I have a collection which contains numerous fields including an array of nested documents comprising of 2 fields, the schema for the nested doc is as follows:

  'bom.$.stockId': {
    type: String,
    regEx: SimpleSchema.RegEx.Id,
  },
  'bom.$.quantity': {
    type: Number,
  },

When performing an update to a single field (the quantity) in the nested document using a Meteor method, the stockId field is not transmitted back to the client when using Redis Oplog.

Here is a screenshot of the DDP messages when using Redis Oplog:

image

And here without Redis Oplog:

image

The publication code (removed validation etc to keep it relevant and concise):

Meteor.publish('Stock.BOM.List', function (stockId, bomList) {
  const fields = {
    _id: 1,
    partNo: 1,
    title: 1,
    location: 1,
    warehouseId: 1,
  };

  return Stock.find({ _id: { $in: bomList }, organisationId }, { fields });
});

The collection update within the method is:

Stock.update({ _id: stockId, organisationId, 'bom.stockId': bomFormValues.chosenStockItem, }, { $set: { 'bom.$.quantity': bomFormValues.quantity, }, });

It seems as though the first nested field does not get transmitted. I have a slightly more advanced array of nested document with 4 fields that does the same thing (i.e 3 get transmitted and 1 is missing).

Let me know if you need more info, thanks!

Large scale improvement for load-mores|limit|skips

Given a page with "Posts" and when you scroll down, they load more to infinity.. We need to treat this somehow.

  1. Idea of caching client-side
  2. Idea of caching server-side and sharing with other publications.

Think!

Maybe both should be done (?) Depending on use-case.

New Processing Strategy ideas: Amnesia

So, currently we have different strategies that maintain a snapshot of the store in memory of the server.

Now, if we have a chat or something between to users, we don't want to store the snapshot server-side at all, so basically it will only keep the active filters, to see if it matches your query.

On insert checks for filter matching and does .added()
On update checks eligibility by db and does .changed() // if it exists
On remove ... // apply remove

This will work fine if changed(id) or removed(id) will not trigger error if they have nothing to remove/change.

https://github.com/peerlibrary/meteor-control-mergebox

Some errors after implementation

I've implemented redis-oplog and I'm seeing some errors in my console.

I20161115-11:46:16.823(1)? Exception in defer callback: MongoError: Positional projection 'users.$.statusSince' does not match the query document.
I20161115-11:46:16.824(1)?     at Object.Future.wait (/Users/maxnowack/.meteor/packages/meteor-tool/.1.4.2.867yju++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:449:15)
I20161115-11:46:16.826(1)?     at SynchronousCursor._nextObject (packages/mongo/mongo_driver.js:1024:47)
I20161115-11:46:16.828(1)?     at SynchronousCursor.forEach (packages/mongo/mongo_driver.js:1058:22)
I20161115-11:46:16.830(1)?     at SynchronousCursor.map (packages/mongo/mongo_driver.js:1068:10)
I20161115-11:46:16.831(1)?     at SynchronousCursor.fetch (packages/mongo/mongo_driver.js:1092:17)
I20161115-11:46:16.834(1)?     at Cursor.(anonymous function) [as fetch] (packages/mongo/mongo_driver.js:907:44)
I20161115-11:46:16.835(1)?     at packages/cultofcoders:redis-oplog/lib/mongo/collection.extension.js:82:20
I20161115-11:46:16.836(1)?     at [object Object]._.extend.withValue (packages/meteor.js:1122:17)
I20161115-11:46:16.838(1)?     at packages/meteor.js:445:45
I20161115-11:46:16.840(1)?     at runWithEnvironment (packages/meteor.js:1176:24)
I20161115-11:46:16.841(1)?     - - - - -
I20161115-11:46:16.842(1)?     at Function.MongoError.create (/Users/maxnowack/.meteor/packages/npm-mongo/.2.2.11_2.13qd484++os+web.browser+web.cordova/npm/node_modules/mongodb-core/lib/error.js:31:11)
I20161115-11:46:16.844(1)?     at queryCallback (/Users/maxnowack/.meteor/packages/npm-mongo/.2.2.11_2.13qd484++os+web.browser+web.cordova/npm/node_modules/mongodb-core/lib/cursor.js:213:36)
I20161115-11:46:16.846(1)?     at /Users/maxnowack/.meteor/packages/npm-mongo/.2.2.11_2.13qd484++os+web.browser+web.cordova/npm/node_modules/mongodb-core/lib/connection/pool.js:455:18
I20161115-11:46:16.847(1)?     at nextTickCallbackWith0Args (node.js:420:9)
I20161115-11:46:16.848(1)?     at process._tickCallback (node.js:349:13)
I20161115-11:46:17.836(1)? Exception in queued task: TypeError: observableCollection.update is not a function
I20161115-11:46:17.837(1)?     at handleUpdate (packages/cultofcoders:redis-oplog/lib/processors/direct.js:26:26)
I20161115-11:46:17.853(1)?     at module.export.exports.default (packages/cultofcoders:redis-oplog/lib/processors/direct.js:11:13)
I20161115-11:46:17.855(1)?     at Object.task (packages/cultofcoders:redis-oplog/lib/redis/RedisSubscriber.js:59:28)
I20161115-11:46:17.856(1)?     at [object Object]._.extend._run (packages/meteor.js:807:18)
I20161115-11:46:17.856(1)?     at packages/meteor.js:785:14
I20161115-11:46:18.960(1)? Exception from sub notificationsCount id ejLC7rcfg4SgfnanB TypeError: Cannot read property '_cursorDescription' of undefined
I20161115-11:46:18.960(1)?     at packages/cultofcoders:redis-oplog/lib/utils/shouldPublicationBeWithPolling.js:11:29
I20161115-11:46:18.961(1)?     at Array.forEach (native)
I20161115-11:46:18.962(1)?     at module.export.exports.default (packages/cultofcoders:redis-oplog/lib/utils/shouldPublicationBeWithPolling.js:6:13)
I20161115-11:46:18.962(1)?     at Subscription.<anonymous> (packages/cultofcoders:redis-oplog/lib/publishWithRedis.js:41:13)
I20161115-11:46:18.963(1)?     at packages/matb33_collection-hooks.js:307:21
I20161115-11:46:18.963(1)?     at [object Object]._.extend.withValue (packages/meteor.js:1122:17)
I20161115-11:46:18.964(1)?     at Subscription._handler (packages/matb33_collection-hooks.js:306:28)
I20161115-11:46:18.964(1)?     at packages/check.js:130:16
I20161115-11:46:18.965(1)?     at [object Object]._.extend.withValue (packages/meteor.js:1122:17)
I20161115-11:46:18.965(1)?     at Object.exports.Match._failIfArgumentsAreNotAllChecked (packages/check.js:129:41)

For the first two I have no idea where they come from. The last one may have something todo with a publish function, that returns this.ready() instead of a cursor.

Error with meteorhacks:kadira?

I just go this error I don't think I changed anything except maybe the update to 1.0.13, any ideas?

I20161128-12:03:41.641(-7)? [PublicationEntry] Performing initial add for observer
I20161128-12:03:41.643(-7)? [RedisSubscriptionManager] Subscribing to channel: __dummy_coll_kshBrXjZse2XcH2MS
I20161128-12:03:41.785(-7)? [PublicationEntry] Performing initial add for observer
I20161128-12:03:41.795(-7)? [RedisSubscriptionManager] Unsubscribing from channel: __dummy_coll_kshBrXjZse2XcH2MS
I20161128-12:03:41.798(-7)? [RedisSubscriptionManager] Subscribing to channel: __dummy_coll_kshBrXjZse2XcH2MS
I20161128-12:03:41.926(-7)? [PublicationEntry] Performing initial add for observer
I20161128-12:03:41.927(-7)? [RedisSubscriptionManager] Unsubscribing from channel: __dummy_coll_kshBrXjZse2XcH2MS
I20161128-12:03:41.928(-7)? [RedisSubscriptionManager] Subscribing to channel: __dummy_coll_kshBrXjZse2XcH2MS
W20161128-12:03:42.066(-7)? (STDERR) /home/james/.meteor/packages/meteor-tool/.1.4.2_3.ochefg++os.linux.x86_64+web.brows
er+web.cordova/mt-os.linux.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:280
W20161128-12:03:42.067(-7)? (STDERR) throw(ex);
W20161128-12:03:42.068(-7)? (STDERR) ^
W20161128-12:03:42.068(-7)? (STDERR)
W20161128-12:03:42.068(-7)? (STDERR) TypeError: Cannot read property 'prototype' of undefined
W20161128-12:03:42.069(-7)? (STDERR) at setLabels (packages/meteorhacks_kadira.js:3683:45)
W20161128-12:03:42.069(-7)? (STDERR) at packages/meteorhacks_kadira.js:3194:5
W20161128-12:03:42.069(-7)? (STDERR) at runWithAFiber (packages/meteorhacks_meteorx.js:192:5)
W20161128-12:03:42.070(-7)? (STDERR) at packages/meteorhacks_meteorx.js:185:5
W20161128-12:03:42.070(-7)? (STDERR) at Array.forEach (native)
W20161128-12:03:42.071(-7)? (STDERR) at packages/meteorhacks_meteorx.js:184:27
W20161128-12:03:42.072(-7)? (STDERR) at Function.time (/home/james/argus/.meteor/local/build/programs/server/profile
.js:301:28)
W20161128-12:03:42.072(-7)? (STDERR) at /home/james/argus/.meteor/local/build/programs/server/boot.js:304:13
W20161128-12:03:42.072(-7)? (STDERR) at /home/james/argus/.meteor/local/build/programs/server/boot.js:345:5
W20161128-12:03:42.073(-7)? (STDERR) at Function.run (/home/james/argus/.meteor/local/build/programs/server/profile.
js:480:12)
W20161128-12:03:42.073(-7)? (STDERR) at /home/james/argus/.meteor/local/build/programs/server/boot.js:343:11

Publish to npm instead of atmosphere

This would probably resolve #35
We could rely on meteor globals with this pattern:

function getMeteorGlobal(packageName, globalName) {
  if (!global.Package || !global.Package[packageName]) return
  return global.Package[packageName][globalName]
}

const Mongo = getMeteorGlobal('mongo', 'Mongo') // from core package
const RedisOplog = getMeteorGlobal('cultofcoders:redis-oplog', 'RedisOplog') // from atmosphere package

Override Mongo.Collection.prototype

Modify the prototype so:

Collection.update(selector, modifier)
->
makes the update -> sends changes for each element individually to redis

Collection.update(selector, modifier, callback)
->
hooks into callback and works as it should, original callback will be started after redis push

RedisOplog not initialized

Hi,

I've followed the usage instructions but publishWithRedis throws an error saying "RedisOplog is not initialized".
I've created a redis.js file at /imports/startup/server/redis.js and also tried to add the code directly into the main.js file of the core server folder. I'm using Meteor 1.4.2.1.

Wrong fields published on update

There is a bug in the handleUpdate method of the default processing strategy:

// …
if (observableCollection.contains(doc._id)) {
  observableCollection.change(doc._id, doc);
} else {
  observableCollection.add(doc);
}
// …

if the update gets called and the document isn't already in the observable collection, it will be added to it. But only with the fields, that were updated.
This bug is totally messing up my published documents and I'm not sure how to fix it.
Any ideas @theodorDiaconu?

I'm working on a test case for this

Publish Composite Implementation

Plan:

  • Create a reader from publishComposite {find, children} to create a Graph with PubNode that has the following properties:

  • root (null if top parent)

  • children

  • selector

  • recomputeSelector()

  • getKey() - creates a unique identifier for this publication.

  • Implement a waterfall strategy, in which if an element is updated within an observable collection, recompute the selector only if it changed.

  • Make it integrated with publication sharing

  • Implement namespacing and channels for each level

Can't get SyntheticMutation to work

We have a publication with a custom channel and it's working well without SyntheticMutation, on both server and client.

When we replace this:
self.collection.update({_id:to},{$push:{messages:message}},{channel:channel});

With either
SyntheticMutation(channel).update({_id:to},{$push:{messages:message}})
Or
SyntheticMutation(self.collection.update).update({_id:to},{$push:{messages:message}},{channel:channel})

The client doesn't get the data anymore.

Keep-alive subscription

For db-heavy subscriptions, sometimes it's better to use more ram.

When publishing allow specifying a keep alive, which will close the subscription after X miliseconds after all observers left.

Initialize RedisOplog automatically if settings for it are configured.

Not exactly an issue, but any call to Meteor.publishWithRedis() before the call to RedisOplog.init() will make the application crash.

Kinda obvious when you know it but might worth mentioning it the doc.

Or even add the possibility to put the init parameters directly in the app settings file and initialize everything automatically.

Awesome package BTW, real game changer IMO.

Fields for channels

When we publish on a channel, can we select fields normally?
return self.collection.find({_id:this.userId}, { fields:{messages:1}, channel:channel} );

Latency compensation doesn't work properly

As mentioned in this post, I've noticed an issue with latency compensation / optimistic ui.
The changed documents will be synced to the client, after the result of a method was received. That causes some flickering. The method on the client side will be reverted because the changes were not sent by the server before the method result was received.
The normal behavior is, that the changes were synced, while the client is waiting for the method result. After the result was received, the client compares invalidates the local working copy (minimongo) and takes the current state from the server.

DDP behavior with redis oplog:
image

Normal DDP behavior:
image

Code Refactoring Plan - Production Ready Label V 1.1

  • PublicationEntry will now only handle a single cursor.
  • Solve race-conditions problems by default, by fetching the data fields the observer.
  • Allow using Synthetic mutation like this: SyntheticMutation.update(string|Mongo.Collection, ...args)
  • Decouple insert/update/remove in it's own class that accepts a collection.
  • Write tests for synthetic mutations
  • Write tests for protect from race conditions
  • Support for operators in synthetic mutations
  • Write tests for collection hooks
  • Decouple tests to work with namespace, channel, normal kind of publication and updates/inserts. Don't rewrite what you can re-use. and decouple raceConditionProtect
  • Support for .observe() oldDoc for update.
  • Update README

Sharing query processors across a fleet.

UPI: unique publication identifier
Processor = Query Watcher

http://redis.io/topics/notifications
http://redis.io/commands/watch

This may be good very super-large scale.

Idea:
We want as few query processors as possible, this is why if we have a fleet of 100 servers and a webshop, if an instance is processing a given query, I want that instance to publish to redis to a custom channel the changes that need to be done to observers, without "thinking" about those changes.

When the instance no longer has direct observers, it will dispatch an event saying: "Hey man, start your own processors, I need to close this"

Careful with concurrency and different use-cases.

This will only be good for trully large-scale.

Query Processor On => UPI store in redis with the number of current observers (?)

Avoid requery if another one is scheduled.

Requery is the heaviest way of diffing a query. If we have 1 insert and 1 update. We may need to do requery 2 times. If a requery is scheduled to run. Don't schedule another one

Implement direct channels reactivity

Direct channels are for publications that subscribe to an _id or multiple _ids, with optional filters, but without limit.

They will only listen to changes done on those _id or _ids, without having to process additional stuff.

Virtual Mutations

You have chat, someone is typing... you don't actually want to save that, it can kill the db, but you can emulate it as if it was written in db, by sending a message to redis.

Think about this. It can be crucial to interactive apps, games, chats, etc.

Same Subscription Cacher

Given 100 users on a meteor instance that view your blog page with 15 blogs per page. You don't want 100 collection stores in memory.

Plan:
Create an md5 hash of name + filters and options.

ActivePublicationStore[md5Hash] = CollectionCacher

Whenever we publish, we simply .addObserver to the CollectionCacher.
When we remove the publish onStop, we kill the ActivePublicationStore if there are no observers left.

redis warning using Mongo bulk insert

    node_redis: Deprecated: The PUBLISH command contains a "null" argument.
    This is converted to a "null" string now and will return an error from v.3.0 on.
    Please handle this in your code to make sure everything works as you intended it to.

Using some code like:

    @bulk = ActionsCollection.rawCollection().initializeUnorderedBulkOp()
...
          @bulk.find _id: action._id
            .updateOne $set:
              start: null
              end: null
....
      Meteor.wrapAsync(@bulk.execute, @bulk)()

Collection Cacher

Create an object that accepts an array of documents and an observer.
Create a method that accepts a new array of documents. Creates a performant diff, updates its cache and creates an optimal queue for dispatching events to observer. (this.added, this.changed, this.removed)
When calling flush() all those events will be triggered one by one.

Reactive publishing fails with recent changes

The last functioning commit is this one:

commit 84fe684c98b1b101eed3cb66ee3eb5dc11b2ba0d
Merge: f24f549 4762210
Author: Theodor Diaconu <[email protected]>
Date:   Mon Nov 28 09:38:59 2016 +0200

    Merge pull request #56 from ramezrafla/patch-2
    
    Remove lack of support for callbacks in readme

We use package peerlibrary:reactive-publish like this:

    Meteor.publish('applications', function() {
      if (!this.userId) return;
      this.autorun(function() {
        var user = Meteor.users.findOne(this.userId,{fields:{'profile.applications.available':1}});
        var ids = user.profile.applications && user.profile.applications.available ? user.profile.applications.available : null;
        if (ids) return self.collection.find({_id:{$in:ids}},{fields:{name:1}});
        return self.collection.find({'default':true},{fields:{name:1}});
      })
    });

Here is the error

I20161128-12:08:13.285(-5)? Exception from sub applications id yofkZPqi4YCuWgLAH RangeError: Maximum call stack size exceeded
I20161128-12:08:13.286(-5)?     at Array.some (native)
I20161128-12:08:13.286(-5)?     at _.some._.any (packages/underscore.js:267:59)
I20161128-12:08:13.287(-5)?     at Function._.find._.detect (packages/underscore.js:217:5)
I20161128-12:08:13.287(-5)?     at Function.Mongo.Collection.get (packages/dburles_mongo-collection-instances.js:32:22)
I20161128-12:08:13.289(-5)?     at new ObservableCollection (packages/cultofcoders:redis-oplog/lib/cache/ObservableCollection.js:25:48)
I20161128-12:08:13.290(-5)?     at packages/cultofcoders:redis-oplog/lib/cache/PublicationEntry.js:20:20
I20161128-12:08:13.291(-5)?     at Array.map (native)
I20161128-12:08:13.291(-5)?     at new PublicationEntry (packages/cultofcoders:redis-oplog/lib/cache/PublicationEntry.js:19:46)
I20161128-12:08:13.291(-5)?     at create (packages/cultofcoders:redis-oplog/lib/mongo/extendObserveChanges.js:15:20)
I20161128-12:08:13.292(-5)?     at Cursor.module.export.exports.default.cursor.observeChanges (packages/cultofcoders:redis-oplog/lib/mongo/extendObserveChanges.js:28:16)
I20161128-12:08:13.292(-5)?     at Cursor.MeteorCursor._depend (packages/peerlibrary_reactive-mongo.js:73:8)
I20161128-12:08:13.292(-5)?     at Cursor.MeteorCursor.(anonymous function) [as fetch] (packages/peerlibrary_reactive-mongo.js:98:12)
I20161128-12:08:13.293(-5)?     at ObservableCollection.init (packages/cultofcoders:redis-oplog/lib/cache/ObservableCollection.js:83:32)
I20161128-12:08:13.293(-5)?     at packages/cultofcoders:redis-oplog/lib/cache/PublicationEntry.js:95:34
I20161128-12:08:13.294(-5)?     at Array.forEach (native)
I20161128-12:08:13.294(-5)?     at PublicationEntry._performInitialAddForObserver (packages/cultofcoders:redis-oplog/lib/cache/PublicationEntry

And we get many of these (but could be unrelated to the above publishing, we have many reactive publishing in our app which could trigger this, just mentioning for completeness)

I20161128-12:11:04.816(-5)? Exception in queued task: TypeError: Cannot read property '_id' of undefined
I20161128-12:11:04.816(-5)?     at packages/cultofcoders:redis-oplog/lib/utils/deepExtend.js:11:13
I20161128-12:11:04.817(-5)?     at Function._.each._.forEach (packages/underscore.js:147:22)
I20161128-12:11:04.817(-5)?     at deepExtend (packages/cultofcoders:redis-oplog/lib/utils/deepExtend.js:10:7)
I20161128-12:11:04.817(-5)?     at ObservableCollection.change (packages/cultofcoders:redis-oplog/lib/cache/ObservableCollection.js:147:9)
I20161128-12:11:04.817(-5)?     at handleUpdate (packages/cultofcoders:redis-oplog/lib/processors/direct.js:26:26)
I20161128-12:11:04.817(-5)?     at module.export.exports.default (packages/cultofcoders:redis-oplog/lib/processors/direct.js:11:13)
I20161128-12:11:04.817(-5)?     at Object.task (packages/cultofcoders:redis-oplog/lib/redis/RedisSubscriber.js:59:28)
I20161128-12:11:04.817(-5)?     at [object Object]._.extend._run (packages/meteor.js:807:18)
I20161128-12:11:04.817(-5)?     at packages/meteor.js:785:14

Add as an option to avoid race-conditions to redis

Given a context where you accuracy of data is critical race-conditions can happen when they want to push to redis after updates have finished at the same time.

X.find({}, {raceConditionProtect: true});

When an update happens, make sure you retrieve from db the fields.

X.update(selector, modifier, { raceConditionProtect: true });

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.