Comments (8)
Not sure what you mean by re-use. When computation reruns, it has to create a new query because it is generally impossible to know what parameters you changed in the query. I hope that Meteor then deduplicates them. Or are you saying that the issue is that query is closed before same new one is created? Maybe there could be some way to postopne closing to see if a new one created is the same.
One thing you can do for checking if parameters changed is to use computed field to compute parameters. Something like:
Meteor.publish('foobar', function () {
const params = new ComputedField(() => {
return {day_of_week: timestamp_to_day_of_week(reactive_timestamp_every_minute)};
}, EJSON.equals);
this.autorun(() => {
return Collection.find(params());
});
});
In this example, while reactive_timestamp_every_minute
changes every minute, ComputedField
triggers invalidation only when output really changes. So only then query is rerun.
Subscription data is not recreating anything. It uses one publish for all data.
from meteor-reactive-publish.
@ramezrafla dug into this a bit over the past few days while I was trying to debug the cause of #37.
Answer is yes, assuming the cursor description is the same (selector + options have not changed) then the Meteor observe polling driver knows to reuse it just the same as it would in a plain old publication.
This package uses normal observeChanges
same as a publish cursor would. See _observeChanges in the mongo driver package. It checks existing multiplexers to see if a cursor with the same cursor description is open and uses it.
from meteor-reactive-publish.
I think the question is if the observe is reused if it is only used inside one computation which gets rerun. So do we close the observe and open exactly the same one again, or does Meteor (or we) do something smarter where if the observe is the same as before, we do not do anything.
On the client, if you have subscribe
inside autorun, Meteor will be smart and will first rerun it, and then see after rerunning if any subscription changed and close old ones, instead of closing first all, rerunning, and then subscribing again.
If you have time, could you investigate this?
from meteor-reactive-publish.
@mitar ah I gotcha. Yeah I have my local setup to check this easily. Will update here once I test.
from meteor-reactive-publish.
@ramezrafla so I created a sample repo which shows that on observer re-run, even if the selector + options are identical, the observer is not re-used .
The setup is pretty simple:
import { Meteor } from 'meteor/meteor';
import { Mongo } from 'meteor/mongo';
const CollectionA = new Mongo.Collection('collectionA');
const CollectionB = new Mongo.Collection('collectionB');
const tickTock = new ReactiveVar(new Date().getTime());
Meteor.setInterval(() => {
tickTock.set(new Date().getTime());
}, 10000);
Meteor.publish('testObserverReuse', function testObserverReuse() {
this.autorun(() => {
const tick = tickTock.get();
console.log('re-running');
CollectionA.findOne(
{ testField: 'abc' },
{ fields: { testField: 1 } }
);
return CollectionB.find(
{ createdAt: { $gt: tick } },
{ fields: { testField: 1 } }
);
});
});
// On client, subscribe
Meteor.subscribe('testObserverReuse');
Then in the PollingObserveDriver I added a line to assign a random id to each driver like:
PollingObserveDriver = function(options) {
var self = this;
self.__driverId = Random.id();
// ...
}
Finally, on the actual _pollMongo call, I log the id to see if changed:
_pollMongo: function () {
// ...
// Normal code excluded for brevity
self._pendingWrites = [];
// Log the id to see if it changed
console.log(self.__driverId);
// ...
}
I found that despite the arguments for the findOne being the same, the driver is not reused. It creates a new one each time the autorun fires again.
We can prove that it is actually querying mongo again by connecting to mongo shell, setting db.setProfilingLevel(2)
and querying db.system.profile.find({ns: 'meteor.collectionA', 'command.filter.testField': 'abc' }).count()
between each successive interval when the reactive var is updated.
What's concerning to me is that I found each autorun trigger two queries to the mongo. I added a branch on the repo you can checkout which does not autorun and you can see only triggers 1 query to mongo.
@mitar does this answer the original question?
from meteor-reactive-publish.
The double query was unexpected. But that combined with the fact that observers set up from within autoruns always poll explains a lot of the cpu spikiness in our app :)
Hopefully we can all work together to solve both issues. I think observer re-use would be a great bonus.
from meteor-reactive-publish.
Hm, I would assume that there will be three queries: one query, and two observers. So CollectionA.findOne
makes a query and then also makes an observe to detect if query has to be rerun. If it has, it reruns the autorun which makes a query and observe again. And then there is an observe for CollectionB.find
.
Which queries do you observe?
Now, based on discussion above, I think there could be maybe multiple optimizations we could:
- When rerunning a computation, we might not want to stop the observe immediately when the computation is invalidated, but we could do that in
afterFlush
, which would allow observe to be reused between reruns. Similar how it is done for subscribe. In our case, it is probably even easier to do because we have observe multiplexer on the server side, so we could do an observe in reurn, and then if it is the same as last time, it would get deduplicated in multiplexer, then inafterFlush
we would tear down old observe, which would just remove an instance from the multiplexer, while the heavy work of really observing would be kept in the multiplexer.- In fact it would be important that you see if
ObserveMultiplexer
is being currently recreated between reruns or not. Just to be sure. To my understanding every time you createObserveMultiplexer
also a observe driver is created, but every time you make an observe itself, it does not necessary meanObserveMultiplexer
is created, if it is the same query.
- In fact it would be important that you see if
- As I mentioned, we currently do both a query and observe for each
collection.find
inside autorun. In theory, we could do just observe and reconstruct the results of the find fromObserveMultiplexer
cache for that particular observe. We can do that because the query we use for database query and observe is the same. I have no idea why Meteor hasObserveMultiplexer
caching all documents, but from my quick experimentation just now it seems it truly caches all documents inside observe, even if you are using justobserveChanges
and notobserve
. - Because I am assuming queries generally do not change between reruns, just data, we could even try to optimize things even further and simply have one observe per query per computation all the time open, and we just do queries against cache always. So instead of starting and closing the observe, we would just open it the first time, detect when something changes, rerun the computation, and return documents from its cache. I am not sure this would really improve much given multiplexer.
I am not sure if all those optimizations should really be done and if they would really improve things overall. They would add more complexity for sure. Which can lead to strange side effects we would not anticipated. I would maybe do the first one, but not the other two. In my personal case I do not have really many changes happening inside autorun to rerun it. It is mostly just to catch rare things like permissions changing and updating what documents an user can see in real-time.
from meteor-reactive-publish.
@mitar thanks for the deep dive. I think you're right about items 2 and 3 adding too much complexity. Item 1 seems attractive here and I'd be eager to try it out. Any suggested approach for where the afterFlush should happen?
I can give this a try on a separate branch and see if it works as expected.
from meteor-reactive-publish.
Related Issues (20)
- Probably we should not be doing handles.push(result) if handles is a computation
- After updating to Meteor 1.4.4.1 ReactiveVar is not defined HOT 14
- Multiple cursors for collection HOT 6
- Make sure DDP._CurrentPublicationInvocation is correctly set during autorun reruns HOT 1
- Update to latest server-autorun dependency HOT 3
- Error on Meteor Update 1.6.1 when using practicalmeteor:mocha HOT 48
- it doesn't work with meteor 1.6.1 ? HOT 6
- Is reactive-publish expected to work with this.changed on a "virtual" collection? HOT 14
- Stopping subscription before it's ready can leave observers running forever HOT 2
- Question: How to publish to specific client side collection? HOT 2
- [Meteor 1.8.1] data not removed from client due to ddp-server internal changes HOT 12
- Question: Meteor 1.8.1 - ReferenceError: ReactiveVar is not defined HOT 4
- Update peerlibrary:reactive-mongo dependency to @0.3.0 HOT 3
- Nested autoruns on server is sending ready message before all autoruns are ready HOT 4
- Memory leak server crash with (0.3.0) HOT 2
- Issue with using observeChanges in autorun function HOT 4
- Update observing query changes patching
- The package overrides or ignores my Mongo poolSize setting HOT 1
- ReferenceError: ReactiveVar is not defined HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from meteor-reactive-publish.