Data loading and management: ideas about guide HOT 13 CLOSED

It's listed in the outline under data loading, so it's definitely something we need to address! I would rather not title it "server side reactivity" though.

Whenever people talk about this topic, they call it "reactive publish" or "publishing joins" but I think those terms themselves are too general. It's about publishing relational data. @mitar, would you mind if I rename this issue thread to that? Or is there something about server-side reactivity that we need that isn't related to relations?

Also, I think a lot of what you are talking about will be in here: #10 (specifically stuff about schema and normalization)

Meteor guide: Data loading and management

Meteor lets you write your UI as if the database is present on the client while maintaining security and the ability to have a decoupled data model. Sound like a contradiction? We'll explain how to use all of the tools together to get the best balance of fast development and maintainability.

1. How to load and use data from the database over DDP
2. How to load just enough data to display your UI while using caching to make sure the UI is as fast as possible
3. When to use local component state and when to have a global store
4. How to build your own client-side reactive data stores with ReactiveVar, ReactiveDict, and Tracker
5. Modifying data stores using Methods
6. How to use data from external APIs on the client and server
   1. HTTP
   2. DDP
   3. Webhooks
7. How to publish and use relational data
8. How to do pagination or infinite scroll so that you can load data incrementally as the user needs it

from guide.

would you mind if I rename this issue thread to that?

Sure, just go ahead. Feel free to edit the content of the issue description as well.

from guide.

Or is there something about server-side reactivity that we need that isn't related to relations?

Yes, as I wrote. Something are for me clear relations (give me more information about the author of this blog post). For which PeerDB and denormalization I think are the best approaches. (PeerDB is a bit old package, so probably it could be improved API-wise a bit, but the ideas I think are still very valid.)

Because you can query for things like, blog posts where the author's name is X (and author's name is otherwise part of users collection). If you try to compute this dynamically and reactively it gets really messy, and slow, because you are recomputing the same stuff for every client all the time (even joins are in general slow, but here you are then doing joins every time data changes).

And then there are things which are dynamic in nature, like depending on the user making a request. Like permission check. And for those reactive publish is really great. Or you can even make queries depending on time (look at the examples in reactive-publish).

(BTW, it would be really great to be able to remove userId from the publish. See peerlibrary/meteor-reactive-publish#1.)

from guide.

Hm, OK, but where would then be space for discussing server side reactivity? And things like how to manipulate server-side publications?

I remembers one other package I made some time ago, middleware, which aims at providing a reusable pipeline for modifications of publications. So you can have a basic publication which then has a stacked set of transformations on top.

from guide.

I think this is one of the really key areas. I feel like achieving all these things together (paginated, cached template-level subscriptions) is still really hard for such a basic feature. Maybe I'm completely wrong on this, but I suspect a lot of Meteor apps just front-load all their data just to sidestep these issues.

Maybe one other thing I would add is how to handle nested fields (see meteor/meteor#3764)

from guide.

I think problem of nested field is premature optimization. See my thoughts here: https://forums.meteor.com/t/nested-subdocuments-a-bad-idea/7859/8

from guide.

@SachaG do you have any thoughts on suggested techniques to achieve those things?

from guide.

This is the best implementation I've come up with so far: https://github.com/TelescopeJS/Telescope/blob/devel/packages/telescope-posts/lib/client/templates/posts_list/posts_list_controller.js

I also made a "simpler" version on MeteorPad as material for a future tutorial, but even that is still fairly complex:

http://meteorpad.com/pad/PmmryJYRxEwXGTudo/Template-Level%20Filtering%20&%20Pagination

Some of the issues involved:

• How do you wait on multiple subscriptions when some might be cached (using SubsManager) and some might not?
• How do you properly paginate to avoid the "flickering pagination" issue?
• How do you keep track of what's causing your Autoruns to rerun?

from guide.

Wow, this is pretty intense! I'll have to look in detail at that to see if there's anything to simplify. Thanks for the code sample!

from guide.

I am unclear where is the best place to put publish endpoints: next to the view/component, or next to the model definition?

from guide.

I've posted a set of topics on #33. I need to read up more on peerdb

from guide.

I think you should also try out reactive-publish. :-)

from guide.

I made two more packages:

• https://github.com/peerlibrary/meteor-control-mergebox
• https://github.com/peerlibrary/meteor-subscription-data

Especially the latter allows much better way of implementing infinite scrolling.

from guide.