Meteor smart package which provides a reactive database layer with references, generators, triggers, migrations, etc. Meteor provides a great way to code in a reactive style and this package brings reactivity to your database as well. You can now define inside your application along with the rest of your program logic also how your data should be updated on any change and how various aspects of your data should be kept in sync and consistent no matter where the change comes from.
Implemented features are:
- reactive references between documents
- reactive reverse references between documents
- reactive auto-generated fields from other fields
- reactive triggers
- migrations
Planned features are:
- versioning of all changes to documents
- integration with full-text search
- strict-typed schema validation
Adding this package to your Meteor application adds the Document object into the global scope.
Both client and server side.
meteor add peerlibrary:peerdb
- peerlibrary:peerdb-migrations – Migrations support for PeerDB documents
Instead of Meteor collections with PeerDB you are defining PeerDB documents by extending Document. Internally it
defines a Meteor collection, but also all returned documents are then an instance of that PeerDB documents class.
Minimal definition:
class Person extends Document
@Meta
name: 'Person'This would create in your database a MongoDB collection called Persons. name must match the class name. @Meta is
used for PeerDB and in addition you can define arbitrary class or object methods for your document which will then be
available on documents returned from the database:
class Person extends Document
# Other fields:
# username
# displayName
# email
# homepage
@Meta
name: 'Person'
# Class methods
@verboseName: ->
@Meta._name.toLowerCase()
@verboseNamePlural: ->
"#{ @verboseName() }s"
# Instance method
getDisplayName: ->
@displayName or @usernameYou can also wrap existing Meteor collections:
class User extends Document
@Meta
name: 'User'
collection: Meteor.usersAnd if you need to access the internal or wrapped collection you can do that by:
Person.Meta.collection._ensureIndex
username: 1PeerDB provides an alternative to Meteor collections query methods. You should be using them to access documents. You
can access them through the documents property of your document class. For example:
Person.documents.find({}).forEach (person, i, cursor) =>
console.log person.constructor.verboseName(), person.getDisplayName()
Person.documents.findOne().getDisplayName()
Person.documents.findOne().emailThe functions and arguments available are the same as those available for Meteor collections, with the addition of:
.documents.exists(query, options)– efficient check if any document matches givenquery.documents.bulkInsert(arrayOfDocuments, [options], callback)– insert multiple documents in bulk, returning the list of IDs and calling an optional callback
bulkInsert has a special handling of references to minimize issues of loading documents referencing documents which
are yet to be inserted. By default, first, all documents are inserted with all optional references delayed. This means,
all optional references are first omitted, and then all documents are updated by the second query, setting values for
all optional references. Reference fields inside arrays are always delayed. Optional options object accepts field:
dontDelay– a list of paths of optional reference fields which should not be delayed
In a similar way we extend the cursor returned from .documents.find(...) with an exists method which operates
similar to the count method, only that it is more efficient:
Person.documents.exists({})
Person.documents.find({}).exists()Person.Meta gives you back document metadata and Person.documents give you access to all documents.
All this is just an easy way to define documents and collections in a unified fashion, but it becomes interesting when you start defining relations between documents.
In the traditional SQL world of relational databases you do joins between related documents every time you read them from the database. This makes reading slower, your database management system is redoing the same computation of joins for every read, and also horizontal scaling of a database to many instances is harder because every read might potentially have to talk to other instances.
NoSQL databases like MongoDB remove relations between documents and leave it to users to resolve relations on their own. This often means fetching one document, observing which other documents it references, and fetching those as well. Because each of those documents are stand-alone and static, it is relatively easy and quick for a database management system like MongoDB to find and return them. Such an approach is quick and it scales easily, but the downside is the multiple round trips you have to do in your code to get all documents you are interested in. Those round trips become even worse when those queries are coming over the Internet from Meteor client code, because Internet latency is much higher.
For a general case you can move this fetching of related documents to the server side into Meteor publish functions by using libraries like meteor-related. It provides an easy way to fetch related documents reactively, so when dependencies change, your published documents will be updated accordingly. While latency to your database instances is hopefully better on your server, we did not really improve much from the SQL world: you are effectively recomputing joins and now even in a much less efficient way, especially if you are reading multiple documents at the same time.
Luckily, in many cases we can observe that we are mostly interested only in few fields of a related document, again
and again. Instead of recomputing joins every time we read, we could use MongoDB's sub-documents feature to embed
those fields along with the reference. Instead of just storing the _id of a related document, we could store also
those few often used fields. For example, if you are displaying blog posts, you want to display the author's name together
with the blog post. You won't really need only the blog post without the author name. An example blog post document
could then look like:
{
"_id": "frqejWeGWjDTPMj7P",
"body": "A simple blog post",
"author": {
"_id": "yeK7R5Lws6MSeRQad",
"username": "wesley",
"displayName": "Wesley Crusher"
},
"subscribers": [
{
"_id": "k7cgWtxQpPQ3gLgxa"
},
{
"_id": "KMYNwr7TsZvEboXCw"
},
{
"_id": "tMgj8mF2zF3gjCftS"
}
],
"reviewers": [
{
"_id": "tMgj8mF2zF3gjCftS",
"username": "deanna",
"displayName": "Deanna Troi"
}
]
}Great! Now we have to fetch only this one document and we have everything needed to display a blog post. It is easy for us to publish it with Meteor and use it as any other document, with direct access to author's fields.
Now, storing the author's name along with every blog post document brings an issue. What if user changes their name? Then you have to update all those fields in documents referencing the user. So you would have to make sure that anywhere in your code where you are changing the name, you are also updating fields in references. What about changes to the database coming from outside of your code? Here is when PeerDB comes into action. With PeerDB you define those references once and then PeerDB makes sure they stay in sync. It does not matter where the changes come from, it will detect them and update fields in referenced sub-documents accordingly.
If we have two documents:
class Person extends Document
# Other fields:
# username
# displayName
# email
# homepage
@Meta
name: 'Person'
class Post extends Document
# Other fields:
# body
@Meta
name: 'Post'
fields: ->
# We can reference other document
author: @ReferenceField Person, ['username', 'displayName']
# Or an array of documents
subscribers: [@ReferenceField Person]
reviewers: [@ReferenceField Person, ['username', 'displayName']]We are using @Meta's fields argument to define references.
In the above definition, the author field will be a subdocument containing _id (always added) and the username
and displayName fields. If the displayName field in the referenced Person document is changed, the author field
in all related Post documents will be automatically updated with the new value for the displayName field.
Person.documents.update 'tMgj8mF2zF3gjCftS',
$set:
displayName: 'Deanna Troi-Riker'
# Returns "Deanna Troi-Riker"
Post.documents.findOne('frqejWeGWjDTPMj7P').reviewers[0].displayName
# Returns "Deanna Troi-Riker", sub-documents are objectified into document instances as well
Post.documents.findOne('frqejWeGWjDTPMj7P').reviewers[0].getDisplayName()The subscribers field is an array of references to Person documents, where every element in the array will
be a subdocument containing only the _id field.
Circular references are possible as well:
class CircularFirst extends Document
# Other fields:
# content
@Meta
name: 'CircularFirst'
fields: ->
# We can reference circular documents
second: @ReferenceField CircularSecond, ['content']
class CircularSecond extends Document
# Other fields:
# content
@Meta
name: 'CircularSecond'
fields: ->
# But of course one should not be required so that we can insert without warnings
first: @ReferenceField CircularFirst, ['content'], falseIf you want to reference the same document recursively, use the string 'self' as an argument to @ReferenceField.
class Recursive extends Document
# Other fields:
# content
@Meta
name: 'Recursive'
fields: ->
other: @ReferenceField 'self', ['content'], falseAll those references between documents can be tricky as you might want to reference documents defined afterwards
and JavaScript symbols might not even exist yet in the scope, and PeerDB works hard to still allow you to do that.
But to make sure all symbols are correctly resolved you should call Document.defineAll() after all your definitions.
The best is to put it in the filename which is loaded last.
One more example to show use of nested objects:
class ACLDocument extends Document
@Meta
name: 'ACLDocument'
fields: ->
permissions:
admins: [@ReferenceField User]
editors: [@ReferenceField User]You can also do:
class ACLDocument extends Document
# Each permission object inside "permissions" could have also
# timestamp and permission type fields.
@Meta
name: 'ACLDocument'
fields: ->
permissions: [
user: @ReferenceField User
grantor: @ReferenceField User, [], false
]ReferenceField accepts the following arguments:
targetDocument– target document class, or'self'fields– list of fields to sync in a reference's sub-document; instead of a field name you can use a MongoDB projection as well, likeemails: {$slice: 1}required– should the reference be required (default) or not. If required, when the referenced document is removed, this document will be removed as well. If not required, the reference will be set tonull.reverseName– name of a field for a reverse reference; specify to enable a reverse referencereverseFields– list of fields to sync for a reference reference
What are reverse references?
Sometimes you want also to have easy access to information about all the documents referencing a given document. For example, for each author you might want to have a list of all blog posts they wrote, as part of their document.
class Post extends Post
@Meta
name: 'Post'
replaceParent: true
fields: (fields) ->
fields.author = @ReferenceField Person, ['username', 'displayName'], true, 'posts'
fieldsWe redefine the Post document and replace it with a new definition which enables
reverse references for the author field. Now Person.documents.findOne('yeK7R5Lws6MSeRQad') returns:
{
"_id": "yeK7R5Lws6MSeRQad",
"username": "wesley",
"displayName": "Wesley Crusher",
"email": "[email protected]",
"homepage": "https://gww.enterprise.starfleet/~wesley/",
"posts": [
{
"_id": "frqejWeGWjDTPMj7P"
}
]
}Sometimes you need fields in a document which are based on other fields. PeerDB allows you an easy way to define such auto-generated fields:
class Post extends Post
# Other fields:
# title
@Meta
name: 'Post'
replaceParent: true
generators: (generators) ->
generators.slug = @GeneratedField 'self', ['title'], (fields) ->
unless fields.title
[fields._id, undefined]
else
[fields._id, "prefix-#{ fields.title.toLowerCase() }-suffix"]
generatorsThe last argument of GeneratedField is a function which receives an object populated with values based on the list of
fields you are interested in. In the example above, this is one field named title from the Posts collection. The _id
field is always available in fields. Generator function receives just _id when document containing fields is being
removed. Otherwise it receives all fields requested. Generator function should return two values, a selector (often just the ID of a document)
and a new value. If the value is undefined, the auto-generated field is removed. If the selector is undefined, nothing is done.
You can define auto-generated fields across documents. Furthermore, you can combine reactivity. Maybe you want to also have a count of all posts made by a person?
class Person extends Person
@Meta
name: 'Person'
replaceParent: true
generators: (generators) ->
generators.postsCount = @GeneratedField 'self', ['posts'], (fields) ->
[fields._id, fields.posts?.length or 0]
generatorsYou can define triggers which are run every time any of the specified fields changes:
class Post extends Post
# Other fields:
# updatedAt
@Meta
name: 'Post'
replaceParent: true
triggers: ->
updateUpdatedAt: @Trigger ['title', 'body'], (newDocument, oldDocument) ->
# Don't do anything when document is removed
return unless newDocument?._id
timestamp = new Date()
Post.documents.update
_id: newDocument._id
updatedAt:
$lt: timestamp
,
$set:
updatedAt: timestampThe return value is ignored. newDocument and oldDocument can be null when a document has been
removed or added, respectively. Triggers are useful when you want arbitrary code to be run when
fields change. This could be implemented directly with observe,
but triggers simplify that and provide an alternative API in the PeerDB spirit.
Why we are using a trigger here and not an auto-generated field? The main reason is that we want to ensure
updatedAt really just increases, so a more complicated update query is needed. Additionally, reference
fields and auto-generated fields should be without side-effects and should be allowed to be called at any
time. This is to ensure that we can re-sync any broken references as needed. If you would use an
auto-generated field, it could be called again at a later time, updating updatedAt to a later time
without any content of a document really changing.
PeerDB does not really re-sync any broken references (made while your Meteor application was not running)
automatically. If you believe such references exist (eg., after a hard crash of your application), you
can trigger re-syncing by calling Document.updateAll(). All references will be resynced and all
auto-generated fields rerun. But not triggers. It is a quite heavy operation.
You can define abstract documents by setting the abstract Meta flag to true. Such documents will not create
a MongoDB collection. They are useful to define common fields and methods you want to reuse in multiple
documents.
We skimmed over replaceParent before. You should set it to true when you are defining a document with the
same name as a document you are extending (parent). It is a kind of a sanity check that you know what you are
doing and that you are promising you are not holding a reference to the extended (and replaced) document somewhere
and you expect it to work when using it. How useful replaceParent really is, is a good question, but it
allows you to define a common (client and server side) document and then augment it on the server side with
server-specific code.
If you would like to run some code after Meteor startup, but before observers are enabled, you can use Document.prepare
to register a callback. If you would like to run some code after Meteor startup and after observers are enabled, you can
use Document.startup to register a callback.
As your application grows you might want to run specialized Meteor instances just to do PeerDB reactive MongoDB
queries. To distribute PeerDB load, configure the number of PeerDB instances using the PEERDB_INSTANCES environment variable.
Suggested setting is that your web-facing instances disable PeerDB by setting PEERDB_INSTANCES to 0, and then you have
dedicated PeerDB instances.
If you are running multiple PeerDB instances, which instance is this? It is zero-based index so if you configured
PEERDB_INSTANCES=2, you have to have two instances, one with PEERDB_INSTANCE=0 and another with PEERDB_INSTANCE=1.
When running multiple instances you want to connect them all to the same database. You have to configure both normal MongoDB connection and also the oplog connection. You can use your own MongoDB instance or connect to one provided by running Meteor in development mode. In the latter case the recommended way is that one web-facing instance runs MongoDB and all other instances connect to that MongoDB.
MONGO_OPLOG_URL=mongodb://127.0.0.1:3001/local
MONGO_URL=mongodb://127.0.0.1:3001/meteor
See tests for many examples. See document definitions in PeerLibrary for real-world definitions.
- matb33:collection-hooks – provides an alternative way to
attach additional program logic on changes to your data, but it hooks into collection API methods so if a change comes
from the outside, hooks are not called; additionally, collection API methods are delayed for the time of all hooks to
be executed while in PeerDB hooks run in parallel in or even in a separate process (or processes), allowing your code to
return quickly while PeerDB assures that data will be eventually consistent (this has a downside of course as well,
so if you do not want that API calls return before all hooks run,
matb33:collection-hooksmight be more suitable for you) - peerlibrary:meteor-related – while PeerDB provides an easy way to embed referenced
documents as subdocuments, it requires that those relations are the same for all users; if you want dynamic relations
between documents,
meteor-relatedprovides an easy way to fetch related documents reactively on the server side, so when dependencies change, your published documents will be updated accordingly - herteby:denormalize – it does similar denormalization, but uses
matb33:collection-hookshooks instead reactivity to maintained denormalization, moreover, it looks likeherteby:denormalizeis much more limited in features than this package, which provides, e.g., also wrapping of documents into JavaScript objects with methods, generators, and reverse fields
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent