Code Monkey home page Code Monkey logo

marsdb's Introduction

Build Status npm version Coverage Status Dependency Status bitHound Overall Score Join the chat at https://gitter.im/c58/marsdb GitHub stars

MarsDB is a lightweight client-side database. It's based on a Meteor's minimongo matching/modifying implementation. It's carefully written on ES6, have a Promise based interface and may be backed with any storage implementation (see plugins). It's also supports observable cursors.

MarsDB supports any kind of find/update/remove operations that Meteor's minimongo does. So, go to the Meteor docs for supported query/modifier operations.

You can use it in any JS environment (Browser, Electron, NW.js, Node.js).

Features

  • Promise based API
  • Carefully written on ES6
  • Very very flexible – just take a look to the plugins section
  • Supports many of MongoDB query/modify operations – thanks to a Meteor's minimongo
  • Flexible pipeline – map, reduce, custom sorting function, filtering. All with a sexy JS interface (no ugly mongo's aggregation language)
  • Persistence API – all collections can be stored (and restored) with any kind of storage (in-memory, LocalStorage, LevelUP, etc)
  • Observable queries - live queries just like in Meteor, but with simplier interface
  • Reactive joins – out of the box

Bindings

Plugins

Meteor compatible client/server

Sometimes you can't use Meteor infrastructure. Maybe you need to build a custom client. Maybe you need to build a custom server with express and other modules. In meteor it can be done with a ton of hack. But the only reason why it's so ugly to do a simple things is because Meteor forces you to use their infrastructure. I'm trying to solve this issue with DDP client/server modules, based on MarsDB.

Examples

Using within non-ES6 environment

The ./dist folder contains already compiled to a ES5 code, but some polyfills needed. For using in a browser you must to include marsdb.polyfills.js before marsdb.min.js. In node.js you need to require('marsdb/polyfills'). It sets in a window/global: Promise, Set and Symbol.

Create a collection

import Collection from 'marsdb';
import LocalForageManager from 'marsdb-localforage';

// Default storage is in-memory
// Setup different storage managers
// (all documents will be save in a browser cache)
Collection.defaultStorageManager(LocalForageManager);

// Create collection wit new default storage
const users = new Collection('users');

Create an in-memory collection

import Collection from 'marsdb';
import LocalStorageManager from 'marsdb-localstorage';

// Set some defaults and create collection
Collection.defaultStorageManager(LocalStorageManager);
const users = new Collection('users');

// But it may be useful to create in-memory
// collection without defined defaults
// (for example to save some session state)
const session = new Collection('session', {inMemory: true});

Find documents

const posts = new Collection('posts');
posts.find({author: 'Bob'})
  .project({author: 1})
  .sort(['createdAt'])
  .then(docs => {
    // do something with docs
  });

Find with pipeline (map, reduce, filter)

An order of pipeline methods invokation is important. Next pipeline operation gives as argument a result of a previous operation.

const posts = new Collection('posts');

// Get number of all comments in the DB
posts.find()
  .limit(10)
  .sortFunc((a, b) => a - b + 10)
  .filter(doc => Matsh.sqrt(doc.comment.length) > 1.5)
  .map(doc => doc.comments.length)
  .reduce((acum, val) => acum + val)
  .then(result => {
    // result is a number of all comments
    // in all found posts
  });

// Result is `undefined` because posts
// is not exists and additional processing
// is not ran (thanks to `.ifNotEmpty()`)
posts.find({author: 'not_existing_name'})
  .aggregate(docs => docs[0])
  .ifNotEmpty()
  .aggregate(user => user.name)

Find with observing changes

Observable cursor returned by a find and findOne methods of a collection. Updates of the cursor is batched and debounced (default batch size is 20 and debounce time is 1000 / 15 ms). You can change the paramters by batchSize and debounce methods of an observable cursor (methods is chained).

const posts = new Collection('posts');
const stopper = posts.find({tags: {$in: ['marsdb', 'is', 'awesome']}})
  .observe(docs => {
    // invoked on every result change
    // (on initial result too)
    stopper.stop(); // stops observing
  }).then(docs => {
    // invoked once on initial result
    // (after `observer` callback)
  });

Find with joins

const users = new Collection('users');
const posts = new Collection('posts');
posts.find()
  .join(doc => {
    // Return a Promise for waiting of the result.
    return users.findOne(doc.authorId).then(user => {
      doc.authorObj = user;
      // any return is ignored
    });
  })
  .join(doc => {
    // For reactive join you must invoke `observe` instead `then`
    // That's it!
    return users.findOne(doc.authorId).observe(user => {
      doc.authorObj = user;
    });
  })
  .join((doc, updated) => {
    // Also any other “join” mutations supported
    doc.another = _cached_data_by_post[doc._id];

    // Manually update a joined parameter and propagate
    // update event from current cursor to a root
    // (`observe` callback invoked)
    setTimeout(() => {
      doc.another = 'some another user';
      updated();
    }, 10);
  })
  // Or just pass join spec object for fast joining
  // (only one `find` will be produced for all posts)
  .join({ authorId: users }) // posts[i].authorId will be user object
  .observe((posts) => {
    // do something with posts with authors
    // invoked any time when posts changed
    // (and when observed joins changed too)
  })

Inserting

const posts = new Collection('posts');
posts.insert({text: 'MarsDB is awesome'}).then(docId => {
  // Invoked after persisting document
})
posts.insertAll(
  {text: 'MarsDB'},
  {text: 'is'},
  {text: 'awesome'}
).then(docsIds => {
  // invoked when all documents inserted
});

Updating

const posts = new Collection('posts');
posts.update(
  {authorId: {$in: [1, 2, 3]}},
  {$set: {text: 'noop'}}
).then(result => {
  console.log(result.modified) // count of modified docs
  console.log(result.updated) // array of updated docs
  console.log(result.original) // array of original docs
});

// Upsert (insert when nothing found)
posts.update(
  {authorId: "123"},
  {$set: {text: 'noop'}},
  {upsert: true}
).then(result => {
  // { authorId: "123", text: 'noop', _id: '...' }
});

Removing

const posts = new Collection('posts');
posts.remove({authorId: {$in: [1,2,3]}})
  .then(removedDocs => {
    // do something with removed documents array
  });

Roadmap

  • Indexes support for some kind of simple requests {a: '^b'}, {a: {$lt: 9}}
  • Documentation

Contributing

I'm waiting for your pull requests and issues. Don't forget to execute gulp lint before requesting. Accepted only requests without errors.

License

See License

marsdb's People

Contributors

c-rack avatar c58 avatar gitter-badger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

marsdb's Issues

How to use it from node.js 4 stable?

I was trying to use marsdb as a replacement of linvodb3 because I would prefer to use promises, however I think I'm giving up.

Here's what happens:

let Mongo = require('marsdb');
let Doc = new Mongo(modelName, schema, options);

gives me the following error:

TypeError: Mongo is not a function

The let Doc is taken from the linvodb3 library, and since there isn't really an example on how to use it on node, I think I'm stuck :(

Thanks in advance for the help! :)

Indexes for simple query operations

Indexes must be used with queries like:

  • {name: "^Art"} – fast search by first characters of name
  • {age: {$gt: 5, $lt: 10}} – fast search for simple numeric(and alphabetic) range

very very low performance? may I make some mistakes?

those days I am looking for good leveldb-base complete database. And I find marsdb and linvodb3.
I don‘t like linvodb3 cos it don't support promise. so first I choose marsdb.
I did a try to insert 1000 records with some data, and found it very slow.
then I tried sqlite3 and sequelize, even it is faster than marsdb.
today I tried linvodb3, it is really very fast.
so I am confused, maybe I make some mistakes on marsdb?
below are the codes:

//-------------------------
var co = require('co');
var randomize = require('randomatic');
var Q = require('q');

//-----------LinvoDB----------
const LinvoDB = require("linvodb3");
LinvoDB.dbPath = './dbBenchmark/linvodb';
let modelName = "doc";
let schema = {}; // Non-strict always, can be left empty
let options = {};
let Doc = new LinvoDB(modelName, schema, options); // New model; Doc is the constructor

function linvodb3Insert(_index) {
    var deferred = Q.defer();
    Doc.insert(
        {
            string1: 'LinvoDB' + randomize('A', 20),
            number1: _.now(),
            indexID: _index
        },
        function (err, newDoc) {
            if (err) {
                deferred.reject(new Error(err.message));
            }
            else {
                deferred.resolve(newDoc);
            }
        });
    return deferred.promise;
}

router.get('/linvodb3Insert', function (req, res) {
    let promises = [];
    for (let i = 0; i < 1000; i++) {
        promises.push(linvodb3Insert(i));
    }
    Q.all(promises).then(function (results) {
        return res.json({msg: 'success'});
    }, function (err) {
        return res.json({msg: 'err'});
    });
});

router.get('/linvodb3Count', function (req, res) {
    Doc.count({}, function (err, count) {
        return res.json({msg: 'success', count: count});
    });
});
//-----------LinvoDB----------

//------marsdb--------
let Collection = require("marsdb").Collection;
let LevelStorageManager = require("marsdb-levelup");
LevelStorageManager.defaultStorageLocation('./dbBenchmark/marsdb');
Collection.defaultStorageManager(LevelStorageManager);
router.get('/marsdbInsert', function (req, res) {
    co(function*() {
        for (let i = 0; i < 1000; i++) {
            const posts = new Collection('posts');
            yield posts.insert(
                {
                    string1: 'MarsDB' + randomize('A', 20),
                    number1: _.now(),
                    indexID: i
                }
            );
        }

        return res.json({msg: 'success'});
    }).catch(function (err) {
        return res.json({msg: 'err'});
    });
});
//---------marsdb------------

//------------sqlite----------
let Sequelize = require('sequelize');
let sequelize = new Sequelize('database', 'username', 'password', {
    dialect: 'sqlite',
    pool: {
        max: 5,
        min: 0,
        idle: 10000
    },
    logging: null,
    storage: './dbBenchmark/sqlite/test.db'
});
let User = sequelize.define(
    'user',
    {
        'indexID': {
            'type': Sequelize.BIGINT(10),
            'allowNull': false
        },
        'string1': {
            'type': Sequelize.STRING(10),
            'allowNull': false
        },
        'number1': {
            'type': Sequelize.BIGINT(20),
            'allowNull': false
        }
    }
);
User.sync();

router.get('/sqliteInsert', function (req, res) {
    co(function*() {
        for (let i = 0; i < 1000; i++) {
            yield User.create({
                string1: 'sqlite' + randomize('A', 20),
                number1: _.now(),
                indexID: i
            });
        }

        return res.json({msg: 'success'});
    }).catch(function (err) {
        return res.json({msg: 'err'});
    });
});
//------------sqlite----------

//--------mongodb----------
let mongoose = require('mongoose');
//多数据集支持
let connection = require(path.join(__dirname, '..', '..', 'dbConnection_chihuo.js')).connection;
let schema1 = mongoose.Schema;

let testCollectionName = 'dbBenchmark';
let testSchema = new schema1({
    //_id
    string1: String,
    number1: Number,
    indexID: Number
});
testSchema.index({_id: 1, indexID: 1});
let tester = connection.model(testCollectionName, testSchema);
router.get('/mongoInsert', function (req, res) {
    co(function*() {
        for (let i = 0; i < 1000; i++) {
            yield tester.create({
                string1: 'mongo' + randomize('A', 20),
                number1: _.now(),
                indexID: i
            });
        }
        return res.json({msg: 'success'});
    }).catch(function (err) {
        return res.json({msg: 'err'});
    });
});
//--------mongodb----------

the benchmark is:
sqlite: 8658ms
marsdb: can't finish, time out. so I only try 100 inserts, and the result is 13927.
linvodb3:325ms
mongodb:966ms

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.