meteor-community-packages / ground-db Goto Github PK

https://atmospherejs.com/ground/db, like

Hi Raix,

I am trying using grounddb in my apps. But it can't be combined with cfs:gridFs packages. My image unable to be uploaded correctly in mongo.

(note: this question also posted at meteor talk)

And another problem is my collection doesn't return id when I am using syntax like this in client:

var _id = Posts.insert(doc);

Here are my collections code (nb: i am combine it with simple schema/collection2 from aldeed):

Posts = new Meteor.Collection("posts");

GroundDB(Posts); //this is not working
//Posts = new GroundDB('posts', {connection: null}); //this is not working too


var Schemas = {};


Schemas.Posts = new SimpleSchema({
    title: {
        type: String,
        label: "Title",
    },
    content: {
        type: String,
        label: "Content",
    },
    imageId: {
        type: String,
        label: "Image",
        optional: true,
    },
    /* AUTOVALUE */
    createdDate: {
        type: Date,
        label: "Created Date",
        autoValue: function() {
            if (this.isInsert)
                return new Date;
        },
        denyUpdate: true,
        optional: true
    },
    updatedDate: {
        type: Date,
        label: "Updated Date",
        autoValue: function() {
            if (this.isUpdate || this.isInsert)
                return new Date();
        },
        optional: true
    },
    createdUserId: {
        type: String,
        label: "Created by",
        autoValue: function() {
            if (this.isInsert)
                return this.userId;
        },
        denyUpdate: true,
        optional: true
    },
    updatedUserId: {
        type: String,
        label: "Updated by",
        autoValue: function() {
            if (this.isUpdate || this.isInsert)
                return this.userId;
        },
        optional: true
    },
});


Posts.attachSchema(Schemas.Posts);


Posts.allow({
    insert: function(userId, doc) {
        return true;
    },
    update: function(userId, doc) {
        return true;
    },
    remove: function(userId, doc) {
        return true;
    },
});

Any solution Raix?
thanks

I tried this in coffeescript:

@Products = new Meteor.Collection2 "products"
 smart:true
 schema:
  name:
   type: String
GroundDB @Products

But I don't see the collection reflected in localstorage :/ am I doing something wrong?

I'm getting an error when calling an insert on an groundDB collection.
This only happens in Safari (on OSX) the same code works in Chrome.

I also tried to do a browser cache clear and a manual localStorage.clear().
No change. Error is still occurring.

Any ideas what the reason is and how to fix it?

This could also be related to this one #28.

This is the error that occurs (in a loop... :( )

Exception in setTimeout callback: hasOwnProperty@[native code]
has@http://localhost:3000/packages/underscore.js?0a80a8623e1b40b5df5a05582f288ddd586eaa18:1123:31
insert@http://localhost:3000/packages/minimongo.js?6aa0e153bfd7e4a28ee447a4cbfaf1fef0870a06:567:13
http://localhost:3000/packages/grounddb.js?2bbe9cd082ee3b33e32032f69c3b37640717c91c:712:38
forEach@http://localhost:3000/packages/underscore.js?0a80a8623e1b40b5df5a05582f288ddd586eaa18:164:26
http://localhost:3000/packages/grounddb.js?2bbe9cd082ee3b33e32032f69c3b37640717c91c:710:13
http://localhost:3000/packages/grounddb.js?2bbe9cd082ee3b33e32032f69c3b37640717c91c:188:11
withValue@http://localhost:3000/packages/meteor.js?439f867e12888606900664d4463e1b3ee3644e44:790:21
http://localhost:3000/packages/meteor.js?439f867e12888606900664d4463e1b3ee3644e44:379:54
http://localhost:3000/packages/meteor.js?439f867e12888606900664d4463e1b3ee3644e44:818:27

Since updating to 0.9.1 I'm getting the following error in my console when I try to instantiate a grounddb collection:

Uncaught TypeError: Cannot read property '_id' of null groundDB.client.js:290

This error is thrown when this line is present in my app:

notifications = new GroundDB('notifications');

Not sure why this has started happening as it was working with 0.9.0 (I think!)

When using collection.insert(myNewObject, function(err, id){... the id attribute is undefined

The method array return by _gDB._loadObject('methods'); contains

[",","0",",","0"]

resulting an exception in when trying to split the method params.
My subscriptions will not become ready and GroundDB is broken for me.
I tried clearing localStorage to get rid of any lingering data, but even without calling any methods on that collection (except .find()) I get this nonsense out of local storage every time...

Meteor 0.9.0 saying GroundDB is incompatible.

What is the plan for 0.9.0+ compatibility? Any testing going on there?

SmartCollections are displaying fine with GroundDB.

However, insert/update/remove functions do not work with SC as the GroundDB uses internal collections API's vs the exposed Meteor collection API's

Not sure how to resolve this. @raix @arunoda

Method call /tasks/update groundDB.client.js?d5b04a6193f6abf7e319f075657f10fe69e952d2:267
Method call /tasks/update groundDB.client.js?d5b04a6193f6abf7e319f075657f10fe69e952d2:267
update failed: Method not found logging.js?f3d52175252df63f6ca42efd87b1e36ee79e80d7:40
Cache methods groundDB.client.js?d5b04a6193f6abf7e319f075657f10fe69e952d2:275
Method call /tasks/update groundDB.client.js?d5b04a6193f6abf7e319f075657f10fe69e952d2:267
update failed: Method not found logging.js?f3d52175252df63f6ca42efd87b1e36ee79e80d7:40
Cache methods groundDB.client.js?d5b04a6193f6abf7e319f075657f10fe69e952d2:275
Cache database: tasks groundDB.client.js?d5b04a6193f6abf7e319f075657f10fe69e952d2:271

When I add items to an array, say [1,2] becomes [1,2,3,4], then when I refreshed the browser while I am offline, the array becomes [1,2,3,4,3,4]. Everything goes back to normal when I get back online.

I'm currently investigating an issue in the automated QA test. The test "Test resume actions - Verify" fails.

Note: This could be an issue with the test tool itself or ground:db

QA Test 4 tests offline operations by creating 3 documents offline, then changes one and removing one.

QA Test 5 goes online, methods are resumed but the server actions update and remove have no effect.

The idear is to have a simple way to keep as much relevant data on ground as possible - so the 5 or 10 Mb we have should be quoted and some algorithms should determine what data gets to live on ground.

This is developed as a separate package
https://github.com/GroundMeteor/subscriptions

Meteor.call and Meteor.apply have a callback which is called asynchronously after the method is complete. As far I can see from some tests and the code, grounddb does not execute callbacks after the method is complete in this mobile scenario:

1. app goes offline
2. method called
3. device restarted
4. app resumed (but still offline)
5. app goes online
6. method complete, but callback not executed

ground:util is showing version 0.0.0, but ground:[email protected] requires 0.0.1

meteor show raix:eventemitter does not show 0.0.2 so ground:[email protected] will not install

Hi I got a following situation: (using a modified todo appliction for offline usage)
I am using apache cordova to bundle the application for my phone.
I am useing meteor 0.8.1.3.
I use grounddb and appcache.
on my phone I disconnect the network and enter something in a field then I reconnect the application and the data is pushed back to the server. I see this working on a browser.
but then after a heartbeat all the data disappears from the screen and in my console I see the following exception:
Exception in setTimeout callback: TypeError: Cannot convert undefined or null to object
at hasOwnProperty (native)
at Function..has (http://rolfst.todolist.meteor.com/packages/underscore.js?0a80a8623e1b40b5df5a05582f288ddd586eaa18:1123:27)
at LocalCollection.insert (http://rolfst.todolist.meteor.com/packages/minimongo.js?4ee0ab879b747ffce53b84d2eb80d456d2dcca6d:549:10)
at http://rolfst.todolist.meteor.com/packages/grounddb.js?f46709e51b0e1f0596708074bb8a9f5188f366c4:712:32
at Function..each._.forEach (http://rolfst.todolist.meteor.com/packages/underscore.js?0a80a8623e1b40b5df5a05582f288ddd586eaa18:164:22)
at http://rolfst.todolist.meteor.com/packages/grounddb.js?f46709e51b0e1f0596708074bb8a9f5188f366c4:710:9
at http://rolfst.todolist.meteor.com/packages/grounddb.js?f46709e51b0e1f0596708074bb8a9f5188f366c4:188:7
at _.extend.withValue (http://rolfst.todolist.meteor.com/packages/meteor.js?148e9381d225ecad703f4b858769b636ff7a2537:794:17)
at withoutInvocation (http://rolfst.todolist.meteor.com/packages/meteor.js?148e9381d225ecad703f4b858769b636ff7a2537:379:45)
at Meteor.bindEnvironment (http://rolfst.todolist.meteor.com/packages/meteor.js?148e9381d225ecad703f4b858769b636ff7a2537:822:22) debug.js:41

I tried grounding Meteor.users but I'm getting awkward behavior after deploying to modulus, this error is also happening in http://grounddb.meteor.com/

Basically after logging in, if you refresh the page, you can't log back in. After deleting the users login token you can log back in but if the token is set you're stuck logging in. Is there a work around for this?

• Update readme
• Update wiki, add info about arch etc.

Looks like simply using meteor add ground:db will break CollectionFS file uploads, at least with GridFS.

After adding ground:db, whenever I upload files, the file object has an _id of undefined. Removing ground:db fixes this.

Without ground:db

  Template.test.events
    'change input' : (e) ->
      file = col.insert e.currentTarget.files[0]
      console.log 'uploaded', file._id # zt7JbYQjWwKNqtFEu

With ground:db

  Template.test.events
    'change input' : (e) ->
      file = col.insert e.currentTarget.files[0]
      console.log 'uploaded', file._id # undefined

we would like to switch because it allows to request as much storage as you like.

If you could give us some pointers we would do the heavy lifting.

Hi,

I'm getting the following error on startup.

Uncaught [object Object] minimongo.js?daa88dc39d67b40b11d6d6809d72361f9ef6a760:422
LocalCollection.insert minimongo.js?daa88dc39d67b40b11d6d6809d72361f9ef6a760:422
(anonymous function) groundDB.client.js?acb6eb111163709518288000a5f2e6c95ace16bb:222
.each..forEach underscore.js?dfb8052aedcb94a47a99988cfcc6d58be576252b:87
self._loadDatabase groundDB.client.js?acb6eb111163709518288000a5f2e6c95ace16bb:221
window.GroundDB groundDB.client.js?acb6eb111163709518288000a5f2e6c95ace16bb:250
key.filter setup.js?0be5d603cc86c3753668a3aef0af431ec2ac56c5:143
ready startup_client.js?ebbfab1ed405df894a65817fe04734afb004a73e:8

The offending line of code is
Meteor.users = new GroundDB(Meteor.users);

Looks like a duplicate document _id shows up for the current logged in user.

The line is executed in the client on Meteor.startup()

I'm adding data as:

var pedido = {
  name: $("#name").val(),
  q: $("#q").val()
}

Pedidos.insert(pedido);

where Pedidos is defined as:

Pedidos = new GroundDB('pedidos');

I stop the server and add an item.
I start the server again.

The item is duplicated on server restart in DB.

{ "name" : "Otra", "q" : "23131231", "_id" : "fXnzburckbAbTc42q" }
{ "name" : "Otra", "q" : "23131231", "_id" : "uypyWuEePmWy3bkLX" }

Is there anything I'm doing wrong?
I'm using Meteor 1.0 and added the package.

At the moment a new GroundDB(null); will be suffixed null - if handing GroundDB a null database you get a client-side only database - but it would be nice to be able to hand an optional suffix/database name.

Add option for doing:

  var foo = new GroundDB(null, 'myLocalDatabase');

or

  var foo = new Meteor.Collection(null);
  var bar = new GroundDB(foo, 'myLocalDatabase');

or

  var foo = new Meteor.Collection(null);
  GroundDB(foo, 'myLocalDatabase');

This will help if we use other caching ressources in the future - current usage is for

• Chrome Packaged Apps.

But could also be:

• filesystem
• indexDB

Hi, I want to do it by my own but not sure where to start with,
I have a mobile application and cannot accept the processing of 200 data in each collection, I just need 16 data per collection with LIFO method,
I saw your code but seems like after many days it's not easy to get in,
but if you explain a bit will help me out,

Thanks
Nicolson Dsouza

I know groundDB support clients switching between on/offline. I am looking into finding a solution that allows clients to be launched offline and syncs data back to server whenever there is a connection. I am just wondering if groundDB support this kind of use case?

The exception happend when meteor app trying to sync current page with mongodb. The exception is like
Exception in setTimeout callback: TypeError: Cannot convert undefined or null to object
at hasOwnProperty (native)
at Function..has (http://192.168.1.113:3000/packages/underscore.js?0a80a8623e1b40b5df5a05582f288ddd586eaa18:1123:27)
at LocalCollection.insert (http://192.168.1.113:3000/packages/minimongo.js?4ee0ab879b747ffce53b84d2eb80d456d2dcca6d:549:10)
at http://192.168.1.113:3000/packages/grounddb.js?3e010a52196931b60a47305d7e6fd225dfee25ca:710:32
at Function..each._.forEach (http://192.168.1.113:3000/packages/underscore.js?0a80a8623e1b40b5df5a05582f288ddd586eaa18:164:22)
at http://192.168.1.113:3000/packages/grounddb.js?3e010a52196931b60a47305d7e6fd225dfee25ca:708:9
at http://192.168.1.113:3000/packages/grounddb.js?3e010a52196931b60a47305d7e6fd225dfee25ca:186:7
at _.extend.withValue (http://192.168.1.113:3000/packages/meteor.js?148e9381d225ecad703f4b858769b636ff7a2537:794:17)
at http://192.168.1.113:3000/packages/meteor.js?148e9381d225ecad703f4b858769b636ff7a2537:379:45
at http://192.168.1.113:3000/packages/meteor.js?148e9381d225ecad703f4b858769b636ff7a2537:822:22

The problem solved when I disabled grounddb.

The code of followings will throw above exception.
Pages = new GroundDB(new Meteor.SmartCollection("pages"));
Settings = new GroundDB(new Meteor.SmartCollection("settings"));
Navigation = new GroundDB(new Meteor.SmartCollection("navigation"));
Session.set('page-slug', null);
pagesSubscription = Meteor.subscribe('pages');
// user_listSubscription = Meteor.subscribe('user_list');
settingsSubscription = Meteor.subscribe('settings');
navigationSubscription = Meteor.subscribe('navigation');

And after disable grounddb the exception solved.
Pages = new Meteor.SmartCollection("pages");
Settings = new Meteor.SmartCollection("settings");
Navigation = new Meteor.SmartCollection("navigation");
Session.set('page-slug', null);
pagesSubscription = Meteor.subscribe('pages');
// user_listSubscription = Meteor.subscribe('user_list');
settingsSubscription = Meteor.subscribe('settings');
navigationSubscription = Meteor.subscribe('navigation');

What is wrong with grounddb? I still want to use grounddb for offline feature.

I wanted to know if offline user login is supported?

I have a question regarding using groundDB with iron-router waitOn function:
What is the correct way to skip waitOn subscription ready and use groundDB data instead?
In the other words: how to check that app is loading offline to prevent it wait on subscription ready, which will never happened?

The limit for localstorage is 3MB which is not enough to store all the master data we have.

we would be willing to finance the development of a new version of ground-db that stores the data in another manner and allows unlimited data volumes

After upgrading to 0.0.2, collection2 features no longer work

For example can not call .attachSchema() anymore on the client.

I have an app using grounddb. After upgrading to 0.9, something weird happened. I did NOT update the meteor version used by my app (still 0.8.3) and I have this error on some platforms (debian 7, ubuntu 12.4) and not others (Mavericks).

$ meteor
[... building other packages ...]
Errors prevented bundling:
While building package ejson-minimax:
error: no such package: '[email protected]'
error: no such package: '[email protected]'
error: no such package: 'raix'

Removing grounddb and commenting out the corresponding code, everything works fine. Hence, I think it should be something related to grounddb.

I think the problem should be a nasty syntax error in grounddb or some of its dependencies. In fact when I create a bundle, I get this other error

$ meteor bundle bundle.tgz

/home/me/.meteor/tools/cef2bcd356/lib/node_modules/fibers/future.js:173
throw(ex);
^
Error
at new JS_Parse_Error (/home/me/.meteor/packages/minifiers/85a69959b3/npm/node_modules/uglify-js/lib/parse.js:196:18)
at js_error (/home/me/.meteor/packages/minifiers/85a69959b3/npm/node_modules/uglify-js/lib/parse.js:204:11)

when I try to install grounddb i get this:
vagrant@packer-virtualbox-iso:~/project/todos$ mrt add grounddb
✓ GroundDB
tag: https://github.com/raix/Meteor-GroundDB.git#v0.0.18
✓ grounddb
tag: https://github.com/GroundMeteor/Meteor-GroundDB.git#v0.0.21

Done installing smart packages

Stand back while Meteorite does its thing

Done installing smart packages

Ok, everything's ready. Here comes Meteor!

=> Errors while scanning packages:

While building package GroundDB:
error: no such package: 'groundDB'

/home/vagrant/.nvm/v0.10.28/lib/node_modules/meteorite/lib/command.js:41
throw "Command exited with " + code + "/" + signal;
^
Command exited with 1/null

Why can't I install grounddb I see them both within my packages directory afterwards but the installation failed in the package file.
and why does it want to install those two versions of grounddb?

I get this issue when I have multiple tabs open with the same application.
Exception in setTimeout callback: TypeError: Cannot convert undefined or null to object

Any ideas?

I just added GroundDB via atmosphere and am at v. 0.0.6

On the client upon load I get the following.

Uncaught ReferenceError: GroundDB is not defined groundDB.client.js?3d5309880c1cd0ef4b83fc58ee0ace6f164c4bf1:88
(anonymous function) groundDB.client.js?3d5309880c1cd0ef4b83fc58ee0ace6f164c4bf1:88
(anonymous function) groundDB.client.js?3d5309880c1cd0ef4b83fc58ee0ace6f164c4bf1:436

No code etc. was added to the app itself.

Thank you for this great package.
I tested it with a normal collection and it works like a charm. Only Meteor.users seems to be different when having custom fields in it.

I deployed it to: http://grounddbusertest.meteor.com/ + repo at: https://github.com/akralj/grounddbUserTest

Steps to reproduce (with chrome or safari):

1. http://grounddbusertest.meteor.com/
2. login as : 111@111 with password: 111111
3. cut off your network
4. press "reload app" button which changes the collection & refreshes browser
5. reconnect network
6. do Meteor.reconnect() or wait up to 10s
   Uncaught Error: It doesn't make sense to be adding something we know exists: 3cKZpRMZmKpRXatwa
   livedata.js?a2e72054ba38809f8cac1e9197d52610e3efef8e:4258

groundDB.db.users is being nuked and meteor thinks one is logged out even if Meteor.userId() still exists in localstorage

It seems like Groundb & the normal Meteor sync overlap. I tested it with a a few versions of meteor and grounddb.

Here the main code.

 if (Meteor.isClient) {
    GroundDB(Meteor.users);
    Deps.autorun(function() {
      return Meteor.subscribe("userData");
    });
    Template.hello.helpers({
      date: function() {
        return Meteor.user().date;
      },
      userId: function() {
        return Meteor.userId();
      }
    });
    Template.hello.events({
      "click button": function() {
        Meteor.users.update({
          _id: Meteor.userId()
        }, {
          $set: {
            date: new Date()
          }
        });
        return Meteor.setTimeout((function() {
          return window.location.reload();
        }), 333);
      }
    });
  }

  if (Meteor.isServer) {
    Meteor.publish("userData", function() {
      return Meteor.users.find({
        _id: this.userId
      }, {
        fields: {
          date: 1
        }
      });
    });
    Meteor.users.allow({
      update: function() {
        return true;
      }
    });
  }

When a client connects and the GroundDB is populated, it allows offline work, which is pretty spectacular.

My concern is let's say the auth tokens are expired on the server while the client is offline, after reconnecting the method calls are run, but without a Meteor.userId, the server doesn't accept the changes. So all the offline work is lost.

You can test this by visiting http://grounddb.meteor.com/, logging in, disconnecting, queuing up some methods, manually modifying the auth-token (to simulate a server-side expiration), then reloading the page. All the offline changes are lost.

Is there a pattern for preventing the queued method calls from running until a Meteor.userId exists? If this were resolved, GroundDB would exactly fit a project I am about to start working on.

Hi,
Is groundDB expected to work on an Android target? Running on browser appears to work fine. But a run android-device throws:
(Meteor 1.0)

W20141116-17:55:35.740(-5)? (STDERR) ReferenceError: GroundDB is not defined

On the server-side there should be a conflict handler for letting the right data win.

• Create an api for building cusom conflictHandlers optional one pr. collection
• find a way of getting the diff between client and server time - this way client can calc the server time - this would be comparable with timestamps from other clients

Builtin conflictHandlers

• Last write to server wins
• The newest document wins
• OT/Merging strategy

There seem to be a way to simplify the code and get support for conflict resolution and better publish/subscription handling.

If done correctly all current code involving method resume will deprecate and give better room for subscription handling in iron:router.

I've tried to install the latest version of GroundDB and Minimax via meteorite, but I got this messages:

While building package GroundDB:
error: no such package: 'ejson-minimax'
error: no such package: 'groundDB'

While building package Minimax:
error: no such package: 'ejson-minimax'

What I ended up doing is cloning those two repositories (Meteor-GroundDB and ejson-minimax), fixing some naming inconsistencies and everything worked as expected.

Hi,

is there a way to get from a client only GroundDB name to the instance?

LocalProducts =                  new GroundDB('local_products', { connection: null })

So what I'm looking for is a way to get from the string "local_products" to the instance LocalProducts to call methods on it.

Cheers,
Gerwin

Hi,

I have a Grounded Collection that I use in a node-webkit "app".
Everything works fine. I can insert data in the collection I can switch to other routes. I can access the data. All cool.

When I reload the app then all the data is gone.

The same thing works from the chrome browser without any issues at all. (Even with reloading)
Any idea what the issue here can be?

Cheers,
Gerwin

Hi raix,

@awwx is working since a few months on an other implementation of offline client database for Meteor. Have you read it, and if so can you explain different and similarities between these two projects/strategies?

Thank you :)

Several autoform methods check to ensure that the collection passed is a instanceof Meteor.Collection. Since GroundDB now returns a proxy rather than extending an instance of Meteor.Collection, these tests fail.

Exception in setTimeout callback: TypeError: undefined is not a function

To reproduce:

Start meteor
Run app
Stop meteor
Insert into a collection
close browser
start meteor
open browser

I have done some investigating and this seems to be caused by GroundDB expecting a _id field to be sent in the insert message. DDP does not appear to do that but instead sends a randomSeed value to generate IDs from.

------ grounddb.client.js (line 555) -------------
var mongoId = _gDB.idParse((method.args && method.args[0])?
method.args[0]._id || method.args[0]:'');

Seems like generating the ID from the random seed might be the right thing to do, or maybe trick the client into thinking that it received the insert message.. Not sure yet. Still looking into it.

Any suggestions are welcome.

all changes made with relative query functions like $inc/ $dec / $push / $pop should be converted into absolute queries.

Reason is method resume currently does a passthrough, this is a problem when resuming method calls since these could actually risk running twice on server/client?

Client resume calls could also be shielded not changing the client db again.

So iron:router is really great when it comes to the way it handles subscriptions - eg. data is loaded when needed. But at the same time its also unloaded at change - eg. one subscription can replace an other.

hmm, so we have to think about this when working offline. We want to increment the data in the offline collection - at the same time we dont want to "publish" everything else but whats stated in the Meteor.subscribe.
The incremental loader should weight the data as to how often its used, changed and the size in order to have a strategy for dealing with quotas.

Tricky,

We could go about it having a GroundDB.publish function that intercepts the Meteor.subscribe - I guess it would all be alot easier if we knew what parts of the data originated from what subscriptions.

Hi,

I've done few changes to SmartCollection server side. Now SC server side is compatible with the Normal Collections.

In the client side Meteor.SmartCollection == Meteor.Collection. So you don't need treat separately for SmartCollections in the client side.

See: http://goo.gl/S5iAe3

Cheers.

So I'm having trouble understanding why you want to remove all local documents when starting On line 262. Makes it impossible to use this in a offline scenario it seems.

When removing this line documents are stored and able to be loaded. So what does it do? Why is it there? =)

Is this still a maintained project? I'm planning to fork it and make an indexedDB version - but I recall you mentioning that you'd like to make it work more modular and have adapters or something like that. Is this still planned or is it best to just fork it and do it myself?