share / sharedb Goto Github PK

With ShareJS 0.7.0-alpha13 and livedb 0.3.0, I am getting the following error:
Error: You're using an old version of livedb. Please update livedb or downgrade ShareJS

AFAICT, these are the latest versions of each.

• Better error when calling submitOp or del on a document that hasn't been fetched

I change the sharedb source file and change rich-text to the default ot-type, and change some rules that proper to rich-text. It works well.

The repo:
https://github.com/FTAndy/sharedb-example
https://github.com/FTAndy/sharedb

If there is any favour I can do(add examples or add more ot-type), let me know. I will make a pr if is necessay.

Each error generated by sharedb should have a unique and documented error code.

• Add a section to README.md with a list of numeric error codes. Error are grouped based on error code prefix eg 41xx. Follow the convention set in https://github.com/share/sharedb-mongo#error-codes.
• Document all existing error codes (see code sites that create error objects with code: (...))
• Add error codes to other error objects. Choose an appropriate value for code (within one of the group where appropriate; or create a new group as needed)

I have been using a connection pool in combination with mongodb using generic-pool (The primary reason for doing this is that I am streaming images from GridFS).

Looking at the API it looks like I would have to instantiate livedb multiple times, is this incompatible with the inprocessDriver?

I am going to give livedb its own connection for now, but do you think connection pooling is something that would make sense in combination with livedb?

Per share/sharedb-mongo#24, it'd be great to have an adapter for a web-scale database such as Postgres 😉

It seems that livedb is renamed or rebuilt with sharedb, but the official demo or readme(in both share & sharedb doc) still use old APIs, such as .client() is actually not available in new sharedb.
I have faced errors with this new sharedb. Could you update your docs & demos please, thanks

Hello—

The docs state how I can fetch the operations leading up to a previous version, but not how I can apply these operations to then obtain a snapshot of that version. I wonder if you could add this to the docs?

Feature request: ideally, I would be able to pass a version argument to the fetch method, to directly request a snapshot of a certain version.

Thanks for share.js and kind regards,

I tried to update my racer installation to v0.5 today, which failed.

In the package.json hiredis is listed as a dependency, but as far as I can see, hiredis is not supported on windows. I quickly glanced over the source and it's not even used, did I miss something?

Maybe I'm just failing at my node-gyp configuration or something. Should the compilation on Windows work?

After upgrading ShareDB 0.11.36 to 1.0.0-beta.3 and ShareDB-Mongo 0.8.7 1.0.0-beta, I'm now getting:

Caught exception: TypeError: callback is not a function
TypeError: callback is not a function
    at Backend.submit (.../node_modules/sharedb/lib/backend.js:154:19)
    at module.exports (.../helpers/DocumentManager.js:17:7)
    at Object.<anonymous> (.../webstrates.js:49:62)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:458:32)
    at tryModuleLoad (module.js:417:12)
    at Function.Module._load (module.js:409:3)
    at Function.Module.runMain (module.js:575:10)
    at startup (node.js:160:18)

When calling:

share.submit(agent, collectionName, documentName, { v: 0, create: [] }, callback);

I've read the CHANGELOG, but that does not seem to indicate that I have to change anything.

The following link is broken (in README.md):

https://github.com/share/sharedb/blob/master/lib/memory.js

Is it possible to drop old operations from the oplog to free up database space after some specified timeout? We don't really need a full history of all operations after a few days (right?), and when operations are generated for every keypress pretty much, the database can grow very fast.

I tried using a Mongo ttl index on the ops but that doesn't work: We get a Missing oplog error from the redis driver when both redis and mongo have been cleared. Do we only need to keep around the latest operation (or 10 say), or is there a better solution? Thanks!

Hello. LiveDB not installing on new node, this is part of log https://gist.github.com/geakstr/d6e70941a10df7208582

Yep

Leaderboard:

C:\Users\alallier\Documents\sharedb\examples\counter>npm run build

> [email protected] build C:\Users\alallier\Documents\sharedb\examples\counter
> mkdir -p dist/ && ./node_modules/.bin/browserify client.js -o dist/bundle.js

The syntax of the command is incorrect.

Counter:

C:\Users\alallier\Documents\sharedb\examples\counter>npm run build

> [email protected] build C:\Users\alallier\Documents\sharedb\examples\counter
> mkdir -p dist/ && ./node_modules/.bin/browserify client.js -o dist/bundle.js

The syntax of the command is incorrect.

Even when removing the -p of mkdir in the build script this still fails

Hello,

I'd like to ask for advice on the current recommended approach for access control in sharedb. The documentation states that sharedb has "Middleware for implementing access control and custom extensions", but what middleware exactly is this referring to and how can it be used to implement access control?

I am trying to get sharedb-access by @zag2art working, but it seems that it is currently broken.

Any help or pointers to any presently working access control solution for Derby would be greatly appreciated. Thank you.

After #75, make sure to document all the APIs used in the example.

See the ShareJS API documentation for reference.

Can you provide example code for https://github.com/share/ShareJS integration?

Basically, see josephg/ShareJS#394

It should apply also to sharedb, since the asynchronous nature of Doc.prototype.destroy is still there.

This fails:

$ npm install livedb
$ node
> require('livedb')
Error: Cannot find module 'livedb'

I can confirm that livedb is in node_modules under that name, and has a package.json that also uses that name.

Compare with livedb-mongo,

$ npm install livedb-mongo
$ node
> require('livedb-mongo')
{ [Function] LiveDbMongo: [Function: LiveDbMongo] }

Which works just fine.

Currently if you call getOps(current version, null) and the version is stored in redis, livedb will check the oplog. It doesn't need to.

I'd like to completely remove some documents from the db including their ops. Is that possible?

Basically when we fetch a certain document, we check if it exists in the db and if not we create it with some initial data. Then when no one else is currently viewing this document we want to remove it from the database.

When I tried deleting the document from the db, when it's time to recreate it if someone tries to view it, I get CORRUPT DATA errors, probably because the document is still in memory?

This is more a question than a bug report, since I don't have much information.

Anyway, I get this

[Error: invalid / missing instruction in op]

from here:

Trace:
  at Object.json.apply (/home/ile/derby/test1/node_modules/derby/node_modules/racer/node_modules/share/node_modules/ottypes/lib/json0.js:210:15)
  at Object.json.incrementalApply (/home/ile/derby/test1/node_modules/derby/node_modules/racer/node_modules/share/node_modules/ottypes/lib/json0.js:224:21)
  at [object Object].Doc._otApply (/home/ile/derby/test1/node_modules/derby/node_modules/racer/node_modules/share/lib/client/doc.js:586:12)
  at [object Object].Doc._submitOpData (/home/ile/derby/test1/node_modules/derby/node_modules/racer/node_modules/share/lib/client/doc.js:694:8)
  at [object Object].Doc.submitOp (/home/ile/derby/test1/node_modules/derby/node_modules/racer/node_modules/share/lib/client/doc.js:726:8)
  at Doc.RemoteDoc.set (/home/ile/derby/test1/node_modules/derby/node_modules/racer/lib/Model/RemoteDoc.js:82:17)
  at set (/home/ile/derby/test1/node_modules/derby/node_modules/racer/lib/Model/mutators.js:37:24)
  at Model._mutate (/home/ile/derby/test1/node_modules/derby/node_modules/racer/lib/Model/mutators.js:15:10)
  at Model._set (/home/ile/derby/test1/node_modules/derby/node_modules/racer/lib/Model/mutators.js:41:15)
  at Model.ref (/home/ile/derby/test1/node_modules/derby/node_modules/racer/lib/Model/ref.js:184:8)
  at AsyncGroup.cb (/home/ile/derby/test1/src/app/index.coffee:98:12)
  at /home/ile/derby/test1/node_modules/derby/node_modules/racer/lib/util.js:54:10
  at Array.subscribeDocCallback [as 0] (/home/ile/derby/test1/node_modules/derby/node_modules/racer/lib/Model/subscriptions.js:157:5)
  at [object Object].Doc._finishSub (/home/ile/derby/test1/node_modules/derby/node_modules/racer/node_modules/share/lib/client/doc.js:453:34
... etc ...

I modified the json0.js a bit:

    else {
      console.trace();
      console.log(c);
      throw new Error('invalid / missing instruction in op');
    }

cis:

{ p: [] }

So, my question is, would you be interested to look at this, and if so, how can I help? Should I dump the database and put it online -- it would be ok to me (how would I do that?).

Another question is, how could I fix this, except by deleting the whole database?

Thanks.

I've tried to run the leaderboard example and the websocket-json-stream is not working.

It had also a lot of missing dependencies which are not included in package.json.The missing ones were serve-static and sharedb-mingo-memory

Error:
util.inherits(WebSocketServerStream, Duplex);
^
ReferenceError: WebSocketServerStream is not defined
at Object. (sharedb-master/examples/leaderboard/node_modules/websocket-json-stream/index.js:23:15)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Module.require (module.js:365:17)
at require (module.js:384:17)
at Object. (sharedb-master/examples/leaderboard/server/index.js:6:27)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)

We are seeing some errors in production that result in this nice place here:

https://github.com/share/livedb/blob/master/lib/redisdriver.js#L157-L166

which is to quote the comment

This is nate's awful hell error state. The oplog is basically corrupted - the snapshot database is further in the future than the oplog.

Looking into the database, this actually was the case, the version of the document inside the mongo collection was higher than the available versions in the ops db.

Does anyone have any ideas how this can happen, what to do with this case?

if rich-text is stable in livedb: this will be useful as I'm working on a project that connects directly to livedb from a quilljs editor.

registerType('rich-text');

and in rich-text (index.js):

module.exports.type = require('./lib/type');

While Redis obviously makes it possible to have multiple Share instances serving the same content, it's an extra dependency to admin for tiny projects that are fine with a single Share server and losing connection state on server reboot/crash.

It would be nice if, when initialized without a Redis client, livedb just provides its own internal PubSub thing.

We are working on a wrapper for elasticsearch. We have not yet implemented the query API, but everything else seems to be passing the adaptations of tests from the mongo, memory, and postgres implementations that we found.

When we try to use this as the backend for shareJS, though, we are running into the following error:

ERROR: CORRUPT DATA DETECTED in document jibe.The Dark Side
If you're using redis, delete data for document. Please file an issue if you can recreate this state reliably.
Op error: Internal data corruption - cannot submit jibe The Dark Side { op: [ 78, 'a' ],
  v: 692,
  src: 'f28b32f7c9038fa2dd1b46defb992932',
  seq: 238,
  preValidate: undefined,
  validate: undefined,
  m: { ts: 1429532663042 } } { channelPrefix: null, dirtyData: undefined }

I inserted another console output at line 285 of liveDB's lib/index.js, and this is the state right before it crashes:

// index.js#285
console.error("index.js 285", submitData, "from = ", from, "to = ", to, "result", ops);

// console output:
index.js 285 { cName: 'jibe',
  docName: 'The Dark Side',
  opData: 
   { op: [ 78, 'a' ],
     v: 692,
     src: 'f28b32f7c9038fa2dd1b46defb992932',
     seq: 238,
     preValidate: undefined,
     validate: undefined,
     m: { ts: 1429532663042 } },
  submitOptions: { channelPrefix: null, dirtyData: undefined },
  callback: [Function],
  start: 1429532663042,
  projection: undefined,
  transformedOps: [],
  expectTransform: true } from =  692 to =  null result []

From what I can gather from looking at the source, the issue is that the current document version that liveDB expects to be able to find in the database doesn't actually exist there yet?

// There's an awful state that should never happen (but did happen to us
// recently) where a driver knows there are more ops for a document, but the
// ops don't seem to exist anywhere. In this case, we need to make sure we
// don't end up in an infinite retry loop.
this.expectTransform = false;

Everything works fine for a seemingly random amount of time, and then crashes horribly. I am unable to recreate the error own my own as well, even with multiple browsers open; I have to have someone else modifying the same document at the same time that I am.

Do you have any experience with errors like this, and how to resolve them? Thanks in advance for any help!

I'm using a single websocket to communicate between my client and ShareDB server. In addition to ShareDB operations, I'm doing some other manual stuff over the websocket.

When I send my own requests over the websocket, ShareDB immediately complains and responds with 4000 "Invalid or unknown message".

How would I filter those custom messages out of the websocket that ShareDB is listening to before it gets to ShareDB, or make ShareDB silently ignore them?

From https://github.com/share/sharedb/blob/master/lib/backend.js#L144

// Submit an operation on the named collection/docname. op should contain a
// {op:}, {create:} or {del:} field. It should probably contain a v: field (if
// it doesn't, it defaults to the current version).
//
// callback called with (err, snapshot, ops)
Backend.prototype.submit = function(agent, index, id, op, callback) {
// ...
callback(err, request.ops);

Hi! preValidate accepts 2 arguments, opData and data. "data" does not have document collection in it. I can already see docName, but I also need collection name. Adding the collection name looks right for me. Do you agree?

{
 data: '  * [] orange\n  * [] apple',
 type: 'https://wiki.ru/types/yawikitextv1',
 v: 6,
 docName: 'users',
 m: { mtime: 1406299061056, ctime: '' }
}

Please explain to me what is the best place in code to add collection name in "data" argument.

From the docs it is not clear if it is possible to use livedb/ShareJS without mongodb.

To use it, you need a database to actually store your data in. A database wrapper for mongo is available in share/livedb-mongo. I hope to add more over time.

Does this mean it is not possible to use ShareJS without mongodb? I already have a redis and mysql instance, I'd rather not add mongo just for live editing capabilities.

Livedb has failing tests with the current stable versions of Node and io.js. This appears to be because of the changes in the Streams API—every failing test is comparing the null that ends the stream to a test value.

One or more of:

• Screencast
• Tutorial
• Guide

Hi, I use Node.js v5.0.0 and npm v3.3.9,the latest livedb and livedb-mongo depends on [email protected] and [email protected], which is out-dated for compile and install.

Build

➜  temp git:(master) ✗ node -v
v5.0.0
➜  temp git:(master) ✗ npm -v
3.3.9

> [email protected] install /Users/lizhuoli/Documents/Web/Node/DocumentCollaborationSystem/node_modules/livedb/node_modules/hiredis
> node-gyp rebuild

  CC(target) Release/obj.target/hiredis/deps/hiredis/hiredis.o
  CC(target) Release/obj.target/hiredis/deps/hiredis/net.o
  CC(target) Release/obj.target/hiredis/deps/hiredis/sds.o
  CC(target) Release/obj.target/hiredis/deps/hiredis/async.o
  CC(target) Release/obj.target/hiredis/deps/hiredis/read.o
  LIBTOOL-STATIC Release/hiredis.a
  CXX(target) Release/obj.target/hiredis/src/hiredis.o
In file included from ../src/hiredis.cc:3:
In file included from ../src/reader.h:4:
In file included from ../../nan/nan.h:74:
In file included from ../../nan/nan_new.h:181:
../../nan/nan_implementation_12_inl.h:172:66: error: too many arguments to function call, expected at most 2, have 4
......
npm WARN install:[email protected] [email protected] install: `node-gyp rebuild`
npm WARN install:[email protected] Exit status 1

and also for livedb-mongo install

npm ERR! peer dep missing: mongodb@~1.4, required by [email protected]

Temp solution

After I use the lastest [email protected] and [email protected], it solve and build success. After one day test with no bugs..I think maybe we should update the dependence ? Thanks

Hi,

No big deal but the name LiveDB kind of clashes with a similar project, http://livedomain.codeplex.com

Is it possible to add metadata when creating a document?
for example
livedb.submit('users', 'fred', {create:{type:'json0', data:[1,2,3],meta:{ownerId:123}}}, function(err, version, transformedByOps, snapshot) {
// I made a document, ma!
});


You have a done a great job of abstracting out the db api, and I am currently working on a custom implementation. However, I would also like to have the option of eliminating redis as the driver in favor of a different DB. I did not see any similar documentation on creating a custom driver. Is it currently possible to use/build a driver that uses something other than Redis?

It's missing bundle.js at http://localhost:8080/dist/bundle.js . Where is this JavaScript file?

Thank you.

Looking forward for feature to do paging through mangodb cursor

The changes made in 606eaf8 break the automatic creation of the redis observer in redisdriver.js (L52). I'm not sure why the parameter order has changed from port,host,options to options,port,host as the options should still be the last parameter - https://github.com/mranney/node_redis#rediscreateclient. The result is that the new client resorts to using the default host and port and ignoring those in the client it is attempting to clone.

It also looks like the port, host and options are also no longer directly accessible, but are stored in client.connectionOption, so something like the below :

this.redisObserver = redisLib.createClient(this.redis.connectionOption.port, this.redis.connectionOption.host, this.redis.options);

The above would break it for old versions of the redis module, but the optionalDependencies version for redis in package.json is ^0.12.1, so no backwards compatibility required?

Workaround for the moment is to create your own observer (i.e. another redis client) and pass that in:

var observerClient = redis.createClient(...);
var driver = livedb.redisDriver(oplogDb, regularClient, observerClient);

From email thread on derby mailing list from Anton Skopin

Hi
When testing projections with Derby 0.6-alpha-8 and alpha-10 I faced with the problem:
html:

<Body:>
  <ul class="countries">
    {{each _page.country}}
        <li class="country">
          <span>{{this.id}}: {{this.name}}</span>
          <input type="text" value="{{this.name}}">
        </li>
    {{/}}
  </ul>

controller:

app.get('/', function(page, model){
    var $query = model.query('countries_test' , q);
    model.subscribe($query, function() {
        $query.ref('_page.country');
        page.render();
    });
});

server:

  store.shareClient.backend.addProjection("countries_test", "countries", 'json0', {id:true, name:true});

The problem is that when the value in field changed (add new char or paste from clipboard) - entered value is duplicated. And further more if i enter more data, additional data not saved in mongoDB.

Thanks.

SSCE
https://github.com/cargobox/livedb-projector/tree/bug_demo

Made this comment here, but I'm creating a bug report for it:

I get this error:

TypeError: Cannot read property 'constructor' of undefined
  at Object.exports.client (/home/ile/rhc/dtest1/node_modules/derby/node_modules/racer/node_modules/share/node_modules/livedb/lib/index.js:31:20)
 at new ShareInstance (/home/ile/rhc/dtest1/node_modules/derby/node_modules/racer/node_modules/share/lib/server/index.js:19:27)

From this line:
if redisObserver.constructor is Object
https://github.com/share/livedb/blob/master/lib/index.coffee#L15

There are a lot of use cases where we need to use async validation logic with Racer: using data from database (model.fetch is async) for validation, using data from another async external sources.

Problem is that preValidate and validate functions are sync. So all our validation logic should be sync also. For example, we can not change racer-access plugin to support async validation, because it uses sync preValidate function.

Long time ago there was a PR to change this situation: #5 where @josephg said that async validation will impact performance.
I created new Issue to discus it from scratch.

What is the way to have async validation with Racer?

Could you please write some more info about "queries". I mean just "as is" for now. It helps me a lot to see into the sources.