deepstreamio / deepstream.io Goto Github PK
View Code? Open in Web Editor NEWdeepstream.io server
Home Page: https://deepstreamio.github.io
License: MIT License
deepstream.io server
Home Page: https://deepstreamio.github.io
License: MIT License
If deepstream supports this, I would like an example in the tutorials section to cover this use case:
( also asked on reddit )
Would be awesome if you guys could implement a way to remove a record's value. E.g.:
var record = client.record.get('user/jwanglof')
console.log(record.get());
Console log:
{
firstname: 'Johan',
surname: 'Wanglof',
pets: [
{pet_id: 'roger_the_rabbit'},
{pet_id: 'max_the_cat'},
{pet_id: 'rosa_the_cow'}
]
}
Lets say that max_the_cat dies (sorry for being gruesome) and I want to remove it from the pets-list. It would be sweet if I could do a delete on just that record by index (or index-name perhaps?):
record.delete('pets[1]');
Create a deepstream.io client for C / C++, that connects to deepstream via TCP and supports Authentication, Permissioning, Records, Events and RPCs. It might make sense to take IoT (Arduino etc.) compatibility into account for the design.
Great resources to get started:
Writing a deepstream client
Deepstream Messaging structure
Detailed Cucumber Specs for Incoming and Outgoing messages (and associated behaviors)
Fully featured NodeJS Client as a reference
Very interesting project. Kudos!
If you don't mind, I'm listing a couple of feature requests that might be discussion worthy.
First up: Offline mode -- allowing mobile devices to go offline, save to local storage and resync + patch when coming back online.
Check before using methods such as valueOf, toString etc. as it might be possible to overwrite them as part JSON.parse
I'm curious to know more about the reasoning and decision to create the proprietary messaging format used by deepstream.io. Were other formats like MessagePack or WAMP considered at any point? Are there any specific advantages you can share?
During a server shutdown, when clients are disconnected by the shutdown, the _server.close callback is never called.
Perhaps engine.io is still accepting connections after close?
It seems currently permissions have to be set per each individual user. Access control based on roles would make management of these permissions a lot easier.
See deepstream.io-client-js#30 for details
Hi, this is not an issue but a question.
How would you handle user registration? I have two ideas about this:
ds.login()
. Problem here is that I need two records one for the auth data and a second record which can observe with deepstream.ds.login({user: anonymous, password: anonymous})
when the user visit the page. So the user can create an record with his username and password and if the user login i would change ds.login()
to ds.login({user: someusername, password: somepassword})
. The problem here is to allow the anonymous user the he can only create an user record.What is the recommended way to achieve an user management system?
Create a deepstream.io client for Python, that connects to deepstream via TCP and supports Authentication, Permissioning, Records, Events and RPCs.
Great resources to get started:
Writing a deepstream client
Deepstream Messaging structure
Detailed Cucumber Specs for Incoming and Outgoing messages (and associated behaviors)
Fully featured NodeJS Client as a reference
Does Deepstream support HTTP streaming beyond socket.io?
I ran into an online service: appbase.io which supports it but response times are ~1 sec.
Would be really helpful to know what changes that have been made to each release.
Every time I try to use the error-callback I get undefined is not a function
and when looking in the rpc-response.js
-file I don't see an error-function in the prototype.
I tried to implement my own error-function but when I send the ERROR-action it seems like the entire connection is disconnecting. The code I'm trying to use:
this._connection.sendMsg( C.TOPIC.RPC, C.ACTIONS.ERROR, [ this._name, this._correlationId, typedData ] );
Is there something I'm missing?
Given that some records can be updated multiple times a second, this can result in putting the database under a bit of stress from continuous writing.
As such it would be ideal to have a filter system where we can optionally specify what records to store and which ones to ignore.
This can be done on a per record basis via the metadata associated when created, or potentially via a regex.
The following is just for reference, and not the decided API.
For example globals ( not as fast, but alot easier and resistant to errors ):
ds.set( 'storage-options', {
include: [ /store-me\//, /store-those\// ], //Only save if matches
exclude: [ /disposable\//, /too-quick\// ]//Only ignore if matches
} );
and per record, which is quicker ( and the API for meta data is still in the works ):
ds.record.getRecord( 'a-record', {
store: false
} );
Hey there, you guys are doing a great job. I just curious what does deepstream.io at under the hood difference with socket.io or faye?
If a client disconnects under extremely heavy load (bug was found at > 1,000,000 updates /sec) the following race condition can occur
c:\dev\deepstream.io-performance\high-troughput\node_modules\deepstream.io\src\utils\subscription-registry.js:42
this.subscriptions[ name ][ i ].send( msgString );
^
TypeError: Cannot read property 'send' of undefined
at SubscriptionRegistry.sendToSubscribers (c:\dev\deepstream.io-performance\high-troughput\node_modules\deepstream.i
o\src\utils\subscription-registry.js:42:36)
at RecordHandler.$broadcastUpdate (c:\dev\deepstream.io-performance\high-troughput\node_modules\deepstream.io\src\r
ecord\record-handler.js:264:30)
at RecordTransition._onCacheResponse (c:\dev\deepstream.io-performance\high-troughput\node_modules\deepstream.io\src
\record\record-transition.js:242:23)
at LocalCache.set (c:\dev\deepstream.io-performance\high-troughput\node_modules\deepstream.io\src\default-plugins\lo
cal-cache.js:8:2)
at RecordTransition._next (c:\dev\deepstream.io-performance\high-troughput\node_modules\deepstream.io\src\record\rec
ord-transition.js:224:22)
at RecordTransition._onRecord (c:\dev\deepstream.io-performance\high-troughput\node_modules\deepstream.io\src\record
\record-transition.js:171:8)
at RecordRequest._onCacheResponse (c:\dev\deepstream.io-performance\high-troughput\node_modules\deepstream.io\src\re
cord\record-request.js:55:8)
at LocalCache.get (c:\dev\deepstream.io-performance\high-troughput\node_modules\deepstream.io\src\default-plugins\lo
cal-cache.js:12:2)
at new RecordRequest (c:\dev\deepstream.io-performance\high-troughput\node_modules\deepstream.io\src\record\record-r
equest.js:32:22)
at RecordTransition.add (c:\dev\deepstream.io-performance\high-troughput\node_modules\deepstream.io\src\record\recor
d-transition.js:127:25)
Example code:
// Client A
var record = deepstream.record.getRecord('user/johan');
record.set('testValue', 1);
record.set('testValue', 1);
// Client B
var record = deepstream.record.getRecord('user/johan');
record.subscribe('testValue', function (newValue) {
console.log('Value changed to: ' + newValue);
});
The subscription on client B will only be fired once even though the value is set twice. Is this some kind of implementation to check that the new value set to a record is different then the one that is already set?
Create a deepstream.io client for Android, that connects to deepstream via TCP and supports Authentication, Permissioning, Records, Events and RPCs. Additionally, for mobile it might make sense to emphasize on offline storage during connection drops. Also worth noting #66 (deepstream client for Java)
Great resources to get started:
Writing a deepstream client
Deepstream Messaging structure
Detailed Cucumber Specs for Incoming and Outgoing messages (and associated behaviors)
Fully featured NodeJS Client as a reference
I have found a race condition in _checkClosed where the close
event from the _tcpEndpoint
fires before the callback for _server.close
. The result is that deepstream.io server
will never emit close
.
This can easily be resolved with Promise.all, but I do not see the use of promises in this file.
The only other solution I can think of is to have the _server.close
callback also call _checkClosed
.
TypeError: Cannot read property 'logger' of null
at RecordDeletion._done (/ds-server/node_modules/deepstream.io/src/record/record-deletion.js:61:15)
at RecordDeletion._checkIfDone (/ds-server/node_modules/deepstream.io/src/record/record-deletion.js:49:8)
at try_callback (/ds-server/node_modules/deepstream.io-cache-redis/node_modules/redis/index.js:592:9)
at RedisClient.return_reply (/ds-server/node_modules/deepstream.io-cache-redis/node_modules/redis/index.js:685:13)
at ReplyParser.<anonymous> (/ds-server/node_modules/deepstream.io-cache-redis/node_modules/redis/index.js:321:14)
at ReplyParser.emit (events.js:107:17)
at ReplyParser.send_reply (/ds-server/node_modules/deepstream.io-cache-redis/node_modules/redis/lib/parser/javascript.js:300:10)
at ReplyParser.execute (/ds-server/node_modules/deepstream.io-cache-redis/node_modules/redis/lib/parser/javascript.js:189:22)
at RedisClient.on_data (/ds-server/node_modules/deepstream.io-cache-redis/node_modules/redis/index.js:547:27)
at Socket.<anonymous> (/ds-server/node_modules/deepstream.io-cache-redis/node_modules/redis/index.js:102:14)
at Socket.emit (events.js:107:17)
at readableAddChunk (_stream_readable.js:163:16)
at Socket.Readable.push (_stream_readable.js:126:10)
at TCP.onread (net.js:538:20)
I'd like to explore deepstream using rethinkdb for storage. There are a few aspects of rethinkdb that make it really appealing: support for joins, rich queries with ReQL, and changefeeds. Thinky takes these concepts to the next level by allowing you to elegantly define your models with relationships between tables, and even validate your data.
Since there are some overlapping concepts with the deepstream ecosystem (i.e. records and search), I'm wondering if there's an elegant approach to having the best of both worlds? I've been looking into creating a custom version of the rethink storage connector that would use thinky instead, and then hooking in the validation somewhere before it is propagated via cache + message bus.
But there are some gotchas. For instance, records are not stored in raw format, but rather { _v: 1, _o: {}, _d: {}, }
which adds complexity for building out proper models and their relationships. Changing the format will no doubt have other implications throughout deepstream, which I'm still investigating.
Do you have any advice or recommendations for this approach?
I have a sample setup consisting of:
testfn1
and testfn3
(functionhost1.js)testfn2
and testfn3
(functionhost2.js)testfn1
, testfn2
, and testfn3
(client1.js)Simulatenously, I run node conductor1.js
node conductor2.js
node functionhost1.js
and node functionhost2.js
and all the connections are successfully established.
When I run client1.js
the first time, it successfully finds the functions on both function hosting deepstream clients. When I run it any subsequent times, it only successfully runs the functions from the deepstream host it is connected to, and gets an ACK timeout for the function running on the other deepstream function-hosting server.
Output from first run:
› node client1.js
Processed testfn1. Response Any Name processed
Processed testfn3. Response Any Name processed
Processed testfn1. Response Any Name processed
Processed testfn3. Response Any Name processed
Processed testfn2. Response Any Name processed
Processed testfn2. Response Any Name processed
Output from second and subsequent runs:
› node client1.js
Processed testfn1. Response Any Name processed
Processed testfn3. Response Any Name processed
Processed testfn1. Response Any Name processed
Processed testfn3. Response Any Name processed
ACK_TIMEOUT
Processed testfn2. Response undefined
ACK_TIMEOUT
Processed testfn2. Response undefined
I have provided a test repository with all the code neccessary to reproduce this issue. This can be accessed at https://github.com/petemill/debug-issuereproduce-deepstream-distributedcalls.
I am running node v4.2.1
Thanks
Hi there. Deepstream is really fantastic. I am curious if you'd ever considered using Rx.js and observables or at least adding some helper methods for people that would like to get an observable back instead of using callback and having to wrap then using Rx.js's .fromCallback method.? If not, no worries and thanks for the great framework.
In deepstream version 0.3.1:
When I rewrite a record message inside canPerformAction, the rewritten record data is seen by all clients except for the originating client. The only way to see the updated record is for the originating client to login again. I've also tried without the redis cache and message connectors.
For example, I'm inserting username into a record inside of canPerformAction. Regardless of whether I subscribe to record changes or not, I still don't see the username on the originating client.
Context:
https://github.com/philmaker1/deepstream.io-starter/blob/master/server/scripts/permission/RecordPermissionHandler.js#L35
https://github.com/philmaker1/deepstream.io-starter/blob/master/client/scripts/Pages.js#L400
Create a deepstream.io client for iOS (Swift / C#), that connects to deepstream via TCP and supports Authentication, Permissioning, Records, Events and RPCs. Additionally, for mobile it might make sense to emphasize on offline storage during connection drops.
Great resources to get started:
Writing a deepstream client
Deepstream Messaging structure
Detailed Cucumber Specs for Incoming and Outgoing messages (and associated behaviors)
Fully featured NodeJS Client as a reference
The current API appears to only allow individual records to be loaded. If you want to run a search for instance you create a dynamic list which returns the record ids which you then use to load the records individually. Is there any way to do this in one step?
The particular use case I'm thinking of is relationships between records, ideally I'd like to query rethinkdb for related records and have them all returned in one go and then subscribe to any changes.
A great setup I'm using a lot is letting the Rethinkdb changefeed update Elasticsearch which in turn is used for reads/querying only.
It would be great to integrate the above with deepstream somehow, so the frontend could do all sorts of advanced querying.
Thoughts?
Create a deepstream.io client for Java, that connects to deepstream via TCP and supports Authentication, Permissioning, Records, Events and RPCs.
Great resources to get started:
Writing a deepstream client
Deepstream Messaging structure
Detailed Cucumber Specs for Incoming and Outgoing messages (and associated behaviors)
Fully featured NodeJS Client as a reference
Mongo Db and? Maybe couch? or cassandra?
Currently, it appears as though deepstream builds the engine.io server manually, as opposed to constructing server and binding it to engine.io. This seems to prevent the ability to attach SSL certificates to the connection.
Using deepstream over HTTPS and WSS connections is required for us, as most browsers block insecure connections when the site is served over HTTPS.
Can HTTPS and WSS compatibility be implemented?
Hi
docs are really no helping when you actually try to play with APIs :(
Like in this case when trying to getRecord() from a nodeJS REPL
> var record = ds.record.getRecord( 'someUser' );
undefined
> --- You can catch all deepstream errors by subscribing to the error event ---
Error: ACK_TIMEOUT: someUser (R)
...
Experimentally I got to this point
ds.on('error', function(err) { console.log(err) });
And now I'm getting someUser
as an error. This isn't very useful, unless you try to print out the arguments
object to get more info about params.
No way to know smth went wrong while waiting for 'ready' event on record
record.whenReady(function(record) {...}); // also no info in docs on callback params, need to guess
AnonymousRecord is confusing too.
Docs are saying that An anonymous record has all the methods, events and properties of a normal record plus a setName() method.
.
Although I get this in node REPL:
> var record = ds.record.getAnonymousRecord();
undefined
> record.whenReady(function(record) { console.log(record.get()); });
TypeError: Object [object Object] has no method 'whenReady'
....
> record.set( {hi: 'there'} );
Error: Can`t invoke set. AnonymousRecord not initialised. Call setName first
Purpose of AnonymousRecord seemed clear to me until I got these errors.
I thought it is a holder for a bunch of values(like form fields input) but with no direct mapping to underlying backend(until everything is validated and I can do setName
with some value that makes sense).
Seems like I was wrong :\
What would be needed for making undo'ing commands work in deepstream?
I'm using the same client/server code but it doesn't behave the same way in case of a TCP connection.
var DeepstreamServer = require( 'deepstream.io' ),
server = new DeepstreamServer();
// Optionally you can specify some settings, a full list of which
// can be found here http://deepstream.io/docs/deepstream.html
server.set( 'host', 'localhost' );
server.set( 'port', 6020 );
// Set host and port for TCP connections
server.set( 'tcpHost', 'localhost' ); //default 0.0.0.0
server.set( 'tcpPort', 6021 ); //default 6021
server.set( 'permissionHandler', {
isValidUser: function( handshakeData, authData, callback ) {
console.log('auth now');
if( authData.username === 'rajan' ) {
callback( null, authData.username );
} else {
callback( 'Invalid user' );
}
},
canPerformAction: function( username, message, callback ) {
callback( null, true );
}
});
// start the server
server.start();
Client HTML code below:
<head>
<script
type="text/javascript"
src="node_modules/deepstream.io-client-js/dist/deepstream.js">
</script>
</head>
<body>
<input type="text" />
<script type="text/javascript">
//ds = deepstream( 'localhost:6021' ).login();
ds = deepstream( 'localhost:6020' ).login({
username: 'rajan',
password: 'sesame'
}, function( success, errorCode, errorMessage ){
//...
console.log(success);
});
record = ds.record.getRecord( 'someUser' );
input = document.querySelector( 'input' );
input.onkeyup = function(){
record.set( 'firstname', input.value );
};
record.subscribe( 'firstname', function( value ){
input.value = value;
});
</script>
</body>
The error it shows in browser console:
Uncaught Error: ACK_TIMEOUT: someUser (R)Client._$onError @ deepstream.js:6470RecordHandler._onRecordError @ deepstream.js:8408Emitter.emit @ deepstream.js:2549Record._onTimeout @ deepstream.js:8996setTimeout (async)Record @ deepstream.js:8482RecordHandler.getRecord @ deepstream.js:8266(anonymous function) @ index.html:23
Am I missing something or need to handle a situation?
I have the RethinkDB-connector activated. If I set a field to null with Deepstream and then tries to get it, it returns an empty object instead of null.
Example:
var record = deepstream.record.getRecord('user/1234');
record.whenReady(function () {
console.log(record.get()); // Output 1
record.set('field', null);
record.discard();
});
var record2 = deepstream.record.getRecord('user/1234');
record2.whenReady(function () {
console.log(record2.get()); // Output 2
console.log(record2.get('field')); // Output 3
record2.discard();
});
Output 1
{id: 1234, field: 'aField', field2: 'anotherField'}
Output 2
{id: 1234, field: null, field2: 'anotherField'}
Output 3
{}
After calling server.stop, and before stopped
event, all clients are disconnected.
Some clients, however, immediately attempt to reconnect, and successfully do connect.
This prevents the server's stopped
event from firing, and the process needs to be killed manually.
Any suggestions to resolve this issue?
The JSDoc above Record.prototype.whenReady
have the @private
-tag. It doesn't make any difference when using the library but the documentation have the function listed ( http://deepstream.io/docs/record.html#whenReady( callback ) ) (btw, you should really fix the ancors so it's possible to link to them directly ;)) and my IDE (WebStorm if that matters) highlights every whenReady()
usage since it's listed as private.
I'm curious if there's any plans to make the transport layer pluggable with other websocket implementations besides engine.io. I played around with swapping in https://github.com/websockets/ws, but had trouble since there are parts of deepstream that seem very coupled with engine.io, especially around connection semantics.
Create a deepstream.io client for .NET (e.g. in C#), that connects to deepstream via TCP and supports Authentication, Permissioning, Records, Events and RPCs.
Great resources to get started:
Writing a deepstream client
Deepstream Messaging structure
Detailed Cucumber Specs for Incoming and Outgoing messages (and associated behaviors)
Fully featured NodeJS Client as a reference
Create a deepstream.io client for Rust, that connects to deepstream via TCP and supports Authentication, Permissioning, Records, Events and RPCs.
Great resources to get started:
Writing a deepstream client
Deepstream Messaging structure
Detailed Cucumber Specs for Incoming and Outgoing messages (and associated behaviors)
Fully featured NodeJS Client as a reference
Merge changes on top of incoming update and re-issue
What if both changes are for the same path? Set merge strategy as option? On Record or on client level?
Kudos for building first truly real-time framework using RethinkDB. A great initiative!
It'd be great to be able to easily build mobile apps using Firebase/Parse like SDKs.
What would be best approach to get started on this?
Setting up WebRTC calls is still a bit of a challenge- it requires a signalling server and a number of explicit steps on the client side. I'd like to abstract all of this away into a simple API that looks like that
/**
* Register this client as a callable entity
*
* @param {String} calleeName
* @param {Function} onCall callback for incoming calls,
* will be called with a response object
*/
ds.webrtc.register( 'userA', function onCall( remoteStream, respone ){
response.accept( localStream );
// or
response.decline( reason );// optional reason
});
/**
* Make a call to a registered callee
*
* @param {String} calleeName
* @param {MediaStream} localStream
*/
ds.webrtc.call( 'userB', localStream, function( error, remoteStream ) {
});
Very good point made by @msoliter:
Let's add
server.set( 'tcpServerEnabled', true );
server.set( 'webServerEnabled', true );
both defaulting to true.
Create a deepstream.io client for Ruby, that connects to deepstream via TCP and supports Authentication, Permissioning, Records, Events and RPCs.
Great resources to get started:
Writing a deepstream client
Deepstream Messaging structure
Detailed Cucumber Specs for Incoming and Outgoing messages (and associated behaviors)
Fully featured NodeJS Client as a reference
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.