Code Monkey home page Code Monkey logo

verse's Introduction

Verse 2.0

Travis CI Build Status Coverage Status Coverity Scan Build Status

Verse 2.0 is network protocol for real-time sharing of 3D data. It is successor of old Verse protocol developed at KTH. Verse 2.0 is still in alpha version.

License

The source code of Verse library is licensed under BSD license. This library could be used for implementation of Verse client. The source code of Verse server is licensed under GNU GPL 2.0 license. For details look at files BSD-LICENSE and GPL-LICENSE.

Important links

Compile

Compilation of Verse 2.0 is tested only on Linux now. Some libraries and development tools are required. Porting to other UNIX like OS should be possible. Support for Mac OS X and other BSD like UNIXese is only experimental.

Requirements

Building

To compile Verse server, libverse.so and example of Verse client open terminal, go to root of verse source code and type:

$ mkdir ./build
$ cd ./build
$ cmake ../
$ make
$ sudo make install

If you want to build debug version, then you have to run cmake with following parameter:

$ cmake -DCMAKE_BUILD_TYPE=Debug ../

If you want to build Verse with Clang, then you have to do more

$ export CC=/usr/bin/clang
$ export CXX=/usr/bin/clang++
$ mkdir ./build
$ cd ./build
$ cmake ../
$ make
$ sudo make install

Folders

  • ./config is directory with example of users.csv file
  • ./doc contains doxyfile for generated doxygen documentation
  • ./example contains source code of example Verse clients
  • ./include contains all .h files
  • ./pki contains example of certificate and private key
  • ./src contains source code
  • ./lib source code for library (used by client as well by server)
  • ./lib/api source code of API
  • ./lib/client source code specific for Verse clients
  • ./lib/common source code shared with Verse server and Verse client
  • ./server source code specified for Verse server
  • ./server/mongodb source code used for saving and loading data from MongoDB
  • ./python contains source code for Python module implemented in C
  • ./unittests contains source code of unit tests

Installation

The verse server and example of verse client is not necessary to install to the system. Before you want to run Verse server you should edit your "database" of users. Go to the ./config folder and edit users.csv:

$ cd ./config

and edit user.csv with your favorite text editor, e.g. vim:

$ vim users.csv

Using

The Verse server can be executed from build directory:

$ ./bin/verse_server

Example of Verse client could be executed from this directory too:

$ ./bin/verse_client localhost

The example of Verse client and Verse server can be started with several arguments. For more details run programs with '-h' option.

When you want to try test Verse client implemented Python, then you have to set up system variable PYTHONPATH to include directory containing Python module "verse.so" :

$ export PYTHONPATH=/path/to/directory/with/verse/module

Then it is possible to run this client:

$ python3 verse_client.py

MongoDB

It is possible to use Verse server without support of MongoDB, but all data are stored only in memory and when server is stopped, then all data are lost. For production purpose it is recommended to configure using MongoDB in server.ini file. Implementation of MongoDB is currently limited, because Verse server load all data to memory during start and saves all data to MongoDB, when server is stopped.

Firewalls

If you use firewall and you want to connect on Verse server, then you will need to open several ports. The Verse server listen on TCP port 12345 and it negotiate new UDP ports in range: 20000 - 20009 (this port numbers and port ranges will be possible to change in configuration files in the future).

When you use Linux OS, then you can use iptables for it. You should be familiar with iptables. If not, then read some documentation about iptables first. To open TCP port use something like this:

$ iptables -I INPUT 10 \ # this add this rule before 10th rule (change this!)
  -m state --state NEW \ # use this rule only for the first packet (optional)
  -s 1.2.3.4/16 \        # allow connection only from some subnet (optional)
  -p tcp --dport 12345 \ # own opening of TCP port 12345
  -j ACCEPT              # accept this packet

To open UDP port in range 50000 - 50009 use something like this:

$ iptables -I INPUT 11 \
  -m state --state NEW \
  -s 1.2.3.4/16 \
  -p udp --dport 50000:50009 \ # open UDP ports in range: 50000 - 50009
  -j ACCEPT

Verse server and all verse client can use IPv6. Configuration of ip6tables is very similar:

$ ip6tables -I INPUT 10 \
  -m state --state NEW \
  -p tcp --dport 12345 \
  -j ACCEPT

$ ip6tables -I INPUT 11 \
  -m state --state NEW \
  -p udp --dport 50000:50009
  -j ACCEPT

Verification

Main parts of Verse protocol were verified using Promela programming language and tool called Spin. For more details go to the directory ./misc/promela.

Testing

If you want to test Verse library, then you have to compile Verse with support for unit test framework called Check and then you can perform several unit tests. To run unit tests you have to go to the build directory and run make with target test:

$ cd ./build
$ make test

Contacts

  • IRC: irc.freenode.net in channel #verse
  • Mailing list: [email protected]
  • Main developer: Jiri Hnidek
  • E-mail: [email protected]
  • Phone: +420 485 35 3695
  • Address: Studentska 2, 461 17, Liberec 1, Czech Republic

verse's People

Contributors

jirihnidek avatar jirivrany avatar swiftcoder avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

verse's Issues

Callback Functions at Verse Server and Custom Commands

It could be useful to have support for callback function and some plug-ins (plug-ins written in C, Python ad-dons, etc.) at Verse server. Thus some code could be executed directly at Verse server. Callback functions at Verse server would not be so useful without custom commands. It means that client could define own custom commands to trigger callback at Verse server.

Better documentation for fundamental concepts.

I've made a new wiki page that you might want to use to slowly introduce people to verse, and verse concepts.

https://github.com/verse/verse/wiki/Overview

I tried to explain what's up with custom types already, and hopefully it's not too far from the truth.

What confuses me is the difference between tag groups and nodes. That might be a good place to start. A tag can only belong to one node, but multiple tag groups? Presuming I'm understanding correctly that's not a feature I'd bring to the forefront. I imagine that most of the time someone will want to subscribe to an entire node or just objects in the node, grouping them like that could be useful in some cases, but confuses the issue a fair bit.

It looks to me like you can't attach a tag directly to a node, it needs to go through a tag group? Is that right? I thought you could attach tags to nodes directly.... Looking at the tag command specification it looks like it accepts a node ID.

What use case do tag groups have? Or have I completly misunderstood how they work?

Once I've got that sorted out I'm going to work on explaining tags and nodes in a bit more depth.

Verse installs to unused library path under arch linux

Verse installs its libraries to "/usr/local/lib64/", but it doesn't look in that directory for them.

open("/usr/lib/tls/x86_64/libverse.so.0.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls/x86_64", 0x7fff39a45900) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/libverse.so.0.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls", 0x7fff39a45900)    = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64/libverse.so.0.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64", 0x7fff39a45900) = -1 ENOENT (No such file or directory)
open("/usr/lib/libverse.so.0.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib", {st_mode=S_IFDIR|0755, st_size=176128, ...}) = 0

QUIC as a transport layer for future Verse efforts?

I realise Verse may not be an active effort at this point, but I still feel there's a lot of potential in the realtime collaborative vision underlying Verse. In the intervening time since Verse 2.0 began, QUIC (and now HTTP/3 over QUIC) have attained or nearly attained the status of standards. In the presence of a widely implemented multi-stream UDP protocol that is designed to eliminate the same head-of-line-blocking issues as Verse 2.0, would it make sense to pivot Verse efforts in that direction?

Python Libarary issues

[alex@localhost build]$ sudo make install
[ 77%] Built target verse_shared_lib
[ 78%] Built target py2_verse
[ 79%] Built target py3_verse
[ 98%] Built target verse_server
[100%] Built target verse_client
Install the project...
-- Install configuration: "Release"
-- Up-to-date: /usr/local/include/verse.h
-- Skipping  : /etc/verse/users.csv
-- Skipping  : /etc/verse/server.ini
-- Up-to-date: /usr/lib64/python2.7/site-packages/verse.so
-- Up-to-date: /usr/lib/python3.4/site-packages/verse.so
-- Up-to-date: /usr/local/lib64/libverse.so.0.1
-- Up-to-date: /usr/local/lib64/libverse.so.0.1.0
-- Up-to-date: /usr/local/lib64/libverse.so
-- Up-to-date: /usr/local/bin/verse_server
Python 3.4.2 (default, Oct  8 2014, 13:44:52) 
[GCC 4.9.1 20140903 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import verse
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /usr/lib/python3.4/site-packages/verse.so: undefined symbol: vrs_get

Let me know if there's anything else you need.

Verse server crashes during start

When Verse server starts and it can't load config file, then it crashes with following backtrace:

Core was generated by `verse_server'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f52c4312e9d in v_list_free (listbase=0x7fff9fd32a80) at /home/jiri.hnidek/GitHub/verse/src/common/v_list.c:221
221         next = item->next;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6_4.5.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.6.x86_64 libcom_err-1.41.12-14.el6_4.4.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 openssl-1.0.0-27.el6_4.2.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007f52c4312e9d in v_list_free (listbase=0x7fff9fd32a80) at /home/jiri.hnidek/GitHub/verse/src/common/v_list.c:221
#1  0x00007f52c431434c in v_hash_array_destroy (hash_array=0x7fff9fd32a80) at /home/jiri.hnidek/GitHub/verse/src/common/v_list.c:918
#2  0x0000000000411cc7 in vs_destroy_ctx (vs_ctx=0x7fff9fd328d0) at /home/jiri.hnidek/GitHub/verse/src/server/vs_main.c:269
#3  0x000000000041223a in main (argc=1, argv=0x7fff9fd32cd8) at /home/jiri.hnidek/GitHub/verse/src/server/vs_main.c:448

Deleting of nodes and trash node

When client wants to destroy/delete node, then this node can't have any child node. It isn't useful, because this approach doesn't use benefits of tree structure. It would be better to allow to destroying nodes with child nodes and nodes with different owners would be moved to the trash node. Every user node will include trash node.

Weird unicode issues

Tested with verse-entities, hopefully latest verse, presuming my enviroment problems have actually been sorted out.

This implies some kind of potential security issue? Or at least the ability to put random control characters into the log.

DEBUG: New connection from: [::ffff:127.0.0.1]:38126 
DEBUG:  +4s     Server listen on TCP port: 12345
DEBUG: SSL handshake succeed.
DEBUG: Server TCP state: RESPOND_methods
DEBUG: Receive message: Socket: 4, [::ffff:127.0.0.1]:38126 Ver: 1, Len: 12
        USER_AUTH_REQUEST, Username: ���← T≤⎻e: N⎺┼e

DEBUG: Se┼d └e⎽⎽▒±e: S⎺c┐e├: 4← [::°°°°:127↓▮↓▮↓1]:38126 Ve⎼: 1← Le┼: 7
        USER_AUTH_FAILURE← Me├▒⎺d⎽: P▒⎽⎽┬⎺⎼d← 

DEBUG: Se⎼┴e⎼ TCP ⎽├▒├e: RESPOND_┤⎽e⎼▒┤├▒
WARNING: Le┼±├▒ ⎺° c⎺└└▒┼d: 9 ☃⎽ ⎽└▒┌┌e⎼ ├▒e┼: (1→1→1→┤⎽e⎼┼▒└e_┌e┼:4→1→1→1)↓
DEBUG: Rece☃┴e └e⎽⎽▒±e: S⎺c┐e├: 4← [::°°°°:127↓▮↓▮↓1]:38126 Ve⎼: 1← Le┼: 13
        USER_AUTH_REQUEST← U⎽e⎼┼▒└e: ← T≤⎻e: Re⎽e⎼┴ed

DEBUG: Se⎼┴e⎼ TCP ⎽├▒├e: CLOSING
DEBUG: T⎼≤ ├⎺ ⎽▒┤├ d⎺┬┼ SSL c⎺┼┼ec├☃⎺┼↓
DEBUG: SSL c⎺┼┼ec├☃⎺┼ ┬▒⎽ ⎽▒┤├ d⎺┬┼↓
DEBUG: /▒⎺└e/▒┌e│/┴e⎼⎽e/⎽⎼c/⎽e⎼┴e⎼/┴⎽_▒▒┼d⎽▒▒┐e↓c:212 c┌⎺⎽e(4)
DEBUG: Se⎼┴e⎼ TCP ⎽├▒├e: CLOSED
DEBUG: ┴⎽_┼⎺de_de⎽├⎼⎺≤_▒┴▒├▒⎼_┼⎺de(): ▒┴▒├▒⎼ ┼⎺de (☃d: ↑1) ┼⎺├ °⎺┤┼d
DEBUG: Se⎼┴e⎼ TCP ⎽├▒├e: LISTEN
DEBUG:  →5⎽     Se⎼┴e⎼ ┌☃⎽├e┼ ⎺┼ TCP ⎻⎺⎼├: 12345
^CDEBUG: S├⎺⎻⎻☃┼± ⎽e⎼┴e⎼
DEBUG: C▒┼ce┌☃┼± c┌☃ ├▒⎼e▒d
DEBUG: W▒☃├ °⎺⎼ Se⎼┴e⎼ ⎽├▒├e CLOSED
DEBUG: N⎺de: 1 de⎽├⎼⎺≤ed
DEBUG: N⎺de: 1▮▮ de⎽├⎼⎺≤ed
DEBUG: N⎺de: 65535 de⎽├⎼⎺≤ed
DEBUG: N⎺de: 1▮▮▮ de⎽├⎼⎺≤ed
DEBUG: N⎺de: 1▮▮1 de⎽├⎼⎺≤ed
DEBUG: N⎺de: 1▮▮2 de⎽├⎼⎺≤ed
DEBUG: N⎺de: 1▮▮3 de⎽├⎼⎺≤ed
DEBUG: N⎺de: 1▮▮4 de⎽├⎼⎺≤ed
DEBUG: N⎺de: 1▮▮5 de⎽├⎼⎺≤ed
DEBUG: N⎺de: 2 de⎽├⎼⎺≤ed
DEBUG: N⎺de: 3 de⎽├⎼⎺≤ed
DEBUG: N⎺de: ▮ de⎽├⎼⎺≤ed

import os
def main():
    [REDACTED]

if __name__ == '__main__':
    main()

Session_new(): type: 0x126ffc0, args: 0x7f38783f8050, kwds: 0x7f386be9aa28
--------
CONNECTING
default cb_receive_user_authenticate(session_id: 0, args: ('', 'root'))
default cb_receive_user_authenticate(session_id: 0, args: ('\x0e\xf5\xf9\xff', 'root'))
default cb_receive_connect_terminate(session_id: 0, args: (4,))
DISCONNECTED

Python 2.x

It is possibility to use only Python 3 now. CMake build system should be modified to build Python module for Python 2.x too.

NoSQL backed of Verse server

Verse server store everything in memory. It isn't reliable solution. It will be necessary to add some back-end for storing data on disk. It will be MongoDB database.

  • Save all data, when server is stopped.
  • Load data from MongoDB, when server is started.
  • Make all code handling nodes at Verse server thread save.
  • Create new thread with low priority used for saving current version of data to MongoDB
  • Do not keep all data in memory, but load it to memory only in situation, when data are requested or changed.

Make OpenSSL optinal

Make dependency at OpenSSL library optional. It will be the first step of replacing OpenSSL with GnutTLS library. The nice side effect of removing OpenSSL will be possibility to use valgrind to discover memory leaks and other errors. It is almost impossible with OpenSSL, because OpenSSL is full of evil code.

Sharing of UDP sockets at Verse server

Verse server currently uses one UDP socket per connection. Verse server by default opens 10 UDP sockets and thus Verse server can handle only 10 clients. It would be wise to have possibility to share UDP socket with more connection, when many clients are connected.

Support for WebSocket at Verse Server

Add support for WebSocket. Verse server will be able to communicate with Web Clients.

There are several libraries that could be used for implementation:

There is also comparison of WebSockets implementations at Wikipedia:

Use GnuTLS

Use GnuTLS as optional package for security tasks (TLS, SHA, etc.)

Support for sandboxed pypy

Being able to run verse in a sandboxed python install has some obvious uses. Couple it with something that connects to verse servers and downloads python scripts and you have a very easy very flexible application delivery platform.

Support for Versing

The API for versing is almost complete and implemented at client, but Verse doesn't do any versing.

Optional Command Compression

Verse client and Verse server has to command compression now. This should be optional, becuase it is not trivial to implement.

Support for Big Data

Add support for better handling of big data. When some client is subscribed to large layer, then verse server puts all items from this layer to the outgoing queue in one step. Thus the size of outgoing queue can grow to big size. Similar problem is at client, when huge data is uploaded to server. There is no mechanism to limit size of data in outgoing queue.

  • Implement uploading big data from server
  • Expose free space of incoming queue to flow control
  • Create API for getting and setting size of in/outgoing queues
  • Drop commands added by client to outgoing queue, when queue is full.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.