sociam / indx Goto Github PK

Strange bug where if you have a single property in an object and you delete all of its values (regardless of which property it is) and UPDATE (commit), then the object disappears from the box entirely

Can be reproduced by taking any object viewed in libtest
deleting all of the values of a property
refreshing
-> object's gone! :(

Max

When I use the diff operation, either through the client or through REST, it doesn't return anything useful:

python bin/client.py http://localhost:8211 webbox webbox --box indxtest --from 321 --to 322 --return_objs ids diff
INFO:root:diff in box indxtest successful, return is: { u'code': 200, u'message': u'OK'}

Or:

python bin/client.py http://localhost:8211 webbox webbox --box indxtest --from 121 --to 322 --return_objs objects diff
INFO:root:diff in box indxtest successful, return is: { u'code': 200, u'message': u'OK'}

(and doing --return_objs diff gives a server error:

CONN DEBUG 2013-09-30 15:41:10,707 REQ:4323581608TOKEN:592fae99-29de-11e3-b8b1-28cfe914080b ObjectStore _get_diff_combined, from_version: 321, to_version: 322 Unhandled error in Deferred: Unhandled Error Traceback (most recent call last): File "/Users/dmrust/git/indx/env/lib/python2.7/site-packages/twisted/internet/defer.py", line 575, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/Users/dmrust/git/indx/lib/indx/objectstore_async.py", line 750, in ver_cb result_d.callback(version) File "/Users/dmrust/git/indx/env/lib/python2.7/site-packages/twisted/internet/defer.py", line 380, in callback self._startRunCallbacks(result) File "/Users/dmrust/git/indx/env/lib/python2.7/site-packages/twisted/internet/defer.py", line 488, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/Users/dmrust/git/indx/env/lib/python2.7/site-packages/twisted/internet/defer.py", line 575, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/Users/dmrust/git/indx/lib/indx/objectstore_async.py", line 562, in <lambda> self._get_latest_ver().addCallbacks(lambda ver: got_versions(to_version, ver), err_cb) File "/Users/dmrust/git/indx/lib/indx/objectstore_async.py", line 541, in got_versions self._get_diff_combined(from_version, to_version_used).addCallbacks(lambda combined_diff: diff_cb(to_version_used, combined_diff, latest_ver), err_cb) File "/Users/dmrust/git/indx/lib/indx/objectstore_async.py", line 431, in _get_diff_combined versions = range(from_version+1, to_version+1) exceptions.TypeError: cannot concatenate 'str' and 'int' objects

e.g. SPARQL-like syntax mixed with mongo-like syntax.

to find people who have "dan" as a friend.

{
{"type": "person",
"friend": "?x"
},
{"@id", "?x",
"name": "dan"
}
}

All of a user's files should be put together so it can be backed-up, moved, nuked, etc in a sensible way.

Suggested approach is to have Postgres create all box stores under a subdirectory

Defaults:
Linux: ~/.webbox
OS X : ~/Documents
Windows : My Documents

As subject, see screenshot, which misses a couple of my boxes.

WebSockets doesn't return error codes when token is missing

See subject

OpenID works, but needs to auth as it.

Important that someone has to create a new DB account ahead of time, via some API and interface, then the user can auth with it, and login properly.

In objectstore_async a call to get a list of all object URIs.

Need to modify the data/objectstore*.sql scripts so that they are more general - specifically if you try to use a postgres user other than 'webbox' they fail.

All apps should go into webbox/apps, not webbox/html/apps and have the correct directory structure, and use the appropriate new class labels.

..and in python library, raise an exception.

python to do this is in devtools/init.py and use used to populate devtools list of apps

Client specifies a nonce in the request so that the response can be linked to the request.

Encapsulate the http request in JSON (there are examples of this), e.g.:

{
"query": "data=FOOBAR%20BAZ&blah=foobar",
"headers": {
"Cookie": "COOKIES"
}
"body": ""
}

etc.

Pull code from webbox/tests.py into a webbox client API.

Change tests to use that new API.

e.g.

python app.py --port 80 --db-username=webbox --db-password=webbox

Each WebSocket opens a new database connection with NOTIFY on a box - this can be optimised to a single NOTIFY connection per box which is republished to each subscriber.

Querying a box with an empty object ({}) causes a malformed SQL error (error in clause WHERE)

I think this might be a problem related to the opening too many connections to postgres, but basically after a while of debugging/hacking websockets quit working and the server has to be restarted.

We should fix this so that our demos don't spontaneously die :)

Example traceback
File "/home/electronic/webbox/env/local/lib/python2.7/site-packages/twisted/python/log.py", line 84, in callWithLogger
return callWithContext({"system": lp}, func, _args, *_kw)
File "/home/electronic/webbox/env/local/lib/python2.7/site-packages/twisted/python/log.py", line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, _args, *_kw)
File "/home/electronic/webbox/env/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, _args, *_kw)
File "/home/electronic/webbox/env/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
return func(args,*kw)
--- ---
File "/home/electronic/webbox/env/local/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 626, in _doReadOrWrite
why = selectable.doRead()
File "/home/electronic/webbox/env/local/lib/python2.7/site-packages/txpostgres/txpostgres.py", line 584, in doRead
_PollingMixin.doRead(self)
File "/home/electronic/webbox/env/local/lib/python2.7/site-packages/txpostgres/txpostgres.py", line 151, in doRead
self.continuePolling()
File "/home/electronic/webbox/env/local/lib/python2.7/site-packages/txpostgres/txpostgres.py", line 123, in continuePolling
state = self.pollable().poll()
psycopg2.OperationalError: asynchronous connection failed

Need support to create a new version based on adding a new object (or replacing an object?) while leaving the rest of the objects as in the current version.

Current API only allows for downloading entire graph, editing it, and replacing the whole thing, which doesn't scale.

Output the JS API documentation and host on the github.io homepage.

Instructions:

Generate docs:

cd docs
npm install
node build.js

Look at docs:

cd build
python -m SimpleHTTPServer 8080

i guess the environment's pythonpath has to be set to webbox/pywebbox/

it's confusing because it was looking at 'pywebbox' the directory as a
module. i think if we rename pywebbox -> client, and create
client/init.py then this will be less confusing

so then in twitter.py it would look like:
from client.pywebbox import WebBox

python bin/client.py http://localhost:8211 webbox webbox --box indxtest add_file --id my_test_file.txt --data ~/tmp/mytestfile.txt --contenttype text/text --version 342 There was a problem: HTTP Error 403: Forbidden

root DEBUG 2013-09-30 15:53:00,090 Calling base render() - /auth/login <class indx.webserver.handlers.auth.AuthHandler at 0x10fad5870> _ subhandlers::6 root DEBUG 2013-09-30 15:53:00,090 subhandler accept ['application/json'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,091 Handler match where path=login and method=POST root DEBUG 2013-09-30 15:53:00,091 subhandler accept ['application/json'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,091 Matching handlers 1 root DEBUG 2013-09-30 15:53:00,091 subhandler accept ['application/json'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,091 Post-auth matching handlers: [{'require_token': False, 'methods': ['POST'], 'accept': ['application/json'], 'prefix': 'login', 'handler': <unbound method AuthHandler.auth_login>, 'content-type': 'text/plain', 'require_auth': False}] root DEBUG 2013-09-30 15:53:00,091 Post-token matching handlers 1 root DEBUG 2013-09-30 15:53:00,091 Using handler AuthHandler login root DEBUG 2013-09-30 15:53:00,091 request args {'box': ['indxtest'], 'username': ['webbox'], 'app': ['INDX CLIClient'], 'password': ['webbox'], 'token': ['None']} root DEBUG 2013-09-30 15:53:00,094 Login request auth for webbox, origin: None root DEBUG 2013-09-30 15:53:00,095 Encoding response with cjson root DEBUG 2013-09-30 15:53:00,095 just called request.finish() with code 200 root DEBUG 2013-09-30 15:53:00,097 Calling base render() - /auth/get_token <class indx.webserver.handlers.auth.AuthHandler at 0x10fad5870> _ subhandlers::6 root DEBUG 2013-09-30 15:53:00,097 subhandler accept ['application/json'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,098 Handler match where path=get_token and method=POST root DEBUG 2013-09-30 15:53:00,098 subhandler accept ['application/json'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,098 Matching handlers 1 root DEBUG 2013-09-30 15:53:00,098 subhandler accept ['application/json'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,098 Post-auth matching handlers: [{'require_token': False, 'methods': ['POST'], 'accept': ['application/json'], 'prefix': 'get_token', 'handler': <unbound method AuthHandler.get_token>, 'content-type': 'application/json', 'require_auth': True}] root DEBUG 2013-09-30 15:53:00,098 Post-token matching handlers 1 root DEBUG 2013-09-30 15:53:00,098 Using handler AuthHandler get_token root DEBUG 2013-09-30 15:53:00,098 request args {'box': ['indxtest', 'indxtest'], 'app': ['INDX CLIClient', 'INDX CLIClient'], 'token': ['None']} root DEBUG 2013-09-30 15:53:00,098 indx_pg2: returning existing pool for db: wb_indxtest, user: webbox root DEBUG 2013-09-30 15:53:00,099 Encoding response with cjson root DEBUG 2013-09-30 15:53:00,099 just called request.finish() with code 200 root DEBUG 2013-09-30 15:53:00,101 Calling base render() - /indxtest/files <class indx.webserver.handlers.box.BoxHandler at 0x10f5b9530> _ subhandlers::8 root DEBUG 2013-09-30 15:53:00,101 subhandler accept ['*/*'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,101 Handler match where path=files and method=PUT root DEBUG 2013-09-30 15:53:00,101 subhandler accept ['application/json'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,101 subhandler accept ['application/json'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,101 Handler match where path=files and method=PUT root DEBUG 2013-09-30 15:53:00,101 subhandler accept ['application/json'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,101 subhandler accept ['application/json'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,101 Matching handlers 2 root DEBUG 2013-09-30 15:53:00,101 subhandler accept ['*/*'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,102 subhandler accept ['application/json'] /// requestaccept */* root DEBUG 2013-09-30 15:53:00,102 Post-auth matching handlers: [{'prefix': 'files', 'handler': <unbound method BoxHandler.files>, 'methods': ['GET', 'PUT', 'DELETE'], 'accept': ['*/*'], 'force_get': True, 'content-type': 'application/json', 'require_token': True, 'require_auth': False}, {'require_token': True, 'methods': ['PUT'], 'accept': ['application/json'], 'prefix': '*', 'handler': <unbound method BoxHandler.do_PUT>, 'content-type': 'application/json', 'require_auth': False}] root DEBUG 2013-09-30 15:53:00,102 _matches_token_requirements, force_get: True root DEBUG 2013-09-30 15:53:00,102 Verify token (verified=True) (00050821-29e0-11e3-b6e7-28cfe914080b) with boxid: indxtest and origin None, to request boxid: indxtest and request origin: None root DEBUG 2013-09-30 15:53:00,102 _matches_token_requirements, force_get: False root DEBUG 2013-09-30 15:53:00,102 Encoding response with cjson root DEBUG 2013-09-30 15:53:00,102 just called request.finish() with code 403

e.g. using http://blueimp.github.com/jQuery-File-Upload/

related to the js client

new version dynamically loads the toolbar which is pretty cool but
that call back has no deferred handler so it's impossible to know when it's done

client should load() and this should cause the toolbar to load too - and the then() callback of that should be called only when toolbar is done

E.g., in app devtools, a prefix would need to be "devtools/api/tests". Putting simply "tests" would make more sense.

It currently fails if a connection pool is open.

for the last object in a box, deleting fails to increment the version or actually delete the object.

this is a bug. :)

right now, objects have the following semantics

if they have no properties they don't exist
if a property has a blank array of things, then it doesn't exist

this creates loads of problems - when you start to surface this stuff to the user. user creates thing. thing disappears 'cause it doesn't have any properties yet.

users think of 'creating' something as a verb, without having to 'pin it down' with a property

i believe this is a design bug

currently a blank page is shown

Like the consumer, but provider too with INDX hostname as namespace.

instead of 404 :)

this is probably a bug.

max

Make a web page and diagrams of the architecture and key points, e.g.:

• Streaming
• Encryption
• Querying
• NewSQL
• Transactions
• Versioning
• Web Sharing
• Distributed ID

used to be angular-example