jivid / akobi Goto Github PK

Turns out we can't use the @tornado.web.asynchronous decorator on code that doesn't subclass tornado.web.RequestHandler, which means we need another way to make sure our event handlers can run in parallel.

I propose a mix of Python's threading and multiprocessing modules along with some built-in Tornado capabilities.

• Threading tutorial
• Multiprocessing + Tornado

What do you guys think?

registry.py is still using logging. We need to change that to from akobi import log

We're throwing data into redis all over the place now and it the key namespaces aren't organized at all. We should target a refactor for this in the next set of features

• Add metadata to apps (description, collaborative/private etc.)
• UI for main page

Currently Event handlers handle all messages of a specific type, but it would probably be nicer if the event handler had an instance for each interview. This way within the handler you wouldn't have to worry about which interview you were dealing with.

My current workaround has passing the current interview into the handle function and saving a dictionary mapping this interview to its associated data.

Currently we just used a random string, this is probably not the best solution.

@jbrick, the collabedit change you pushed on Friday still pulls our javascript resources on interview.html from 127.0.0.1 and not from akobi.info. Can you please push a fix for this immediately? Let me know as soon as you do and I'll merge it in.

We need some kind of status indicator to indicate whether we are online / offline, as well as who is currently in the interview.

If we want to do some testing we should setup a system for defining the different message types we pass around and generating mocks for them. Allowing for server / client testing.

On the interview creation and app selection screen the user should be able to drag and drop the different modules they desire. As well as the ability to size and place them.

At the moment there's a race condition that exists and has been addressed through the use of an init_finished message.

1. Client connects and sends initialization message
2. If client sends application data prior to server completing app instantiation we crash

This workflow should be reviewed and examined in more detail.

Currently video and collabedit application don't support person leaving and rejoining should fix this.

From the redis-py source, it looks like redis-py's ConnectionPool has a limit on max_connections set at 2^31 by default. Maybe we need to set a smaller limit ourselves. This is a little pre-emptive and we should re-assess when we have some more information about our overall Redis usage.

We should move the images from the meetings to another folder, maybe dropbox? They're currently already 8MB, which also gets added to the .git folder, making the whole repo 16MB. Github allows only 1GB repo sizes and the speed at which we're going, we will probably hit that quite fast

Will need to revisit how frequently client is sending heartbeats as we continue to develop the application.

Currently, heartbeats are only processed on the server by recording a client's most recent heartbeat timestamp and sending an empty response back to them.

The following functionality should be added to the heartbeat:

• records number of heartbeats for each client in Redis
• enforces a timeout value where, if a heartbeat is not received in that amount of time from a client, the server closes the web socket to that client

In order to meet 100% of our essential requirements we need a mechanism to save notes locally.

1 liner

We need a turn server hosted for relaying data when nat traversal is not possible.

The collabedit state machine has a lot of boiler plate code right now, especially to send invalid message errors out if the states are wrong. We should be able to make the entire state machine a lot cleaner, maybe with an abstracted out state machine class and decorator-based state transitions so that writing an app with states like collabedit becomes much easier.

The message_type_to_handler function should add 'Handler' to the end of the handler name so it can be used directly to query the registry

Each application should have a method called when the interview is finished. Such that the application can do stuff on interview finished such as saving a video etc.

@jbrick can you fix that in your next pull request.

Currently to use an application we have to import it in handlers/interview.py we should do this another way.

Currently we store the class in the registry and instantiate it every time in the handler.onMessage. We should update this to just store one instance in the registry.

Right now, the logfile is created in the location where the script is run from, which leads to many stray logfiles around the file system. The script should create the file only in the location of the script, and not where the script is run from

We should also remove the line that logs every time we get a message from the client. Some of our applications like Collabedit send up tons of messages per second, which eats up disk space because of the logs. More than 50% of our logs right now are just saying "Received message..."

divij@fydp-akobi:/var/www/dev$ grep "Received message from" akobi_output.log | wc -l
16183
divij@fydp-akobi:/var/www/dev$ wc -l akobi_output.log
29077 akobi_output.log

Collabedit is acting very flaky in my testing on the ui branch right now. I'm seeing behaviour where the application only works ~30% of the time.

A brief recap of what actually happens in the collabedit loop on the server side:

1. Wait for two clients to join. Do nothing till then
2. Upon the second client joining, start the sync loop
3. Ask the first client for its diff
4. Send this diff down to the second client
5. Ask the second client for its diff
6. Send this diff down to the first client
7. Return to Step 3

The one issue with this already is that it doesn't handle any failures on either side via retries or otherwise. i.e. if one client fails to send up a diff, the loop stalls.

A second issue is with timing on the part of the second client. Basically, on the client side, each client sets up its CollabEditView and subsequently subscribes to a couple of events on the EventBus pertaining to the synchronization loop. The problem we're having is that the server has already completed step 5 of its loop and is waiting for the second client to send up the diff, but the second client hasn't finished subscribing to the events from the bus, so it's not able to respond with a diff, thereby halting our collabedit loop.

An interesting observation is that we don't have this issue in the current version of the code on develop. My hypothesis is that with the existing code, there is no extra rendering taking place for the textarea since it's directly entered in the HTML. In my ui branch, I have React rendering our textbox, which is much slower than the browser just parsing HTML and doing the render. This extra time leads to a delay in the CollabEditView and the event subscription.

To fix this, I propose some kind of collabedit setup, similar to the interview setup we have using the init_interview message type. Using this, the server can be absolutely sure that both clients are ready to send up their diffs before asking for them. Further, if we add some sort of retry mechanism, we can greatly improve the overall reliability of the system

Right off the bat the background should not be bright blue.

Unsure of the technical aspects of the flow that need to change. Will leave to @jivid to comment.

I would like to be a committer please. I have many popular BitBucket repos!

• Abstract interview to its own class
• Abstract socket to its own class
• Create relationship between interview and socket within interview class
• Verify message coming from sockets have a valid client and interview IDs
• Ensure all messages have client and interview IDs before sending to applications
• Add message validation to BaseApplication for other apps to use
• Delay assigning client ID till authentication has completed

We should think about rolling a wrapper around the websocket. Just an idea looking at #43

no need to make it fancy but it'd be nice to get a basic app of this knocked out

We need a text chat application allowing two users to chat. This will consist of both the backend and frontend work to implement it.

We need a standard method that builds a message in our created format

• needs to check that all required fields are present before allowing the message to be created
• need client side and server side method (JS and Python)
• format is as follows (JS):
  datetime: ,
  type: ,
  clientID: ,
  interviewID: interviewID>,
  data : {}

Every time setInterval is called, there needs to be a corresponding clearInterval on closing of the websocket as well. Maybe this process of setting and clearing an interval could be abstracted out to a utility function in utils.js?

I'm thinking something like:

// In utils.js
function periodicEvent(callback, interval) {
    var event = setInterval(callback, interval);
    EventBus.on('socket_closed', function() {
        clearInterval(event));
    });
}

// In notes.js
utils.periodicEvent(
    $.proxy(this.captureAndSync), this), 15000
);

• Add dropdown allowing user to change programming language (like Stypi)
• Write JS to treat cursor as a separate entity within the editor
• Sync cursors with both sockets to show opposite person's position in the editor

We want a way to get only non-essential apps from the registry. An essential app would be something that needs to be present for all interviews, like Heartbeat. This would help with code in the IndexHandler

As we build more of the registry and initializer, the interviews themselves are starting to carry more and more state. I think it makes sense to abstract out interviews to their own class so we can maintain state local to the interview as opposed to in global dicts and variables.

A perfect example of this is the _apps_instantiated variable we were toying around with in the initializer in #51. The initializer itself should be stateless so it can be called on every client connection, but at the same time we want to not re-instatiate apps for an interview. A flag for _apps_instantiated still makes the most sense, but it could just be on an instance of the interview class as opposed to on the initializer.

I fear if the interview-state logic is moved out of the initializer it will get forced into the registry, which isn't the right place for it either.

At the moment we're managing sockets/client connections in both collabedit.py and video.py.

It would be nice if we had an abstraction that both these applications could leverage to get access to a clients via socket.

Sakshi added server side logging levels, we need this in the client side as well. Eg heartbeats should only log at a debug level or console will just be full of heartbeats all the time.