For our visualizations, we use data from Untappd - an online tool for keeping track of past beers consumed. The dataset has geographical, cardinal and ordinal parameters. We visualized some of the attributes in this data and formed a dashboard - comprising of a geographical map representation, a bar graph and a pie chart. Each of these charts are interlinked with the others - supporting brushing and linking. This allows interactive exploration of the dataset by filtering on different dimensions, in a visual manner.

The visualizations were developed using the popular web-based visualization library, D3, and a dimensional charting library based on D3, DC.js. To capture and track colors from a webcam feed, we used a javascript-based computer vision library, tracking.js. We then extended this library to improve color tolerance, and to work with the gesture recognition subsystem. Gesture recognition is primarily achieved with the help of University of Washington’s $P recognizer, which we trained to recognize shapes generated by color tracking, and fire corresponding interaction events.

Depending on which of the three colors currently tracked, the system is either in a pointing mode, click-action mode, or a gesture mode. While the color mapping is not fixed and the user has a simple method to change the mapping of colors, we have provided a set of 3 default colors in our system. Feedback from the user-interface in response to a user action is critical, especially since the interaction is not through direct on-screen manipulation. To assist pointer guidance, we use a cursor (depicted by a circle) which moves in a coordinate system that is linearly scaled with respect to the user’s color movement. Since this cursor tries to emulate the functionality of a mouse cursor, we had to implement code for performing picking (choosing underlying components from a pair of coordinates), and events such as hover and clicks. Gesture articulation is also tied to visual feedback on screen, as the user traces a shape. The way these gestures work is that the user needs to be above the chart/ graph that he wants to execute the gesture on using the mouse cursor. Once the user executes the gesture, it will call the determined interaction for the particular graph. If the graph has that event listener enabled it will execute the gesture, otherwise it will just return the execution feedback. A gesture is completed when the user switches off from the gesture state to any other state. The tracked shape, after filtering and smoothing, is passed to the $P recognizer for classification.

$P recognizer’s existing training data was removed, and custom shapes for our application were used to train the recognizer. While choosing shapes for gestures, we tried to not use shape pairs that might have a high rate of false-positives or misrecognition -- such as a circle and a square. We also tried to use shape pairs that have opposite orientations or connotations, to signify reversible interactions. For example, zoom-ins and zoom-outs are represented by an upward caret and a downward caret, respectively.