divya-gopinath / chatterbox Goto Github PK

View Code? Open in Web Editor NEW

Multimodal chat application supporting voice-to-text, emoji suggestions based on facial expressions, and hands-free scrolling with gaze tracking. Developed with Mary Thielking (forked from our private repo) for an MIT class.

JavaScript 22.87% HTML 1.75% Python 71.68% CSS 3.70%

chatterbox's Introduction

Chatterbox

What's Chatterbox?

Chatterbox is a multimodal networked chat interface with three core features:

Emoji suggestions based on emotion classification from webcam data
Hands-free scrolling using gaze tracking from webcam data
Voice-to-text functionality

Filename	Description
`chat.js`	Modifies `index.html` depending on the state of the application. Combines `speech.js`, `scroll.js`, and `logging.js` into the interface.
`chatServer.js`	Starts and runs a Node.js server using Express.
`emojiBank.js`	Contains a dictionary used to suggest emojis based on the emotion classification.
`index.html`	Initial layout of the interface. Modified using `chat.js`
`logging.js`	Logs desired mouse and keyboard events to a Google spreadsheet. The data was then processed after user testing.
`package.json`	Used by `npm` to keep track of dependencies.
`scroll.js`	Triggers scrolling up or down when gaze tracking is turned on and a cutoff is reached at the top or bottom of the screen.
`speech.js`	Processes speech using `webkitSpeechRecognition` API
`style.css`	Provides styling to the interface.
`WebGazer.js`	Sourced from https://webgazer.cs.brown.edu/ and used to implement gaze tracking.
`python_chat/`	Python 3 code from initial implementation. Filenames are similar to those described above.
`sounds/`	Sounds used to provide auditory feedback

Using the Chat

Setup

Install Node.js
Navigate to directory 6835-project
Run npm install

Running the Chat

Run npm start
Open a browser and go to localhost:3000

Or go to https://agile-citadel-17828.herokuapp.com/.

Runs on Chrome, Firefox, and Edge. Tested on Windows, Linux, and Mac. Works best in a well-lit area, with the laptop on a desk or table so that the webcam is closer to eye level than it would be from a user's lap.