Code Monkey home page Code Monkey logo

drewdevero / tri-sense Goto Github PK

View Code? Open in Web Editor NEW
1.0 1.0 0.0 33.22 MB

A hand motion tracking project: MVP - A dictionary look up app with returning word, pronunciation(written form), part of speech, and vocal file giving auditory example of pronunciation. Accessibility level considered adequate as determined by Axe Accessibility. Stretch goal - detect hand position, relate to American Sign Language alphabet, pull words from Merriam Webster's Dictionary API and provide functionality to sign out words and return them as audio and text (can scale back to still provide viable product within the limited timeframe).

Home Page: https://drewdevero.github.io/Tri-Sense/

HTML 1.74% CSS 2.95% JavaScript 32.40% Java 5.15% Jupyter Notebook 57.76%

tri-sense's Introduction

Tri-Sense

Tri-Sense Site Screenshot 1st level project(MVP): A dictionary look up app with returning word, pronunciation(written form), part of speech, and vocal file giving auditory example of pronunciation. Accessibility level considered adequate as determined by Axe Accessibility.
2nd level project(stretch goals) - Detect hand position and show constant stream live through webcam incorporating another technology, MediaPipe hand tracking, properly through react. Make that functionality capable of being activated or deactivated via client. Show mapping of fingers so user can easily relate hand position to sign language alphabet and numbers 0 - 9.
3rd level project(serious stretch goals)- relate live to American Sign Language alphabet, pull words from Merriam-Webster's Dictionary API and provide functionality to sign our words returned as audio and text (can scale back to written input only to still provide viable 1st level product(MVP) within the limited timeframe). Meant as the start of a much larger project; meant for submissions to initiatives like the Google Developers Solution Challenge; initiatives that foster the growth of constructive projects with positive impacts.

Database Implementation (Integrated with Heroku Deployment)

  • Create a database using PostgreSQL
  • Create model with table for users using Java
  • Create model with table for words using Java
  • Create model with table for ASL letters A-Z and number 0-9 using Java and consuming the HandSpeak API
  • Generate CRUD capability using React and Java
  • Stretch Goal Implement user accessable word bank that updates based on database information

User-Story

Used primarily by:
Client who is hearing impaired; I want to type out words and relate them to signed out words; convert words to sounds and written form so that I can more efficiently communicate with all other people in a multitude of situations.

End user goal:
Dictionary App for word lookup and audio returned as written and vocalized characters/words; App for tracking hand motion by webcam.

End business goal:
Camera access. Merriam-Webster's Dictionary API.

Acceptance criteria:
Refer to American Sign Language Alphabet. Refer to Merriam-Webster's API. Return written characters. Return vocal versions of characters. Enable webcam hand tracking through react app. Accessibility level considered adequate as determined by Axe Accessibility.

Tri-Sense Mockup

*Level One project (MVP) and Level Two project (strech goal) achieved. All acceptance criteria achieved.

Sources: MediaPipe documentation and package: https://google.github.io/mediapipe/getting_started/javascript.html Finger Pose documentation and package: https://openbase.com/js/fingerpose/documentation General connecting webcam to browser capability: https://www.kirupa.com/html5/accessing_your_webcam_in_html5.htm General create files via javascript: https://code-boxx.com/create-save-files-javascript/ LabelBox for image annotation to use in training and testing cases: https://app.labelbox.com/ Tensorflow 2 Object Detection API tutorial: https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/index.html Tensorflow object detection overview: https://github.com/nicknochnack/TFODCourse

tri-sense's People

Contributors

drewdevero avatar

Stargazers

David Kirby avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.