- This project demonstrates a real-time hand gesture recognition system using Python, OpenCV, and Gemini AI by Google.
- The system allows users to solve math problems by drawing them in the air.
- The repository contains all the code and resources needed to implement this project.
- Real-time hand detection and tracking using OpenCV and cvzone.
- Recognition of specific hand gestures to perform actions like drawing, clearing canvas, and requesting math problem solutions.
- Integration with Gemini AI for solving math problems based on gestures.
- Python 3.10.12
- cvzone
pip install cvzone
- OpenCV
pip install opencv-python
- Gemini AI
pip install google-generativeai
- Python packages
- numpy
pip install numpy
- pillow
pip install pillow
- numpy
- C++ compiler - install Visual Studio
-
Clone the repository:
git clone https://github.com/LasithaAmarasinghe/Hand-Gesture-Math-Solver.git cd Hand-Gesture-Math-Solver
-
Create and activate a virtual environment (optional but recommended):
python -m venv venv source venv/bin/activate # On Windows use `venv\Scripts\activate`
-
Install the required packages:
pip install -r requirements.txt
-
Get a Gemini API key
You can get an API key from here
-
Replace API key in the code with the obtained API key
-
Execute the main script
python math_solver.py
-
Interact with the Application
- Live Feed
- The application will display the live feed from your webcam.
- Perform Hand Gestures
- Draw: Use one finger to draw on the canvas.
- Clear: Use one finger to clear the canvas.
- Solve Math Problem: Use all fingers except the thumb to submit a math problem for solving.
- Live Feed
-
getHandInfo(img)
- Purpose: Detects and tracks the user's hand in each frame of the webcam feed.
- Implementation: Uses the HandTrackingModule to identify and monitor hand movements.
-
draw(info, prev_pos, canvas)
- Purpose: Draws on the canvas based on the detected hand gestures.
- Parameters:
info
: Information about the detected hand gestures.prev_pos
: Previous position of the drawing.canvas
: Canvas to draw on.
-
sendToAI(model, canvas, fingers)
- Purpose: Sends the drawn content to Google's Generative AI for math problem solving based on recognized gestures.
- Parameters:
model
: Instance of the Generative AI model.canvas
: Canvas containing the drawn content.fingers
: Array indicating the state of fingers (e.g., for gesture recognition).
![](https://private-user-images.githubusercontent.com/106037441/340177476-196a14ec-6067-494c-8e06-69873fa418f3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MjE1OTIsIm5iZiI6MTcyNDUyMTI5MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTc3NDc2LTE5NmExNGVjLTYwNjctNDk0Yy04ZTA2LTY5ODczZmE0MThmMy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQxNzQxMzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02MjBlMjM5NjI5ZjkwMjIzOTBjY2M1NmUzZWVhMzVhNWEyZDc4ZmZkZmFhYzc2MzJkMWZkYjgzOTJmMDFkOGQ1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.qrRPOJ_FzAGl5K6vkyjTjZxNOtIBdRpLNdzX1BveIfI)
Left-Right Hand Detection
![](https://private-user-images.githubusercontent.com/106037441/340176842-c0eaf562-bb89-4b47-b9f8-4bb96ec3e442.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MjE1OTIsIm5iZiI6MTcyNDUyMTI5MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTc2ODQyLWMwZWFmNTYyLWJiODktNGI0Ny1iOWY4LTRiYjk2ZWMzZTQ0Mi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQxNzQxMzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kY2YwMGMxMzUzYjcxZWVjODdkZjQ0MWNlOWYwMzBjYWZkZjU3ODUxY2FjZGQ1N2I5NjkwMjM0ZGI2ZTQxY2RiJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.tajdcPn0z9hnSedaJU2aZDDzYVIEnN9Jwpmkv0UfzgE)
![](https://private-user-images.githubusercontent.com/106037441/340176849-a7c1a542-8b85-42a2-87fa-bb20ff280a1e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MjE1OTIsIm5iZiI6MTcyNDUyMTI5MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTc2ODQ5LWE3YzFhNTQyLThiODUtNDJhMi04N2ZhLWJiMjBmZjI4MGExZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQxNzQxMzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lZDJhZDAxY2QwMDIzZGFiOTk4NjlhNzUwZjdmMWJlOGY3NDA3OGYwYzBlOWUxYzY1ZTc1Mjg2YjkwMDJmNjAzJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.G5jodpTrMDmuGc6sn0CnawoTTss5r9tcf9cIAtrhoz8)
Hand Gesture Detection
![](https://private-user-images.githubusercontent.com/106037441/340180522-56434be0-8338-41e2-b060-5b14e69f6325.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MjE1OTIsIm5iZiI6MTcyNDUyMTI5MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTgwNTIyLTU2NDM0YmUwLTgzMzgtNDFlMi1iMDYwLTViMTRlNjlmNjMyNS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQxNzQxMzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kZDkzNTVjOGYwNjc3Yjk1M2YyMjViY2Y3YmRjOWFmMTk5MjgzOWMxZWI2YTgxYjRmZWZlNGJkZjBlMmEyMjgzJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.3MOLzup1pWefhDUgW-eohXeyCfhPEwG1yZbz8Aecf7E)
![](https://private-user-images.githubusercontent.com/106037441/340181207-b66a3d35-f7f7-4b75-b7db-c93ee40d6841.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MjE1OTIsIm5iZiI6MTcyNDUyMTI5MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTgxMjA3LWI2NmEzZDM1LWY3ZjctNGI3NS1iN2RiLWM5M2VlNDBkNjg0MS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQxNzQxMzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iNTRmODNlYTU1MjBkMTlhMDhiMTg3Y2M3ZGFiMTA3Njg2OTk4OTU0MWUyNzE0YzA5MWY1MTViMTExM2VjMmU0JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.4cffd63GmmBoUJNkwBJ_viv84SHTwUtugb70ptJmWkA)
Math Expression Detection & Canvas
![](https://private-user-images.githubusercontent.com/106037441/340166001-e9558800-a17f-432d-a4f5-d00d0a352cfb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MjE1OTIsIm5iZiI6MTcyNDUyMTI5MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTY2MDAxLWU5NTU4ODAwLWExN2YtNDMyZC1hNGY1LWQwMGQwYTM1MmNmYi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQxNzQxMzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kODZjMDU2ODI2NjllM2M3NDQ2MGJmYzQ3MmZhMjViYzM2M2Y5MDQ5NDQyMjIyYmI1MDU3YTEwYWYxYjE5ZTYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.BCXz_UzGl4-49Qppedegfz5nk3Z-UnIA3SVG5xWR4KI)
![](https://private-user-images.githubusercontent.com/106037441/340166035-64664179-a2cc-43a2-af2b-409a6beb4325.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MjE1OTIsIm5iZiI6MTcyNDUyMTI5MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTY2MDM1LTY0NjY0MTc5LWEyY2MtNDNhMi1hZjJiLTQwOWE2YmViNDMyNS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQxNzQxMzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zMGU5YWViNjVkNmI4OWQ0NTBhZTc5MDFlYzc5ZTJiYmNhYjI2NDZiMTMzNTJhOTcxZDMyMWI4Nzc2OGIyNWYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.d3gy3ykRYwGWYG9VF1uzFNA3e8GgopkQBqyukYl07m0)
- This project is licensed under the MIT License. See the LICENSE file for details.