We introduce a large image dataset HaGRID (HAnd Gesture Recognition Image Dataset) for hand gesture recognition (HGR) systems. You can use it for image classification or image detection tasks. Proposed dataset allows to build HGR systems, which can be used in video conferencing services (Zoom, Skype, Discord, Jazz etc.), home automation systems, the automotive sector, etc.
HaGRID size is 716GB and dataset contains 552,992 FullHD (1920 ร 1080) RGB images divided into 18 classes of gestures. Also, some images have no_gesture
class if there is a second free hand in the frame. This extra class contains 123,589 samples. The data were split into training 92%, and testing 8% sets by subject user_id
, with 509,323 images for train and 43,669 images for test.
The dataset contains 34,730 unique persons and at least this number of unique scenes. The subjects are people from 18 to 65 years old. The dataset was collected mainly indoors with considerable variation in lighting, including artificial and natural light. Besides, the dataset includes images taken in extreme conditions such as facing and backing to a window. Also, the subjects had to show gestures at a distance of 0.5 to 4 meters from the camera.
Example of sample and its annotation:
For more information see our arxiv paper HaGRID - HAnd Gesture Recognition Image Dataset.
Clone and install required python packages:
git clone https://github.com/hukenovs/hagrid.git
# or mirror link:
cd hagrid
# Create virtual env by conda or venv
conda create -n gestures python=3.9 -y
conda activate gestures
# Install requirements
pip install -r requirements.txt
docker build -t gestures .
docker run -it -d -v $PWD:/gesture-classifier gestures
We split the train dataset into 18 archives by gestures because of the large size of data. Download and unzip them from the following links:
Gesture | Size | Gesture | Size |
---|---|---|---|
call |
39.1 GB | peace |
38.6 GB |
dislike |
38.7 GB | peace_inverted |
38.6 GB |
fist |
38.0 GB | rock |
38.9 GB |
four |
40.5 GB | stop |
38.3 GB |
like |
38.3 GB | stop_inverted |
40.2 GB |
mute |
39.5 GB | three |
39.4 GB |
ok |
39.0 GB | three2 |
38.5 GB |
one |
39.9 GB | two_up |
41.2 GB |
palm |
39.3 GB | two_up_inverted |
39.2 GB |
train_val
annotations: ann_train_val
Test | Archives | Size |
---|---|---|
images | test |
60.4 GB |
annotations | ann_test |
3.4 MB |
Subsample has 100 items per gesture.
Subsample | Archives | Size |
---|---|---|
images | subsample |
2.5 GB |
annotations | ann_subsample |
153.8 KB |
or by using python script
python download.py --save_path <PATH_TO_SAVE> \
--train \
--test \
--subset \
--annotations \
--dataset
Run the following command with key --subset
to download the small subset (100 images per class). You can download the
train subset with --trainval
or test subset with --test
. Download annotations for selected stage by --annotations
key. Download dataset with images by --dataset
.
usage: download.py [-h] [--train] [--test] [--subset] [-a] [-d] [-t TARGETS [TARGETS ...]] [-p SAVE_PATH]
Download dataset...
optional arguments:
-h, --help show this help message and exit
--train Download trainval set
--test Download test set
--subset Download subset with 100 items of each gesture
-a, --annotations Download annotations
-d, --dataset Download dataset
-t TARGETS [TARGETS ...], --targets TARGETS [TARGETS ...]
Target(s) for downloading train set
-p SAVE_PATH, --save_path SAVE_PATH
Save path
We provide some pre-trained models as the baseline with the classic backbone architectures and two output heads - for gesture classification and leading hand classification.
Classifiers | F1 Gestures | F1 Leading hand |
---|---|---|
ResNet18 | 98.72 | 99.27 |
ResNet152 | 99.11 | 99.45 |
ResNeXt50 | 98.99 | 99.39 |
ResNeXt101 | 99.28 | 99.28 |
MobileNetV3_small | 96.78 | 98.28 |
MobileNetV3_large | 97.88 | 98.58 |
Vitb32 | 98.49 | 99.13 |
Also we provide SSDLite model with MobileNetV3 large backbone to solve hand detection problem.
Detector | mAP |
---|---|
SSDLite | 71.49 |
You can use downloaded trained models, otherwise select a classifier and parameters for training in default.yaml
.
To train the model, execute the following command:
python -m classifier.run --command 'train' --path_to_config <PATH>
Every step, the current loss, learning rate and others values get logged to Tensorboard.
See all saved metrics and parameters by opening a command line (this will open a webpage at localhost:6006
):
tensorboard --logdir=experiments
Test your model by running the following command:
python -m classifier.run --command 'test' --path_to_config <PATH>
python demo.py -p <PATH_TO_DETECTOR>
The annotations consist of bounding boxes of hands in COCO format [top left X position, top left Y position, width, height]
with gesture labels. Also annotations have markups of leading hands
(left
or right
for gesture hand) and leading_conf
as confidence for leading_hand
annotation. We provide user_id
field that will allow you to split the train / val dataset yourself.
"03487280-224f-490d-8e36-6c5f48e3d7a0": {
"bboxes": [
[0.0283366, 0.8686061, 0.0757000, 0.1149820],
[0.6824319, 0.2661254, 0.1086447, 0.1481245]
],
"labels": [
"no_gesture",
"one"
],
"leading_hand": "left",
"leading_conf": 1.0,
"user_id": "bb138d5db200f29385f..."
}
- Key - image name without extension
- Bboxes - list of normalized bboxes
[top left X pos, top left Y pos, width, height]
- Labels - list of class labels e.g.
like
,stop
,no_gesture
- Leading hand -
right
orleft
for hand which showing gesture - Leading conf - leading confidence for
leading_hand
- User ID - subject id (useful for split data to train / val subsets).
Object | Train + Val | Test | Total |
---|---|---|---|
gesture | ~ 28 300 | ~ 2 400 | 30 629 |
no gesture | 112 740 | 10 849 | 123 589 |
total boxes | 622 063 | 54 518 | 676 581 |
This work is licensed under a variant of Creative Commons Attribution-ShareAlike 4.0 International License.
Please see the specific license.
You can cite the paper using the following BibTeX entry:
@article{hagrid,
title={HaGRID - HAnd Gesture Recognition Image Dataset},
author={Kapitanov, Alexander and Makhlyarchuk, Andrey and Kvanchiani, Karina},
journal={arXiv preprint arXiv:2206.08219},
year={2022}
}