TinyTracker: Ultra-Fast and Ultra-Low-Power Edge Vision In-Sensor for Gaze Estimation
π§π»βπ Pietro Bonazzi *1,
π§πΌβπ Thomas RΓΌegg*1,
Sizhen Bian 1,
Yawei Li1,
Michele Magno1
*denotes equal contribution
ETH Zurich, Switzerland
in IEEE Sensor 2023 (Lecture Presentation)
- Install requirements
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
- Install tensorflow
- get haarcascade detector for the face and store it in a new directory called "haarcascade"
mkdir haarcascade
cd haarcascade
wget https://github.com/opencv/opencv/blob/master/data/haarcascades/haarcascade_frontalface_default.xml
- run WebcamDemoTflite.py
python WebcamDemoTflite.py
If everything works without errors, you can now track your gaze with your webcam and compare TinyTracker to TinyTrackerS. Red point: TinyTracker Pink point: TinyTrackerS
TinyTracker and TinyTrackerS are highly efficient, fully quantized models for 2D gaze estimation designed to maximize the performance of the edge vision systems considered in the study (i.e Sony IMX500, Sony Spresense and Coral Dev Micro).
Name | Input res. | Params | MAC | Size [Kb] |
---|---|---|---|---|
TinyTracker | 112x112 | 455k | 11.8M | 608 |
TinyTrackerS | 96x96 | 85.8k | 5.3M | 169 |
Here we show how to train our code on the GazeCapture Dataset. You can download the data used in the paper here.
Unzip dataset by:
mkdir GazeCapture
tar -xf GazeCapture.tar -C ./GazeCapture
cd GazeCapture
ls | grep ".tar.gz$" | xargs -I {} tar -xf {}
ls | grep ".tar.gz$" | xargs -I {} rm {}
Process dataset by running:
python prepareDataset.py --dataset_path /path/to/Dataset --output_path /path/to/Processed/Dataset
To train TinyTracker:
python train_TinyTracker.py --data-path /path/to/Processed/Dataset
To train TinyTrackerS:
python train_TinyTrackerS.py --data-path /path/to/Processed/Dataset
You can control other arguments with these commands:
--data-path # Path to the dataset
--num-epochs # Number of epochs to train
--batch-size # Batch size for training
--image-size # Image input size
--learning-rate # Learning rate for training
--pretrained # Path to pretrained model
--backbone # Choosing which backbone to use
--grid-emb # If grid embedding should be used
--grey # If input images are greyscale
python convert_tlfite.py --path path/to/model.htf5
To convert tflite to .cc you can use this command
xxd -i path/to/model.tflite > path/to/model.cc
The project structured as following :
.
βββ demo # Demo Script folder
β βββ SpresenseDemo.py # Spresense Demo Script
βββ models # Model Checkpoints
β βββ TinyTracker.tflite # TinyTracker model checkpoint
β βββ TinyTrackerS.tflite # TinyTrackerS model checkpoint
βββ src # assets, instructions, requirements file for virtual environments
β βββ evaluate_tflite.py # Evaluation script for tflite models
β βββ ITrackerDataTF.py # Dataloader for GazeCapture dataset
β βββ json_utils.py # Helper functions for json files
β βββ TinyTrackerS.py # TinyTrackerS model class
β βββ TinyTrackerTF.py # TinyTracker model classes
βββ convert_tflite.py # Tflite conversion Script
βββ prepareDataset.py # Dataset processing script
βββ train_TinyTracker.py # Training script for TinyTracker
βββ train_TinyTrackerS.py # Training script for TinyTrackerS
βββ WebcamDemoTflite.py # Webcam Demo for both TinyTracker and TinyTrackerS
.
@INPROCEEDINGS{bonazzi2023tinytracker,
author={Bonazzi, Pietro and RΓΌegg, Thomas and Bian, Sizhen and Li, Yawei and Magno, Michele},
booktitle={2023 IEEE SENSORS},
title={TinyTracker: Ultra-Fast and Ultra-Low-Power Edge Vision In-Sensor for Gaze Estimation},
year={2023},
pages={1-4},
keywords={Energy consumption;Computer vision;Machine vision;Estimation;AI accelerators;Vision sensors;Cameras},
doi={10.1109/SENSORS56945.2023.10325167}}
@inproceedings{ruegg2023tinytrackers,
title={Gaze Estimation on Spresense},
author={Thomas RΓΌegg, Pietro Bonazzi, Andrea Ronco},
year={2023},
booktitle={Sony Challenge},
}