TinyTracker: Ultra-Fast and Ultra-Low-Power Edge Vision In-Sensor for Gaze Estimation
🧑🏻🚀 Pietro Bonazzi *1,
🧑🏼🚀 Thomas Rüegg*1,
Sizhen Bian 1,
Yawei Li1,
Michele Magno1
*denotes equal contribution
ETH Zurich, Switzerland
in IEEE Sensor 2023 (Lecture Presentation)
- Install requirements
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
- Install tensorflow
- get haarcascade detector for the face and store it in a new directory called "haarcascade"
mkdir haarcascade
cd haarcascade
wget https://github.com/opencv/opencv/blob/master/data/haarcascades/haarcascade_frontalface_default.xml
- run WebcamDemoTflite.py
python WebcamDemoTflite.py
If everything works without errors, you can now track your gaze with your webcam and compare TinyTracker to TinyTrackerS. Red point: TinyTracker Pink point: TinyTrackerS
TinyTracker and TinyTrackerS are highly efficient, fully quantized models for 2D gaze estimation designed to maximize the performance of the edge vision systems considered in the study (i.e Sony IMX500, Sony Spresense and Coral Dev Micro).
| Name | Input res. | Params | MAC | Size [Kb] |
|---|---|---|---|---|
| TinyTracker | 112x112 | 455k | 11.8M | 608 |
| TinyTrackerS | 96x96 | 85.8k | 5.3M | 169 |
Here we show how to train our code on the GazeCapture Dataset. You can download the data used in the paper here.
Unzip dataset by:
mkdir GazeCapture
tar -xf GazeCapture.tar -C ./GazeCapture
cd GazeCapture
ls | grep ".tar.gz$" | xargs -I {} tar -xf {}
ls | grep ".tar.gz$" | xargs -I {} rm {}
Process dataset by running:
python prepareDataset.py --dataset_path /path/to/Dataset --output_path /path/to/Processed/Dataset
To train TinyTracker:
python train_TinyTracker.py --data-path /path/to/Processed/Dataset
To train TinyTrackerS:
python train_TinyTrackerS.py --data-path /path/to/Processed/Dataset
You can control other arguments with these commands:
--data-path # Path to the dataset
--num-epochs # Number of epochs to train
--batch-size # Batch size for training
--image-size # Image input size
--learning-rate # Learning rate for training
--pretrained # Path to pretrained model
--backbone # Choosing which backbone to use
--grid-emb # If grid embedding should be used
--grey # If input images are greyscale
python convert_tlfite.py --path path/to/model.htf5
To convert tflite to .cc you can use this command
xxd -i path/to/model.tflite > path/to/model.cc
The project structured as following :
.
├── demo # Demo Script folder
│ └── SpresenseDemo.py # Spresense Demo Script
├── models # Model Checkpoints
│ ├── TinyTracker.tflite # TinyTracker model checkpoint
│ └── TinyTrackerS.tflite # TinyTrackerS model checkpoint
├── src # assets, instructions, requirements file for virtual environments
│ ├── evaluate_tflite.py # Evaluation script for tflite models
│ ├── ITrackerDataTF.py # Dataloader for GazeCapture dataset
│ ├── json_utils.py # Helper functions for json files
│ ├── TinyTrackerS.py # TinyTrackerS model class
│ ├── TinyTrackerTF.py # TinyTracker model classes
├── convert_tflite.py # Tflite conversion Script
├── prepareDataset.py # Dataset processing script
├── train_TinyTracker.py # Training script for TinyTracker
├── train_TinyTrackerS.py # Training script for TinyTrackerS
├── WebcamDemoTflite.py # Webcam Demo for both TinyTracker and TinyTrackerS
.
@INPROCEEDINGS{bonazzi2023tinytracker,
author={Bonazzi, Pietro and Rüegg, Thomas and Bian, Sizhen and Li, Yawei and Magno, Michele},
booktitle={2023 IEEE SENSORS},
title={TinyTracker: Ultra-Fast and Ultra-Low-Power Edge Vision In-Sensor for Gaze Estimation},
year={2023},
pages={1-4},
keywords={Energy consumption;Computer vision;Machine vision;Estimation;AI accelerators;Vision sensors;Cameras},
doi={10.1109/SENSORS56945.2023.10325167}}
@inproceedings{ruegg2023tinytrackers,
title={Gaze Estimation on Spresense},
author={Thomas Rüegg, Pietro Bonazzi, Andrea Ronco},
year={2023},
booktitle={Sony Challenge},
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent