Classification of Cooking Ingredients Using Deep Learning
This directory contains code for training and evaluating of Inception v3 model on cooking ingredients. It contains scripts that will allow you to finetune the model trained on the 1000 classes of the ImageNet to 178 classes containing fruits, vegetables and other cooking ingredients. It also contains code for converting the images to TensorFlow's native TFRecord format.
Project Organization
Preparing the Datasets
Fine-tuning of Inception V3
Evaluating performance
Run Demo Server
├── LICENSE
├── Makefile <- Makefile with commands like `make data` or `make train` (not used yet)
├── README.md <- The top-level README for developers using this project.
├── data
│ ├── external <- Data from third party sources. (not used yet)
│ ├── interim <- Intermediate data that has been transformed. (not used yet)
│ ├── processed <- The final, canonical data sets for modeling which can be ingested by tensorflow.
│ └── raw <- The original, immutable data dump (raw images).
│
├── docs <- A default Sphinx project; see sphinx-doc.org for details (not used yet)
│
├── models <- Downloaded pretrained tensorflow models and models trained on our dataset
│ ├─ downloaded_tf_models
│ └── inception_v3
│
├── notebooks <- Jupyter notebooks. Naming convention is a number (for ordering),
│ the creator's initials, and a short `-` delimited description, e.g.
│ `1.0-jqp-initial-data-exploration`.
│
├── references <- Data dictionaries, manuals, and all other explanatory materials. (not used yet)
│
├── reports <- Generated analysis as HTML, PDF, LaTeX, etc. (not used yet)
│ └── figures <- Generated graphics and figures to be used in reporting (not used yet)
│
├── requirements.txt <- The requirements file for reproducing the analysis environment, e.g.
│ generated with `pip freeze > requirements.txt`
│
├── src <- Source code for use in this project.
│ ├── __init__.py <- Makes src a Python module
│ │
│ ├── data <- Scripts to download or generate data
│ │
│ ├── features <- Scripts to turn raw data into features for modeling (not used yet)
│ │
│ ├── models <- Scripts to train models and then use trained models to make (not used yet)
│ │ predictions
│ │
│ ├── visualization <- Scripts to create exploratory and results oriented visualizations (not used yet)
│ │
│ │
│ └── slim <- Clone of https://github.com/tensorflow/models/tree/master/slim
│ │ Slim is a high level API to Tensorflow which make the training more convenient
│ │
│ ├── eval_image_classifier.py <- Evaluate a model on a dataset
│ ├── train_image_classifier.py <- Train a model on a dataset
│ └── datasets <- Scrips to convert the raw data to dataset which can be used with slim
│ │
│ ├── convert_scoodit_178.py
│ ├── convert_scoodit_test_snaps.py
│ ├── dataset_factory.py
│ ├── scoodit_178.py
│ └── scoodit_178_test_snaps.py
└── tox.ini <- tox file with settings for running tox; see tox.testrun.org
Two datasets have been used in this project: scoodit_178
and scoodit_178_test_snaps
.
scoodit_178
contains ~180K images from ImageNet across 178 classes.
scoodit_178_test_snaps
contains ~4K images of fruits, vegetables and
other groceries acquired for the validation of the model.
The archives in each subdirectory must be unpacked. You can use src/notebooks/01_nsckir_temp_extract_archives.ipynb
to unpack all the archives at once. After extraction the data must have the following structure (NO BLANKS IN FOLDER NAMES):
${PROJECT_FOLDER}/data/raw/scoodit_178/acorn_squash_n07717410/image.jpeg
${PROJECT_FOLDER}/data/raw/scoodit_178/acorn_squash_n07717410/another_image.jpeg
...
${PROJECT_FOLDER}/data/raw/scoodit_178/almond_n07750586/image.jpeg
${PROJECT_FOLDER}/data/raw/scoodit_178/almond_n07750586/another_image.jpeg
...
${PROJECT_FOLDER}/data/raw/scoodit_178/apple_n07739125/image.jpeg
${PROJECT_FOLDER}/data/raw/scoodit_178/apple_n07739125/another_image.jpeg
...
Run src/data/create_scoodit178_train_test_split.sh
to put 95% of the
images in the directory for training data/raw/scoodit_178/train/
and the remaining 5% in the directory for validation data/raw/scoodit_178/test/
.
After that you should have the following data structure (NO BLANKS IN FOLDER NAMES):
${PROJECT_FOLDER}/data/raw/scoodit_178/train/acorn_squash_n07717410/143.jpeg
${PROJECT_FOLDER}/data/raw/scoodit_178/train/acorn_squash_n07717410/51234.jpeg
...
${PROJECT_FOLDER}/data/raw/scoodit_178/train/acorn_squash_n07717410/almond_n07750586/1467.jpeg
${PROJECT_FOLDER}/data/raw/scoodit_178/train/acorn_squash_n07717410/almond_n07750586/8765.jpeg
...
${PROJECT_FOLDER}/data/raw/scoodit_178/train/acorn_squash_n07717410/apple_n07739125/4356.jpeg
${PROJECT_FOLDER}/data/raw/scoodit_178/train/acorn_squash_n07717410/apple_n07739125/543.jpeg
...
${PROJECT_FOLDER}/data/raw/scoodit_178/test/acorn_squash_n07717410/134.jpeg
${PROJECT_FOLDER}/data/raw/scoodit_178/test/acorn_squash_n07717410/341.jpeg
...
${PROJECT_FOLDER}/data/raw/scoodit_178/test/acorn_squash_n07717410/almond_n07750586/43245.jpeg
${PROJECT_FOLDER}/data/raw/scoodit_178/test/acorn_squash_n07717410/almond_n07750586/3245.jpeg
...
${PROJECT_FOLDER}/data/raw/scoodit_178/test/acorn_squash_n07717410/apple_n07739125/1456.jpeg
${PROJECT_FOLDER}/data/raw/scoodit_178/test/acorn_squash_n07717410/apple_n07739125/654.jpeg
...
Go to src/slim/datasets
and run convert_scoodit_178.py
to convert the raw data to TFRecord format
(You have to check the folder paths in the scripts. Might be that there are some absolute paths you have to adjust).
When the script finishes you will find several TFRecord files created:
${PROJECT_FOLDER}/data/processed/scoodit_178/scoodit_178_train_00000-of-00172.tfrecord
...
${PROJECT_FOLDER}/data/processed/scoodit_178/scoodit_178_train_00171-of-00172.tfrecord
...
${PROJECT_FOLDER}/data/processed/scoodit_178/scoodit_178_validation_00000-of-00016.tfrecord
...
${PROJECT_FOLDER}/data/processed/scoodit_178/scoodit_178_validation_00015-of-00016.tfrecord
${PROJECT_FOLDER}/data/processed/scoodit_178/labels.txt
The folder structure must be EXACTLY the same as in
data/raw/scoodit_178/
.
Go to src/slim/data/
and run convert_scoodit_test_snaps.py
. Again, check if all tha paths
in the script are correct.
When the script finishes you will find several TFRecord files created:
${PROJECT_FOLDER}/data/processed/scoodit_178_test_snaps/scoodit_178_validation_00000-of-00004.tfrecord
...
${PROJECT_FOLDER}/data/processed/scoodit_178_test_snaps/scoodit_178_validation_00003-of-00004.tfrecord
${PROJECT_FOLDER}/data/processed/scoodit_178_test_snaps/labels.txt
Now you are ready to train and evaluate the model.
Download and extract the pretrained checkpoint of Inception V3 from this link
and save it to models/downloaded_tf_models/
Go to src/slim/
The script src/slim/scripts/finetune_inception_v3_on_scoodit_178_all_steps.sh
includes all steps and parameters
which retrain the inception_v3 model on the scoodit_178 data set. The final model will be saved
in model/inception_v3/scoodit_178/
. The total run time on AWS p2.16xlarge is about 10 hours
using the parameter in the script. There might be potential to reduce this time by changing
the learning rate and the number of steps. The final model achieves 92% top5 accuracy
on the validation set.
Go to src/slim/
and run:
python eval_image_classifier.py \
--checkpoint_path=${MODEL_DIR} \
--eval_dir=${MODEL_DIR}/test_snaps \
--dataset_name=scoodit_178_test_snaps \
--dataset_split_name=validation \
--dataset_dir=${PROJECT_FOLDER}/data/processed/scoodit_178_test_snaps \
--model_name=inception_v3
${MODEL_DIR}
is the folder where your fine-tuned model is saved
Go to src/slim/
and run:
./run_server.sh
The command starts a server "gunicorn" server where the trained model is served for inference. The server is accessible
in the browser under <aws_ec2_instance_ip_adress>:5001
. The API can also be queried directly e.g.
curl --request POST --data-binary "@sample.jpg" <aws_ec2_instance_ip_adress>:5001
A slightly faster API is available in the repo scoodit_server
curl --request POST --data-binary "@sample.jpg" <aws_ec2_instance_ip_adress>:5000
Caution: this is a demo server. There is no authentication implemented.
Cloned and adapted from https://github.com/hetaoaoao/tensorflow_web_deploy