Using efficientnet to provide embeddings for retrieval. Read the blog post at https://medium.com/@rom1504/image-embeddings-ed1b194d113e
Why this repo ? Embeddings are a widely used technique that is well known in scientific circles. But it seems to be underused and not very well known for most engineers. I want to show how easy it is to represent things as embeddings, and how many application this unlocks. Checkout the demo first!
- download some pictures
- run inference on them to get embeddings
- simple knn example, to understand what's the point : click on some pictures and see KNN
Run pip install image_embeddings
- run
image_embeddings save_examples_to_folder --images_count=1000 --output_folder=tf_flower_images, this will retrieve 1000 image files from https://www.tensorflow.org/datasets/catalog/tf_flowers (but you can also pick any other dataset) - produce tf records with
image_embeddings write_tfrecord --image_folder=tf_flower_images --output_folder=tf_flower_tf_records --shards=10 - run the inference with
image_embeddings run_inference --tfrecords_folder=tf_flower_tf_records --output_folder=tf_flower_embeddings - run a random knn search on them
image_embeddings random_search --path=tf_flower_embeddings
Optionally if you want to use the embeddings in numpy (in other languages), run image_embeddings embeddings_to_numpy --input_path=tf_flower_embeddings --output_path=tf_flower_numpy. In particular this can be used in the web demo
$ image_embeddings random_search --path=tf_flower_embeddings
image_roses_261
160.83 image_roses_261
114.36 image_roses_118
102.77 image_roses_537
92.95 image_roses_659
88.49 image_roses_197
Explore the Simple notebook for more details.
You can try it locally or try it in colab
The From scratch notebook provides an explanation on how to build this from scratch.
Downloader from tensorflow datasets. Any other set of images could be used instead
image_embeddings.downloader.save_examples_to_folder(output_folder, images_count=1000, dataset="tf_flowers")
Save https://www.tensorflow.org/datasets/catalog/tf_flowers to folder Also works with other tf datasets
Create tf recors from images files, and apply inference with an efficientnet model. Other models could be used.
Write tf records from an image folders
Run inference on provided tf records and save to folder the embeddings
Convenience methods to read, build indices and apply search on them. These methods are provided as example. Use faiss directly for bigger datasets.
Run embeddings from path and return a tuple with
- embeddings as a numpy matrix
- an id to name dictionary
- a name to id dictionary
Build a simple faiss inner product index using the provided matrix of embeddings
Search the query embeddings and return an array of (distance, name) images
Display one picture from the given path and image name in jupyter
Display the results from search method
Load the embeddings, apply a random search on them and display the result
Load the embeddings from the input folder as parquet and save them as
- json for the id -> name mapping
- numpy for the embeddings
Particularly useful to read the embeddings from other languages
Make sure you use python>=3.6 and an up-to-date version of pip and
setuptools
python --version
pip install -U pip setuptools
It is recommended to install image_embeddings in a new virtual environment. For
example
python3 -m venv image_embeddings_env
source image_embeddings_env/bin/activate
pip install -U pip setuptools
pip install image_embeddings
pip install image_embeddings
First, clone the image_embeddings repo on your local machine with
git clone https://github.com/rom1504/image_embeddings.git
cd image_embeddings
make install
To install development tools and test requirements, run
make install-dev
To run unit tests in your current environment, run
make test
To run lint + unit tests in a fresh virtual environment, run
make venv-lint-test
To run black --check:
make lint
To auto-format the code using black
make black
- simple downloader in python
- simple inference in python using https://github.com/qubvel/efficientnet
- build python basic knn example using https://github.com/facebookresearch/faiss
- build basic ui using lit element and some brute force knn to show what it does, put in github pages
- use to illustrate embeddings blogpost
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent