rom1504 / image_embeddings Goto Github PK

View Code? Open in Web Editor NEW

143.0 3.0 32.0 16.55 MB

Using efficientnet to provide embeddings for retrieval

Home Page: https://rom1504.github.io/image_embeddings/

License: MIT License

Jupyter Notebook 97.77% Makefile 0.14% Python 2.09%

embeddings computer-vision representation-learning efficientnet retrieval recommendation-system

image_embeddings's Issues

Video datasets

Hello,

I would like to get the embeddings for each frame of a video dataset.
I'm using CV2 to get the frames, so I end up with frames of (720, 1280, 3). Being the first two dimensions the resolution and 3 the color channels.

How can I get the embedding of each frame?
I don't see a function to load new datasets in the .ipynb from scratch.

Thanks

Invalid argument: input depth must be evenly divisible by filter depth: 1 vs 3?

What This Error?

OP_REQUIRES failed at conv_ops_fused_impl.h:716 : Invalid argument: input depth must be evenly divisible by filter depth: 1 vs 3

module 'image_embeddings' has no attribute 'inference'

How to solve this problem

Regards
Shafi

In some folders I am getting this error OverflowError: cannot convert float infinity to integer

Hi @rom1504

First of all thanks for the great work that you've done, I am facing this error is some of the folders that I am getting running similarity search in. Please help

 File "/home/cctvserver/Documents/Work/SimilaritySearch/Embeddings/image_embeddings-master/image_embeddings/inference/inference.py", line 92, in tfrecords_to_write_embeddings
    embeddings = images_to_embeddings(model, shard, batch_size)
  File "/home/cctvserver/Documents/Work/SimilaritySearch/Embeddings/image_embeddings-master/image_embeddings/inference/inference.py", line 119, in images_to_embeddings
    return model.predict(dataset.batch(batch_size).map(lambda image_raw, image_name: image_raw), verbose=1)
  File "/home/cctvserver/miniconda3/envs/tracker/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 130, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/home/cctvserver/miniconda3/envs/tracker/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1613, in predict
    callbacks.on_predict_end()
  File "/home/cctvserver/miniconda3/envs/tracker/lib/python3.8/site-packages/tensorflow/python/keras/callbacks.py", line 582, in on_predict_end
    callback.on_predict_end(logs)
  File "/home/cctvserver/miniconda3/envs/tracker/lib/python3.8/site-packages/tensorflow/python/keras/callbacks.py", line 979, in on_predict_end
    self._finalize_progbar(logs)
  File "/home/cctvserver/miniconda3/envs/tracker/lib/python3.8/site-packages/tensorflow/python/keras/callbacks.py", line 1026, in _finalize_progbar
    self.progbar.update(self.seen, list(logs.items()), finalize=True)
  File "/home/cctvserver/miniconda3/envs/tracker/lib/python3.8/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 581, in update
    numdigits = int(np.log10(self.target)) + 1
OverflowError: cannot convert float infinity to integer

Make it possible to configure how many examples are used in downloader

try to do inference in ui with tensorflow.js and allow the user to provide their own image

fix fashion mnist and mnist

InvalidArgumentError: input depth must be evenly divisible by filter depth: 1 vs 3

just a color issue

batch mode and multithread

Hello,

Thx for the package ?
How to setup batch mode and multithread easily ?
(inference on CPU).

Think one should be able load some small fine tuner head for quick fine tuning
and save the model back.

Publish ui package and make it possible to provide embeddings and images

Could be a good follow up to the CLI

Indexing only bright pixels in an image?

Hi Romain

Do you have a solution for this? facebookresearch/faiss#1497

simple ui for knn

will go to github pages

Provide notebook doing inference before knn for "your own image" effect

python downloader

is that possible / easy ?
doing a js downloader is very simple

provide end to end command with CLI

Maybe requiring only a root dir to save things

image extension .jpeg hardcoded

You have hardcoded extension .jpeg which makes it harder to try other images

<ipython-input-42-a444c2d0177a> in <module>
      1 p=1
      2 print(id_to_name[p])
----> 3 image_embeddings.knn.display_picture(path_images, id_to_name[p])
      4 results = image_embeddings.knn.search(index, id_to_name, embeddings[p])
      5 image_embeddings.knn.display_results(path_images, results)

~/miniconda3/envs/tracker/lib/python3.8/site-packages/image_embeddings/knn/knn.py in display_picture(image_path, image_name)
     54 
     55 def display_picture(image_path, image_name):
---> 56     display(Image(filename=f"{image_path}/{image_name}.jpeg"))
     57 
     58

How to generate embeddings of an input image ?

i want to know how to deal with a new input image , how we make embeddings and how to compare with our saved embeddings ?

put notebook on colab for easy demo

inference

should come with :

decent setup.py + requirements.txt setup
a notebook for usage
some instructions in the readme
how to make tfrecords for it (read/write) : include mapping (id vs string)

Flexibility in choosing models for image embeddings

Hi,
In the API: image_embeddings.inference.run_inference(tfrecords_folder, output_folder, batch_size=1000), can we have a model parameter such that we can use the same API to get embeddings from any model we choose.
The default model can be EfficientNetB0 and we can modify it to compare between different versions.
(Label: Enhancement)

Add option in CLI to provide image to search

instead of only random search

image embeddings blog post

simple knn example

ValueError: Expect x to be a non-empty array or dataset.

I am trying to create an embedding for some google images I downloaded. This is my structure:

When I execute this

image_embeddings.inference.write_tfrecord(image_folder="tmp/test_images",
                                          output_folder="tmp/test_tensors",
                                          num_shards=10)

image_embeddings.inference.run_inference(tfrecords_folder="tmp/test_tensors",
                                         output_folder="tmp/test_output",
                                         batch_size=1000)

[id_to_name2, name_to_id2, embeddings2] = image_embeddings.knn.read_embeddings("tmp/test_output")
index2 = image_embeddings.knn.build_index(embeddings2)

I get

ValueError: Expect x to be a non-empty array or dataset.

Althought it fails, files are generated:

But if I try to search with that embedding in another index of images that I have,

results = image_embeddings.knn.search(another_index, id_to_name2, embeddings2[0], k=1)
results = [i for i in results if i[1]!=id_to_name2[p]]
image_embeddings.knn.display_results(JPEG_FOLDER, results)

I get:

KeyError: 36

I tried different numbers of shards and different numbers of batches. None of them work, what could the reason be?

Full traces:

ValueError                                Traceback (most recent call last)
<ipython-input-40-405d145f78be> in <module>()
     15 image_embeddings.inference.run_inference(tfrecords_folder="tmp/test_tensors",
     16                                          output_folder="tmp/test_output",
---> 17                                          batch_size=1000)
     18 
     19 [id_to_name2, name_to_id2, embeddings2] = image_embeddings.knn.read_embeddings("tmp/test_output")

3 frames
/usr/local/lib/python3.7/dist-packages/image_embeddings/inference/inference.py in run_inference(tfrecords_folder, output_folder, batch_size)
    154     Path(output_folder).mkdir(parents=True, exist_ok=True)
    155     model = EfficientNetB0(weights="imagenet", include_top=False, pooling="avg")
--> 156     tfrecords_to_write_embeddings(tfrecords_folder, output_folder, model, batch_size)

/usr/local/lib/python3.7/dist-packages/image_embeddings/inference/inference.py in tfrecords_to_write_embeddings(tfrecords_folder, output_folder, model, batch_size)
     90     for shard_id, tfrecord in enumerate(tfrecords):
     91         shard = read_tfrecord(tfrecord)
---> 92         embeddings = images_to_embeddings(model, shard, batch_size)
     93         print("")
     94         print("Shard " + str(shard_id) + " done after " + str(int(time.time() - start)) + "s")

/usr/local/lib/python3.7/dist-packages/image_embeddings/inference/inference.py in images_to_embeddings(model, dataset, batch_size)
    117 
    118 def images_to_embeddings(model, dataset, batch_size):
--> 119     return model.predict(dataset.batch(batch_size).map(lambda image_raw, image_name: image_raw), verbose=1)
    120 
    121 

/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py in predict(self, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing)
   1740             callbacks.on_predict_batch_end(end_step, {'outputs': batch_outputs})
   1741       if batch_outputs is None:
-> 1742         raise ValueError('Expect x to be a non-empty array or dataset.')
   1743       callbacks.on_predict_end()
   1744     all_outputs = nest.map_structure_up_to(batch_outputs, concat, outputs)

ValueError: Expect x to be a non-empty array or dataset.

KeyError                                  Traceback (most recent call last)
<ipython-input-58-ad901d8c8c41> in <module>()
----> 1 results = image_embeddings.knn.search(index, id_to_name2, embeddings2[0], k=1)
      2 results = [i for i in results if i[1]!=id_to_name2[p]]
      3 image_embeddings.knn.display_results(JPEG_FOLDER, results)

1 frames
/usr/local/lib/python3.7/dist-packages/image_embeddings/knn/knn.py in search(index, id_to_name, emb, k)
     50 def search(index, id_to_name, emb, k=5):
     51     D, I = index.search(np.expand_dims(emb, 0), k)  # actual search
---> 52     return list(zip(D[0], [id_to_name[x] for x in I[0]]))
     53 
     54 

/usr/local/lib/python3.7/dist-packages/image_embeddings/knn/knn.py in <listcomp>(.0)
     50 def search(index, id_to_name, emb, k=5):
     51     D, I = index.search(np.expand_dims(emb, 0), k)  # actual search
---> 52     return list(zip(D[0], [id_to_name[x] for x in I[0]]))
     53 
     54 

KeyError: 36

rom1504 / image_embeddings Goto Github PK

image_embeddings's Issues

Recommend Projects

Recommend Topics

Recommend Org