nicolafan / image-captioning-cnn-rnn Goto Github PK
View Code? Open in Web Editor NEWTensorflow/Keras implementation of an image captioning neural network, using CNN and RNN
License: MIT License
Tensorflow/Keras implementation of an image captioning neural network, using CNN and RNN
License: MIT License
The BLEU metric is a numerical metric used in image captioning.
They need to be implemented inside the src/models/metrics.py
file (not sure if it is the correct place by the way).
I think this should not be implemented as a tensor metric that can be used by Tensorflow, but as a metric that has to be applied directly to strings.
Basically, we will provide the ground truth caption string and a string predicted by the model. How the prediction string is produced depends on the implementation to try: sampling, beam search, or max likelihood, but is not of interest for the BLEU Implementation.
I create this issue just to remember to update it when a clear pipeline, with CLI arguments etc., is defined.
Add a CLI option to model training that provides the possibility to continue training an existing model.
If the option is specified, with the model filename, we should ignore all the parameters that refer to the model architecture or raise an error if they are specified.
The prediction plot shows the cartesian axes over all the images (except for the last one). Remove the axes (file is src/models/predict_model.py
).
Use the Subclassing API to divide the ShowAndTell
model into two sub-models: the encoder and the decoder. This is required since at prediction time the model is applied at each step of beam search on the same image over and over again. The image encoding is unique and so it can be calculated just once, at the beginning of the process.
The file is /src/models/predict_model.py
.
Implement the content of the predict_model.py
file.
In the file a stateful model should be loaded to use it for prediction.
The current implementation requires that an image has to be given as an input to the model together with a sequence of length MAX_CAPTION_LENGTH
, where the first element of the sequence is 1
*corresponding to the <start>
token) and all the others are zeros (for masking).
Then we take the model output at the first timestep, corresponding to the softmax probability distribution over the vocabulary, and sample one word from this probability distribution (remember that here the 0-th neuron corresponds to the token <start>
which has index 1 in the vocabulary and so on). Sampling can be performed:
These three methods should all be implemented.
When we sample a word, we create the input for the next timestep, made of the same image (which will be encoded by the model but not used - adjust this) and a new caption sequence where the first element is the index of the sampled word and all the other elements are 0. The fact that the model is stateful, means that it will keep the state it had at the previous timestep and can continue the prediction of the sequence without problems. We will sample until MAX_CAPTION_LENGTH
or until <end>
gets produced by the model.
This issue is related to the performances that we want to improve with #1 .
The current implementation of the beam search only works with batch size equal to 1. There are some slight changes that can be made to support mini-batches with a size greater than 1.
Load some model weights so that users can directly use the model without previous training.
Should think about some way to make evaluation (computation of the BLEU scores over different beam widths) faster, since yesterday it required around 4 hours for an evaluation on a grid search over 3 beam widths.
Since there are new files in the project structure, the README should be updated with brief explanations.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.