oslandia / deeposlandia Goto Github PK

When aggregation is done, label IDs are currently chosen as the smallest IDs in each group. However it generates some issues when designing label generators.

One possible solution to overcome this could be to generate new integer IDs from 0 to N-1 (N being the number of aggregated labels), to associate them to aggregated labels, and to keep a trace of original labels in the configuration file.

After PR #68 merging, some symbolic links will be necessary in order to interact with images.

As images are not necessarily stored at the same place depending on the server implementation, and as they are not directly stored in the project repository, the relative information must be managed and maintained for example through a config file.

Adapt the convolutional_layer, maxpooling_layer, fullyconnected_layer methods and so on to the Keras framework.

After a few training iterations, the model converge towards a situation such that popular labels (largely represented in the dataset) are always set as 'true', whilst rare labels are always set as 'false'.

For now, the populate method for AerialDataset implements a double for-loop for image creation, and a similar for-loop for label computing.

One may think that both double loops may be simplified in a single one where operations on images and labels could be undertaken.

Without any visible reason, test_semantic_segmentation_labelling may fails to the following assertion:

assert list(bsum) == list(asum)

The problem seems to come from np.bincount, which does not produces MAX-MIN bins, if largest values are not into a.

Some dosctrings are still written under the basic Sphinx autodoc format. A standardization step must be accomplish: each docstring should be formatted under the numpydoc format.

Adapt the Dataset API so as to consider new behaviors:

• check if a dataset can be used to address a specific research problem (feature detection, object localization, semantic segmentation, ...)
• prepare the dataset in order to use it for a specific research problem

As the training set is too large for being considered once as a whole during training process, it is batched and the corresponding metrics are computed in an accumulative way with TensorFlow value/update ops. However this way of doing seems incompatible with shared tf.Summary.FileWriters, and consequently, training and validation metrics are not plotted together on TensorBoard.

See related Stackeroverflow question for details.

Is there a TensorFlow-compatible workaround for that?

Prepare the model reusing by writing a function dedicated to model testing (i.e. with unknown images).

The dataset refactoring seems to have broken the paramoptim.py script. One gets:

    label_ids = [item['id'] for item in label_config if item['is_evaluate']]
TypeError: 'int' object is not subscriptable

The dataset generator is probably falsely built.

For now, the semantic segmentation model on Tanzania dataset is totally untrained. The goal of this issue is to train a decent model in order to update the web application, and refer to it in the README.

A more ambitious kind of image detection problem is semantic segmentation, i.e. the association of each pixel with a class of the glossary, so as to get a fully-labeled picture (see Mapillary dataset, and some examples in this repo).

Other state-of-the-art models exist in this way, one can refer for instance to SEC or Mask-RCNN.

Results after passing testing dataset looks wrong: true predictions seem unreasonably weak compared false ones. Maybe a problem in the test() method, during batching process, as this observation arises only when testing dataset is batched.

The transition between TensorFlow to Keras has meant a lot of changes in the code. Some new features are also available. It is time for releasing!

Steps to do before releasing:

• merge PR #62
• clean the deeposlandia repository (part of PR #64 )
• change the version number in setup.py (part of PR #64 )
• read the code to catch some minor formatting issues, useless comments or outdated/missing docstrings (part of PR #64 )
• update README.md (part of PR #64 )

Write a summary() method inspired from Keras API. The aim is to get a printable version of the neural network, that provides information about the numbers of layers and parameters in the network.

Adapt the define_batch method so as to undertake the subsequent code with the Keras framework.

Take advantage of the code refactoring to design unit tests concurrently with new API design.

What to test:

• layers (convolution, maxpooling, dense, transposed_convolution)
• dataset creation
• dataset population
• model creation
• model training
• model inference
• model backup

The following command throws an error:

python deeposlandia/datagen.py -D mapillary

It seems that the problem comes from the testing set generation, as the algorithm tries to creates preprocessed version of labelled images (which do not exist for testing set).

This function can be used as a metrics or loss function.

See:

• https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/
• http://angusg.com/writing/2016/12/28/optimizing-iou-semantic-segmentation.html

Add a command argument to specify a desired level of monitoring. For instance:

• level 0 = no monitoring
• level 1 = monitoring of most important scalar variables
• level 2 = monitoring of all scalar variables
• level 3 = full-monitoring (scalar summaries + histogram and image summaries)

In the current state of the code, the learning rate is totally useless: the learning rate is considered as an parameter of the ConvolutionalNeuralNetwork class constructor, however it is not an attribute of this class.

In the new Keras-based API, the learning rate must be defined with the optimizer function, as follows (for instance):

opt = Adam(learning_rate=0.001, decay=0.0001)
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['acc'])

Current implementation of the model do not consider validation during training process. As a consequence we don't know if the model can generalize or if there is any overfitting.

We can have, for instance

deeposlandia/webapp/main.py instead of deeposlandia/webapp.py. And we can move all related web app files into this subpackage.

For now, you cannot have a random data generator because the seed is set by default in the function.

I think you can have a None value. But we have to generate a random integer to set the seed when yo call the Keras function generators because you must have the same seed for the image training generator and label generator.

I think we should separate the input and output folders which can share the same root directory, e.g. root_dir = '../data/streetscene'.

• input folder with the raw datasets

  • input/mapillary where you can find the tranining and validation folders and the further preprocess folders (resized and/or aggregated images)
  • input/shapes

• output folder with the different output result for the different dataset

  • output/mapillary
  • output/shapes

and create a folder for each run. A number of parameter names and the date can occur in the folder name. A meta.json or YAML file inside this folder which summarizes the list of input parameters and hyptohesis.

On the model of other datasets, one should prepare a jupyter notebook to introduce a little bit more the Tanzania dataset (labels, image examples...).

On the model of the job done for feature detection, we could extend semantic segmentation networks to some well-known architectures. See for example this blog post for a review of common architectures. We could implement the following ones:

• U-net : a classic decoder-encoder architecture, more sophisticated than the simple architecture.
• dilated convolutions : use dilated convolutions as a improvement with respect to upsampling operation
• DeepLab (v3) : state-of-the-art architecture with dilated convolutions and a pyramidal structure.

We have some command line parameters which occurred multiple times according to the module: train, test, populate, etc.

We should factorize some parameters in a dedicated module. Maybe we can implement a true "CLI" with subcommands such as:

deeposlandia dataset
deeposlandia train --some-params
deeoposlandia test --some-params --other-params

As a common technique to improve the network performances (measured by accuracy, for instance), the batch normalization encoutered a huge success in neural network literature. See the paper of Szegedy and Ioffe (2012) for more details and TensorFlow documentation to have some clues on how to implement it within the project.

Since the creation of a new file dedicated to metrics (training_metrics.csv) in the instance output folder after each training process, the best checkpoint filename is not recovered.

Indeed we do:

checkpoints = [item for item in os.listdir(output_folder) if os.path.isfile(output_folder, item)]
if len(checkpoints) > 0:
    model_checkpoints = max(checkpoints)

This list comprehension prevents from picking a folder instead of a file, but it does not consider the case of files which are out of the checkpoint-epoch-XXX.h5 (with XXX being the training epoch) template.

Consequently we need to modify the list comprehension if clause.

It seems that the project dependencies are now incomplete, as geospatial data analysis tools are now used with most recent implemented datasets.

In order to communicate the results of the neural network training, one possibility could be to design a small web application. It would allow visitors to use a trained model in order to get feature detection or semantic segmentation relatively to a sample of picture.

To keep it simple: let's imagine a web app that could load a trained model (h5) from a server, do inference on a single image (either loaded from the server, or chosen by the client, but the former seems to be a good first step), and return result onto the web page.

• build a simple Flask web application structure
• select a random Shapes picture from the server
• create an image uploader in order to infer labels on client images
• make the server infer on Shapes image labels for feature detection problem
• make the server infer on Mapillary image labels for feature detection problem
• make the server infer on Shapes image labels for semantic segmentation problem
• make the server infer on Mapillary image labels for semantic segmentation problem

Currently on Tensorboard, the monitored variables appear as <instance_name>_<variable_name>. This should be simplified as <variable_name>.
Example:
shape_128_medium_dashboard_wrapper/shape_128_medium_dashboard_global/acc_global instead of dashboard_wrapper/dashboard_global/acc_global to measure the global accuracy when considering the shape dataset with 128*128 pixel images and medium-sized network.

The shape dataset has to be extended to labeled images, i.e. labeled version of each image must be produced during dataset population, and corresponding file path must be integrated into the dataset structure stored as a json file.

This work has still be done for Mapillary dataset.

Also find a name for the package. Some propositions:

• from street.models import Segmentation
• from deeposlandia.models import Segmentation
• from streetscene.models import Segmentation
• from street_scene.models import Segmentation

Note that the shapes dataset doesn't deal with street scene images. Is is relevant to name it "street*something"?

In the case of semantic segmentation for shapes dataset, it seems that background pixels are as important as regular label pixels. For now, they are not considered in the learning algorithms, however the results are poor. Maybe we can investigate this point to improve semantic segmentation result in such a dataset.

Confusion matrix is normalized regarding the number of labels and the batch size. It works fine when computing the global confusion matrix, however it leads to a wrong normalization result for single label metrics. In such cases, the normalization must be done only considering the batch size.

In order to simplify the code of this project, a major improvement should be to switch from TensorFlow library to Keras one.

Keras would allow us to handle data in an easier way; TensorBoard will be still callable as well.

It seems that checkpoints are not considered any more since the introduction of object-oriented code conception in the project. The accurate files are checked, however the global_step variable is not updated accordingly: consequently the model training begin from scratch whilst there is an old trained version.

python deeposlandia/datagen.py -D mapillary -s 192 gives the following error:

Traceback (most recent call last):
  File "deeposlandia/datagen.py", line 85, in <module>
    train_dataset = MapillaryDataset(args.image_size, config_path)
  File "/home/rde/src/deeposlandia/deeposlandia/dataset.py", line 178, in __init__
    self.build_glossary(glossary_filename)
  File "/home/rde/src/deeposlandia/deeposlandia/dataset.py", line 199, in build_glossary
    label['evaluate'], category, label["contains_id"],
KeyError: 'contains_id'

The glossary can not be built as it expects some keys that are missing in input configuration file. These keys correspond to aggregated case.

A case distinction must be done during the process regarding the label aggregation status.

Some timers could be added in an optional fashion (through a command argument, for instance) to model.py, to measure the different training phases:

• back propagation
• training monitoring
• validation
• back-up

Design a experiment set to optimize the hyperparameters used during training process. This should follow the following steps:

• define which hyperparameters to tune;
• define a possible range for each of these hyperparameters;
• randomly generated a bunch of experiments (random choice is cheaper than grid choice, with a satisfying efficiency);
• run all experiments;
• choose the most accurate set of hyperparameters.

Adapt the Model API so as to make it closer from popular machine learning libraries. Consider these methods:

• fit : train model
• predict : do inference on test data
• save : serialize model
• score : give main metric(s)

The module train.py and test.py follow both the same pattern:

• argument parsing
• folder building
• dataset creation
• dataset populating
• glossary management
• model creation
• model training/testing

Some of these steps could be factorized to enhance the code clarity.

For some unit tests, you generate some data files (JSON, images). Maybe use a pytest fixture to create and clean-up a temporary dir for each test.

See https://docs.pytest.org/en/latest/tmpdir.html#the-tmpdir-factory-fixture

In order to reuse known images in the demo pages, one should consider unique URLs that correspond to every image in the web application.
Exemple:

APP_URL/predictor_demo/semantic_segmentation/shapes/shape_01234.png

will give the prediction result for image shape_01234.png.

For instance, the URL is the same for each image:

APP_URL/predictor_demo/semantic_segmentation/shapes

Metrics (accuracy, IoU, dice_coef) are currently computed for the whole images, without any distinction between classes. To gain on description quality, and be able to compare with state-of-the-art results and contest leaderboards (see Mapillary, CityScapes, or AerialImage examples), we should add class-specific metric results in training_metrics.csv.

It should be interesting to add the best instance metrics in the best-instance-<img_size>-<aggregation>.json file (produced by paramoptim.py when exploring hyperparameters) as well.

When I launch the test with pytest, I've got these messages:

=================== warnings summary =========================
tests/test_generators.py::test_featdet_shape_generator
  deeposlandia/deeposlandia/generator.py:24: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
    mask = img == label

tests/test_generators.py::test_semseg_shape_generator
  deeposlandia/deeposlandia/generator.py:49: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
    mask = img == label

-- Docs: http://doc.pytest.org/en/latest/warnings.html

Refactor model design so as to build it in a object-oriented framework. Consider a Dataset object as well to allow the code generalization (using it with alternative datasets).

Objective: more flexibility, ability to test more network architectures, more datasets...