Code Monkey home page Code Monkey logo

tensorflow-deeplab-lfov's Introduction

DeepLab-TensorFlow

This is an implementation of DeepLab-LargeFOV in TensorFlow for semantic image segmentation on PASCAL VOC dataset.

Model Description

The DeepLab-LargeFOV is built on a fully convolutional variant of the VGG-16 net with several modifications: first, it exploits atrous (dilated) convolutions to increase the field-of-view; second, the number of filters in the last layers is reduced from 4096 to 1024 in order to decrease the memory consumption and the time spent on performing one forward-backward pass; third, it omits the last pooling layers to keep the downsampling ratio of 8.

The model is trained on a mini-batch of images and corresponding ground truth masks with the softmax classifier on the top. During training, the masks are downsampled to match the size of the output from the network; during inference, to acquire the output of the same size as the input, bilinear upsampling is applied. The final segmentation mask is acquired using argmax over unnormalised log scores from the network. Optionally, a fully-connected probabilistic graphical model, namely, CRF, can be applied to refine the final predictions. On the test set of PASCAL VOC, the model shows 70.3% of mean intersection-over-union.

For more details on the underlying model please refer to the following paper:

@article{CP2016Deeplab,
  title={DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs},
  author={Liang-Chieh Chen and George Papandreou and Iasonas Kokkinos and Kevin Murphy and Alan L Yuille},
  journal={arXiv:1606.00915},
  year={2016}
}

Requirements

TensorFlow needs to be installed before running the scripts. TensorFlow>=0.11 is supported.

To install the required python packages (except TensorFlow), run

pip install -r requirements.txt

or for a local installation

pip install -user -r requirements.txt

Caffe to TensorFlow conversion

To imitate the structure of the model, we have used .caffemodel files provided by the authors. The .util/extract_params.py script saves the structure of the network, i.e. the name of the parameters with their corresponding shapes (in TF 'HNWC' format), as well as the weights of those parameters (again, in the TF format). These weights can be used to initialise the variables in the model; otherwise, the filters will be initialised using the Xavier initialisation scheme, and biases will be initiliased as 0s. To use this script you will need to install Caffe. It is optional, and you can download two already converted models (model.ckpt-init and model.ckpt-pretrained) here.

Dataset

To train the network, we use the augmented PASCAL VOC 2012 dataset with 10582 images for training and 1449 images for validation.

Training

We initialised the network from the .caffemodel file provided by the authors. In that model, the last classification layer is randomly initialised using the Xavier scheme with biases set to zeros. The loss function is the pixel-wise softmax loss, and it is optimised using Adam. No weight decay is used.

The train.py script provides an ability to monitor model performance by snapshotting current results: Besides that, one can change the input size and augment data with random scaling.

To see the documentation on each of the training settings run the following:

python train.py --help

Evaluation

After the training, the model shows 57% mIoU on the Pascal VOC 2012 validation dataset. The model initialised from the pre-trained .caffemodel shows 67% mIoU on the same dataset. Note that in the original DeepLab each image is padded so that the input is of size 513x513 and CRF is used, which can be one of the reason of the lower score (~70.3% mIoU).

To see the documentation on each of the evaluation settings run the following:

python evaluate.py --help

Inference

To perform inference over your own images, use the following command:

python inference.py /path/to/your/image /path/to/ckpt/file

This will run the forward pass and save the resulted mask with this colour map:

Missing features

At the moment, the post-processing step with CRF is not implemented. Besides that, the weight decay is missing, as well.

Other implementations

tensorflow-deeplab-lfov's People

Contributors

drsleep avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tensorflow-deeplab-lfov's Issues

deeplab predict a blank segmentation masks

@DrSleep Hi sir, I wish you can help me with this,

System information

What is the top-level directory of the model you are using:
https://github.com/tensorflow/models/tree/master/research/deeplab
** Have I written custom code (as opposed to using a stock example script provided in TensorFlow):**
No
**OS Platform and Distribution (e.g., Linux Ubuntu 16.04):**ubuntu 18.04
**TensorFlow installed from (source or binary):**binary
**TensorFlow version (use command below):**1.13.1 CPU
**Bazel version (if compiling from source):**N/A
**CUDA/cuDNN version:**9.2/7.1
**GPU model and memory:**GeForce GTX 970M

Exact command to reproduce:
python train.py \ --logtostderr \ --vis_split = "train" \ --model_variant = "xception_65" \ --atrous_rates = 6 \ --atrous rates = 12 \ --atrous rates = 18 \ --output_stride = 16 \ --decoder_output_stride = 4 \ --training_number_of_steps = 1000 --train_crop_size = 513 \ --train_batch_size = 1 \ --train_crop_size = 513 \ --Fine_tune_batch_norm=False \ --Tf_initial_checkpoint = "./ Data / Init_models / Deelabv3_pascal_train_aug \ model.ckpt" --Initialize_last_layer = False \ --Last_layers_contain_logits_only = True \ --train_logdir="./data/log/train" \ --dataset_dir="./data/tfrecord" \ --dataset="pascal_voc_seg"

Describe the problem

Hi i trained deeplab model and does not predict anything just a black background, i don't know what's the problem
-training on custom dataset (iris data)
-data = RGB image + ( 0-1) label: 400 * 300
-classe=2
-convert to record format:
-training step:
==> Loss = 0.2 ~ 0.1
-convert to .pb step
python export_model.py \ --logtostderr \ -model_variant = "xception_65" \ --atrous_rates = 6 \ --atrous_rates = 12 \ --atrous_rates = 18 \ --output_stride = 16
until this step everything looks fine
-test step
Capture
1
2
3

Different mean IOU on validation set

Hi,
Thanks a lot for open-source this code. It is very helpful. However, I run into a problem when running the evaluation code on validation set. I did not change anything but just load the model.ckpt-pretrained. But surprisingly, it gets 70.4 instead of 67 as mentioned in the README. Can I ask you what could be the reason?
Thanks a lot!

using my own data

If I want to train a new model on my own data, how to change the number of segmented classes ?

Question

Hello, When I load the ckpt, there is an error.

path = '/home/wam/DeepLab/model/model.ckpt-pretrained'
f = open(path, 'rb')
weights = cPickle.load(f)
Traceback (most recent call last):
File "", line 1, in
EOFError

Regarding the initialization and image preprocessing

  1. Did you reproduce a mIoU of 57.% with all the parameters you provided in the train.py? I just got 52.%....
  2. if there any difference between initializing the deeplab-lfov model with vgg16.ckpt weights pretrained on ImageNet (for the conv1-conv5, nomal distribution for the rest) and with the .caffemodel provided by author?
  3. why images are not loaded to a range of [0,1] and normalized with a standard mean and std if model is initialized with vgg pretrained weights? will it be a cause to the resulted mIoU about 5% lower than author's?

Thank you!

Model transformation problem

Hello, can the tensorflow-based model be trained on the caff-based pre-training model?As a beginner, I am not clear about this question, can you answer for me?
图片

Terrible evaluate results on modified vgg deeplab_v2 structure

Hello DrSleep,

I am not sure whether it is appropriate to ask here, but the tensorflow-deeplab-resnet repo did not contain vgg backbone currently. Therefore, I was trying to modify your lfov codes to use vgg deeplab_v2 structure (i.e. no avg_pool5, and ASPP on fc6 with 4 different atrous holes and sum them up at fc8). Below is the codes I modified (also I modified the network structure "net_skeleton.ckpt" in util folder):

@@ -112,7 +112,7 @@ class DeepLabLFOVModel(object):
         v_idx = 0 # Index variable.
         
         # Last block is the classification layer.
-        for b_idx in xrange(len(dilations) - 1):
+        for b_idx in xrange(len(dilations) - 3):
             for l_idx, dilation in enumerate(dilations[b_idx]):
                 w = self.variables[v_idx * 2]
                 b = self.variables[v_idx * 2 + 1]
@@ -138,18 +138,34 @@ class DeepLabLFOVModel(object):
                                          ksize=[1, ks, ks, 1],
                                          strides=[1, 1, 1, 1],
                                          padding='SAME')
-                current = tf.nn.avg_pool(current, 
-                                         ksize=[1, ks, ks, 1],
-                                         strides=[1, 1, 1, 1],
-                                         padding='SAME')
-            elif b_idx <= 6:
-                current = tf.nn.dropout(current, keep_prob=keep_prob)
+
+        current_before_fc = current
+        fc_layers = []
+        for i in range(len(dilations[5])):   # dilations[5] = [6, 12, 18, 24]
+            w = self.variables[v_idx * 2]
+            b = self.variables[v_idx * 2 + 1]
+            v_idx += 1
+            conv = tf.nn.atrous_conv2d(current_before_fc, w, dilations[5][i], padding='SAME')
+            current = tf.nn.relu(tf.nn.bias_add(conv, b))
+            current = tf.nn.dropout(current, keep_prob=keep_prob)
+
+            w = self.variables[v_idx * 2]
+            b = self.variables[v_idx * 2 + 1]
+            v_idx += 1
+            conv = tf.nn.conv2d(current, w, strides=[1, 1, 1, 1], padding='SAME')
+            current = tf.nn.relu(tf.nn.bias_add(conv, b))
+            current = tf.nn.dropout(current, keep_prob=keep_prob)
+
+            w = self.variables[v_idx * 2]
+            b = self.variables[v_idx * 2 + 1]
+            v_idx += 1
+            conv = tf.nn.conv2d(current, w, strides=[1, 1, 1, 1], padding='SAME')
+            current = tf.nn.bias_add(conv, b)
+
+            fc_layers.append(current)
+
+        current = tf.add_n(fc_layers)
         
-        # Classification layer; no ReLU.
-        w = self.variables[v_idx * 2]
-        b = self.variables[v_idx * 2 + 1]
-        conv = tf.nn.conv2d(current, w, strides=[1, 1, 1, 1], padding='SAME')
-        current = tf.nn.bias_add(conv, b)
 
         return current

It is able to run by loading the trained model provided by http://liangchiehchen.com/projects/released/deeplab_aspp_vgg16/prototxt_and_model.zip. However, the evaluation is terrible, the mean IoU is only 0.5%. (It works good on the original lfov vgg version)

Would you please kindly give me some hints about this issue so that I can create a pull request to include the vgg deeplab_v2 structure?

Thanks!

Decoding .gif segmentation masks

Is there an issue with the decode_label code? I've been working on segmented masks with .gif extension but it throws an error. Here is the stack.

W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: Invalid PNG header, data size 5913
[[Node: create_inputs/DecodePng = DecodePngchannels=1, dtype=DT_UINT8, _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: Invalid PNG header, data size 5913
[[Node: create_inputs/DecodePng = DecodePngchannels=1, dtype=DT_UINT8, _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: Invalid PNG header, data size 5913
[[Node: create_inputs/DecodePng = DecodePngchannels=1, dtype=DT_UINT8, _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/framework/op_kernel.cc:993] Out of range: FIFOQueue '_1_create_inputs/batch/fifo_queue' is closed and has insufficient elements (requested 16, current size 0)
[[Node: create_inputs/batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_UINT8], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/batch/fifo_queue, create_inputs/batch/n)]]

how to use model after training.

Hi, Author, thanks for your work, but I am wondering if I start with only the deeplab_resnet_init.ckpt, and trained my own model with my own set of images, I don't see a deeplab_resnet.ckpt in my folder, I wonder where is the trained model stored and How can I use them.

thanks

the predicted results are all 0s

I has been used the data of VOC2012 and the your code to train the models, but the predicts was all 0s during train process. Would you give me some advises. Thank you very much.

EOFError when using cPickle.load()

When I tried to use the converted model like "model.ckpt-init" or "model.ckpt-pretrained", I met the following error:
Traceback (most recent call last):
File "/home/wxy/Proj/test.py", line 21, in
model = DeepLFOV(weights_path)
......
File "/home/wxy/Proj/test.py", line 53, in _create_variable
weights = cPickle.load(f) # load pre-trained weights
EOFError

Missing segmented images

Hi,
I'm trying to train the model using their train and validation files (train.txt, val.txt in the repositorie) but I have several images missing.
I download the file from the PASCAL Visual Object Classes site, the file was:
VOCtrainval_11-May-2012.tar (1,9 Gb)
But this tar containt only 2913 segmented images, your train.txt have ~10000 images
An other problem is that you path to the segmented images is "/SegmentationClassAug/" but this folder doesn't exists. Only exist "SegmentationClass" and "SegmentationObject"
What is the correct dataset to replicate your treinament?

Regards

Error in Image Reading

In function read_images_from_disk( )
h_new = tf.to_int32(tf.mul(tf.to_float(tf.shape(img)[1]), scale))
w_new = tf.to_int32(tf.mul(tf.to_float(tf.shape(img)[1]), scale))

h_new and w_new have the same value

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.