deepsceneseg / adapnet-pp Goto Github PK

Hi,

I've been playing around with your AdapNet++ and SSMA and it looks really good, love the SSMA paper very well written!
Though, regarding the code, there were some troubles getting it all started as there are some things which are not that clear from the notes. So for anyone who is running in to troubles:

• For raining a AdapNet++ network I deleted this part of the code so that it does not try to initialize with anything:

 ckpt = tf.train.get_checkpoint_state(os.path.dirname(os.path.join(config['checkpoint'],
                                                                      'checkpoint')))
    if ckpt and ckpt.model_checkpoint_path:
        saver = tf.train.Saver(max_to_keep=1000)
        saver.restore(sess, ckpt.model_checkpoint_path)
        step = int(ckpt.model_checkpoint_path.split('/')[-1].split('-')[-1])+1
        sess.run(tf.assign(global_step, step))
        print 'Model Loaded'

    else:
        reader = tf.train.NewCheckpointReader(config['intialize'])
        var_str = reader.debug_string()
        name_var = re.findall('[A-Za-z0-9/:_]+ ', var_str)
        import_variables = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)
        initialize_variables = {} 
        for var in import_variables: 
            if var.name+' ' in  name_var:
                initialize_variables[var.name] = var

        saver = tf.train.Saver(initialize_variables)
        saver.restore(save_path=config['intialize'], sess=sess)
        saver = tf.train.Saver(max_to_keep=1000)
        print 'Intialized'

and added saver = tf.train.Saver(). It looks to me as if there is no option for simple training. Both these methods try to initialize with some sort of pre-trained weights..

• There was no __init__.py file in the models folder of the SSMA repo. This results in an import error. This gave me a big headache as I didn't know about the existence or need of this file ;).
• The convert_to_tfrecords.py scripts are different for SSMA and AdapNet repo, a simple len(name) check could make them compatible
• Altered line 29 in convert_to_tfrecords.py to all_list = file_handler.read().splitlines() to get rid of the previously present \n in the list.
• Your images have to be cropped before you convert them.
• Your labels need to be in grayscale. (maybe implement a check?)
• The fact that you have to download this repo, make 2 models and then use these for the SSMA network is a bit cumbersome.

Now my dataset is rather small so I would like to initialize with pre-trained weights as you mention in the notes. I had troubles finding these online. Do you have a download link?

I am also having trouble getting tensorflow to use my gpu. Another environment where I have tensorflow 1.12 works fine. But tensorflow1.4 does not see my gpu for some reason... I have CUDA 9.2 and also installed this wheel: https://github.com/mind/wheels/releases/tag/tf1.4-gpu-cuda9 with no result. You have any tips for this?

Cheers

I am trying to freeze the graph and run Tensorflow Transform Graph on the script, but I am not able to even infer from the frozen graph. I tried the freeze script provided by you, but still facing the issue. The output .pb files, when read to infer exits with ValueError that Input to node passed float when float_ref expected.
Can you suggest a possible fix? Thanks in advance.

I didn't find it by searching, could you point me out? Thank you

Thank you for your amazing work.

I am trying to use Adapnet-pp to segment images of a dataset created by myself involving construction sites scenes. Due to the fact that my dataset is quite small, I would like to use the pretrained weights on Freiburg forest dataset and then fine tune the architecture. According to you, which layers should be freezed and which trained?

Moreover, my dataset has a different number of classes and when I restore the checkpoints it gives me an error, because the shape of last layers depend on the number of classes, which was 6 for Freiburg forest dataset; how should I fix this?

Thank you in advance

Could you please share (to download) pre-trained unimodels for forest dataset: rgb and depth

Hello, I am trying to run the training script with the following config file "config/freiburg_forest_train.config":

gpu_id: '0' model: 'AdapNet_pp' num_classes: 6 intialize: 'init_checkpoint/Adapnet_pp_init.ckpt' checkpoint: ' ' train_data: 'dataset/tf_records' batch_size: 8 skip_step: 5 height: 384 width: 768 max_iteration: 300 learning_rate: 0.000001 save_step: 100 power: 0.9

I am using a GeForce 1060 with 6GB and an Nvidia-docker with tensorFlow-1.14.0-gpu

Not sure what the problem is. I tried to find a solution but couldn't find anything.

The checkpoint files are also in my folder init_checkpoint
-rw-rw-r-- 1 kame kame 90M Sep 13 09:22 AdapNet_pp_init.ckpt.data-00000-of-00001
-rw-rw-r-- 1 kame kame 12K Sep 13 09:22 AdapNet_pp_init.ckpt.index
-rw-rw-r-- 1 kame kame 5.7M Sep 13 09:22 AdapNet_pp_init.ckpt.meta
`

The full Traceback:
Traceback (most recent call last): File "train.py", line 129, in <module> main() File "train.py", line 126, in main train_func(config) File "train.py", line 64, in train_func reader = tf.train.NewCheckpointReader(config['intialize']) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 636, in NewCheckpointReader return CheckpointReader(compat.as_bytes(filepattern)) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 648, in __init__ this = _pywrap_tensorflow_internal.new_CheckpointReader(filename) tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for init_checkpoint/Adapnet_pp_init.ckpt

Hello , thanks for the interesting work , can you provide the inference.py file ?

Thanks for your nice work.
I got two questions.
(1) When training the crop of 768x768, what is the batch size for AdapNet and SSMA? Do you use one GPU or multiple GPUs? If multiple GPUs, need synchronized batch normalization?
(2) What's the full resolution used for training ScanNet? The rgb images' resolution is 1296x968, and depth images' resolution is 640x480. What are the sizes you used to train AdapNet_pp and SSMA.

Hello,

first of all I want to say that I really like your work (the implementation and the AdapNet paper). I am trying to use it for a project and trained a model on my own dataset. Training an evaluation are working fine (as far as i can tell at the moment) but i would like to use the model for some predictions, just like the examples in the readme (image folder) but only in grayscale (same format as used for the creation of the .tfrecords dataset). Unfortunately it seems like there is no possibility to do this with the code from the repository, or maybe it is possible, but I just missed something. I tried to get the predictions from the evaluation script but this did not quite work out. So I wanted to ask how you got the predictions from the trained model, maybe it would even be possible for you to share the code.

Thank you very much :)

Cityscapes (void + 11 classes)

I cannot to load the checkpoint of the RGB model,I use the path in config(checkpoint)

Key resnet_v2_50/beta1_power not found in checkpoint

I get this error while trying to convert my trainset into tf records :

python convert_to_tfrecords1.py --file sorted_train_set.txt --record tf_records_trainset

Exception
Traceback (most recent call last):
File "convert_to_tfrecords1.py", line 54, in convert
assert len(label.shape)==2
AttributeError: 'NoneType' object has no attribute 'shape'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "convert_to_tfrecords1.py", line 98, in
main()
File "convert_to_tfrecords1.py", line 95, in main
convert(file_list, record_name, mean_flag)
File "convert_to_tfrecords1.py", line 57, in convert
raise( AssertionError( "Label should be one channel!" ) )
AssertionError: Label should be one channel!

Note that images are read in grayscale as I modified line 52 to :
label = cv2.imread(name[1], cv2.IMREAD_GRAYSCALE)
in order for it to work with converting the trainset ,but I don't understand why it's refusing to work for the train set .

I have a dataset containing both RGB and Segmented images as ground truth , the readme.txt included in the annotated dataset stated this :

GT_color :folder containing the groundtruth masks for semantic segmentation Annotations are given using a color representation, where each corresponds to a specific class. This is primairly provided for visualization. For training, create a corresponding ID image by assigning the colors to a specific class ID as given below.

Class R G B ID

Void - - - 0

Road 170 170 170 1

Grass 0 255 0 2

Vegetation 102 102 51 3

Tree 0 60 0 3

Sky 0 120 255 4

Obstacle 0 0 0 5

I don't understand what's meant by "creating corresponding ID image" , aren't the segmented images labelled already by each area's color ? meaning that the rgb are the labels ?

I've tried to restore the model from SUN RGB checkpoint. Yet, I've received an error:

> NotFoundError (see above for traceback): Key resnet_v2_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint

OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key resnet_v2_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint

Here's my config:

gpu_id: '0'
model: 'SSMA'
num_classes: 38
checkpoint: '/home/user_foo/Downloads/sun_ssma_depth/model'
test_data: '/home/user_foo/workspace/ssma/ssma_source/dataset/tf_records_name'
batch_size: 5
skip_step: 1
height: 384
width: 768

I haven't received such an error with any other pretrained models.

I try to train the model on YCB dataset and the result on evaluating the test data from dataset is good but the result of real-world data is not very good. Is there any data preprocessing method missing in the training phase? Because I didn't find it. Thanks a lot!

Data
Augment the training data. In our work, we first resized the images in the dataset to 768x384 pixels and then apply a series of augmentations (random_flip, random_scale and random_crop).

Convert the training data (augmented), test data and validation data into the .tfrecords format. Create a .txt file for each set having entries in the following format:

how to get the training data(augmented) ? just to process the picture and then save it ,and put the new image to the txt file?

Hi,
Thanks for your great job, it inspired me deeply. The class of standard evaulation of cityscape if of 19 classes. If its convenient to share corresponding trained models?

Hi,

I am trying to use your compute_iou() function but I think my inputs to the compute_output_matrix() function are not what they should be... At the moment I am trying this:

num_classes= 3
output_matrix = compute_output_matrix(label_max =2,pred_max=2, output_matrix=model.softmax)
iou = comput_iou(output_matrix)

Assuming that the max label = max pred = num_classes -1?
However I get a NoneType index..

Side note

While scanning through your code I saw this:

 ### Upsample/Decoder
        with tf.variable_scope('conv41'):
            self.deconv_up1 = self.tconv2d(self.eAspp_out, 4, 256, 2)
            self.deconv_up1 = self.batch_norm(self.deconv_up1)

        self.up1 = self.conv_batchN_relu(tf.concat((self.deconv_up1, self.skip2), 3), 3, 1, 256, name='conv89') 
        self.up1 = self.conv_batchN_relu(self.up1, 3, 1, 256, name='conv96')
        with tf.variable_scope('conv16'):
            self.deconv_up2 = self.tconv2d(self.up1, 4, 256, 2)
            self.deconv_up2 = self.batch_norm(self.deconv_up2)
        self.up2 = self.conv_batchN_relu(tf.concat((self.deconv_up2, self.skip1), 3), 3, 1, 256, name='conv88') 
        self.up2 = self.conv_batchN_relu(self.up2, 3, 1, 256, name='conv95')
        self.up2 = self.conv_batchN_relu(self.up2, 1, 1, self.num_classes, name='conv78')

And from reading your paper it seems that skip1 and skip2 should be the other way around no?

Thank you for sharing your great work.
Let me ask you a question about the provided initial weights init_checkpoint/Adapnet_pp_init.ckpt.*. According to the model definition in models/AdapNet_pp.py, this codebase implements ResNet v2 modified with the proposed multiscale residual units within block 3&4. However, I found the initial weights rather include parameters for ResNet v1. Is this appropriate to reproduce results or you have another file for the modified ResNet v2 trained on ImageNet?

Hi,

I have run the adaptnet-pp model on my own dataset with 2 classes. Can you guide me on how to convert the final checkpoint into a single ".pb" file which I can use for inference on other images?

Moreover, I will also need the names of the input and output nodes to be able to export the model checkpoint into a .pb file.

Looking forward to hearing back from you.

I ran the evaluate.py on the freiburg forest test data using the checkpoint provided in your readme for RGB data, but the mIoU is this:
mIoU: 3.464414239463952
mIoU: 4.031567725489401 total_data: 135

I'm clearly not doing something right.

the num_classes=12, the checkpoint is the RGB model:
when I use the checkpoint to evalute:
8.62191438941 total_data: 500

why it is so low? -->I think it is not the finally model

when I train it ,use the intialize, the has the error:

ValueError: No variables to save

Originally posted by @coldsheephot in #9 (comment)

My environment : python 3.6.7 and Tesnorflow 1.10 .

• I augmented and resized the annotated images within the freiburg dataset and was able to generate train 70000 RGB images and their segmented ground truth labels .

• generated two text files (train and test ) containing full paths as mentioned in the README

• created the tf_records_train file and the tf_records_test file .

• Resolved the python2 -> python3 errors .

I run train.py with the initialization checkpoint provided : I always get loss = 0 .

I change the config file and feed in the pre-trained checkpoint (the one where RGB mIoU = 83.18) and I replace in line 70 and 71 var.name with var.name.replace(':0','') in train.py then run it : I get No variables to save (yes I'm sure of the pretrained chkpt path , I've even put it in the init folder next to the init chkpt )

I try to run evaluate.py with : init chkpt , the pretrained model.chkpt you provided , and a checkpoint of a training that has been done on 3400 imges (still going to test if I train the 70K images that will change something ) and I get low mIoU of respectively 8.xxx , 8.xxx and 10.xxxx and this has been tested on both 20K test dataset and a few hundreds of images dataset , yet resulting in the same results .

I've been stuck with this for days , and I would like any clear explanation on what I missed or any fix the issue .