When I run this command: train(net, optimizer, 50, scheduler), I get an error on Potsdam dataset:
---> 23 losses[iter_] = loss.data[0]
24 mean_losses[iter_] = np.mean(losses[max(0,iter_-100):iter_])

IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number.

Does anyone have this problem?

How would the performance change if using per-image mean rather than dataset-mean for training? Would there be significant degradation in the results?

If I am not wrong, the color value of class 5 should be: 255 255 0
instead, it was mentioned :255 0 255

$ git init
$ git submodule update
Cloning into '/......../DeepNetsForEO/caffe'...
error: no such remote ref ec4f1fff70b7e67a6f9ba31b6a22d685183ecf67
Fetched in submodule path 'caffe', but it did not contain ec4f1fff70b7e67a6f9ba31b6a22d685183ecf67. Direct fetching of that commit failed.

I have just directly git cloned your fork

Dear Dr. Audebert @nshaud :

Thank you for your wonderful work "Beyond RGB: Very high resolution urban remote sensing with multimodal deep networks" [1]. However, I can't find the nDSM of Vaihingen Dataset. Could you provide it ? Thank you very much !

[1] Audebert, Nicolas, Bertrand Le Saux, and Sébastien Lefèvre. "Beyond RGB: Very high resolution urban remote sensing with multimodal deep networks." ISPRS Journal of Photogrammetry and Remote Sensing 140 (2018): 20-32.

Best,

Hi,
@nshaud
what changes should be made to use the CPU rather than the GPU? You have to remove the calls to cuda () definitely, but with what to replace them?
Could you explicitly define all the lines to be modified to work on cpu?

Thanks in advance

Hello,
I am trying to use caffe to perform pointwise classification on earth image. apparently, this is what is done here. Following that tutorial: http://adilmoujahid.com/posts/2016/06/introduction-deep-learning-python-caffe/ , I understand how to use caffe to do label classification. the question is how to work with pointwise classification. I tought that maybe the notebooks mentioned here could help. Could you tell me where could I find the notebooks mentioned.

thank you
colin

Hi, I'm very interested in your work. I have read your paper. It's very helpful. And i wish to know more about your experiments details.
Are the weights of the encoder in V-fusenet initialized with those of VGG-16 trained on ImageNet, or SegNet encoder trained on IRRG images and Composite images?

In convert_gt.py, the mapping in convert_from_color has

(0, 0, 0): 0
(0, 0, 0): 6

while convert_to_color, has the mapping

0: (255, 255, 255)
6: (0, 0, 0)

One of these doesn't seem right. The label images seem to have rgb (255,255,255) for impervious surfaces, which should be getting mapped to 0 in convert_from_color. The script still appears to work correctly as the 2D array is zero initialized.

Hello, I am using the pytorch version of your code to detect wetlands from a 3-band geotiff made up of topographic indices. My verification data is made up of manually surveyed wetland delineations, so it has an irregular shape. I use the wetland surveys as ground truth wetland and nonwetland areas for training and testing purposes by splitting the surveyed area into separate tiles and randomly splitting these into the two groups. Of course, some of the training tiles end up having no data pixels, representing land cover that is not known to be either wetland or nonwetland because it extends further than the limits of the original wetland surveys. If I limit my training set to only those tiles without nodata pixels, I get reasonable predictions. However, I'm already VERY limited in my training data size, relative to most deep learning applications, and would rather include all training data possible by letting the code just ignore these sets of pixels.

So far I've tried:

1. setting the nodata pixels to a value outside of the range(len(LABELS)) i.e., nonwetlands = 0, wetlands = 1 and nodata >1 or < 0
2. including nodata value in the list of target labels i.e., nodata=0, nonwetlands=1, wetlands=2 and then adjusting the WEIGHTS parameter to give nodata a weight of 0.
3. Both 1 and 2 PLUS setting ignore_index = [nodata value] when calling cross_entropy()

These attempts either cause loss = nan or produce poor results after a couple of epochs (predict everything as nonwetland)

Do you have other ideas for signaling to pytorch to ignore a certain type of pixel? I'm very new to deep learning and pytorch so any insight would be very helpful! Thanks! Example training image and corresponding label tile that have nodata pixels

there is something strange in the result.
i train the net and then i run the inference.py script with top_mosaic_09cm_area1.tif
i see that in the gt of this image there is a little mistake and the problem is that in the prediction there is the same mistake.
i was thinking maybe there is a problem with my traning but your pre-trained model has the same result.

this is the gt:

you see that the house is not full covered ?

and this is the prediction:

what do you think?

For me it has no sense that the solver "solver_isprs_vaihingen_irrg.prototxt" in the models folder point to the net with the same name.

The not should not be the "segnet_isprs_vaihingen_irrg.prototxt" file??

Hi, during the execution of training.py, I got this error:

Traceback (most recent call last):
File "training.py", line 223, in
solver = caffe.SGDSolver(SOLVER_FILE)
RuntimeError: basic_string::_S_create

in config.py my SOLVER_FILE is

SOLVER_FILE = '../models/solver_isprs_vaihingen_irrg.prototxt'

Thanks

I use the potsdam data ,when i run the create_lmdb.py i got this error:

Traceback (most recent call last):
  File "create_lmdb.py", line 143, in <module>
    create_image_lmdb(target_folder, samples, bgr=True)
  File "create_lmdb.py", line 92, in create_image_lmdb
    sample = sample[:,:,::-1]
IndexError: too many indices for array

but i test the image independently it runs well:

sample = io.imread('/home/tukrin1/Breeze/rs/Potsdam/RELEASE_FOLDER/Potsdam/potsdam_128_128_32/irrg_train/5411.png')
sample = sample[:,:,::-1]
print sample.shape
>> (128, 128, 3)

i'm so confused about the error ,any idea about what to do ?
thanks

Hi,
In the notebook, the parser to load the Potsdam data must be modified :
Original : all_ids = ["".join(f.split('')[3:5]) for f in all_files]
What works : all_ids = ["".join(f.split('')[5:7]) for f in all_files]

And there is a typo in DATA_FOLDER for Potsdam : it begins with a "3" not with a "Y".

Hello,
I'm new to studying neural networks and Python.
I have such a question, How can I upload my image to the network and how to make a prediction.
I also ported the i-python notebook to google colab. https://drive.google.com/open?id=1fnrPFHrdfKyG9JUTchaOMtXQnxzfqocj
Thank you for understanding.(sorry for my english)

Hello Sir,
Please I would like to change the ISPRS Postdam dataset by another one but I always receive this error in the training
Concerning the error, please refer to the attached screenshot
Thank you in advance for your help
Best regards

Since PyTorch 4.0, some things have been deprecated and should be fixed:

• Variable does not exist anymore
• loss.data[0] should now be loss.item()
• volatile flag does not do anything, use no_grad instead
• .to() method allows us to write device-agnostic code

Dear,
In the segnet_isprs_vaihingen_irrg.prototxt the value of the outputs is equal to 6,
however the real number of outputs of the dataset is 7 ('imp_surfaces', 'building', 'low_vegetation', 'tree', 'car', 'clutter', 'unclassified').

So, I think that I don't understand somethin, because, I try to adapt your example to my dataset with 3 labels (A,B and 'unclassified), setting the value of the outputs to 2, and I obtain an error: 'error == cudaSuccess (77 vs. 0) an illegal memory access was encountered'

I try setting 3 in the outputs, and It works...but this is not what I want, because in this case I'm training the net with the unclassfied values, and I don't want to do this. I only want to train the net with A and B labels.

I try to create the dataset with nan values, but it crash in the lmdb creation.

Is there any way to avoid the unclassified values in the net? Can you help me?
Thanks in advance

python training.py --niter 40000 --update 1000 --init vgg16weights.caffemodel --snapshot trained_network_weights/

I'm getting this error
Caffe definition prototxt written in ./train_segnet.prototxt.
Saved network graph in ./net_figure.png.
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0427 12:48:27.036200 24162 io.cpp:36] Check failed: fd != -1 (-1 vs. -1) File not found: ./reference_models/solver.prototxt
*** Check failure stack trace: ***
Aborted (core dumped)

Any idea! thanks !

when i try to train a model i found this error and i cant solve it.

I1006 22:27:25.480180 26506 layer_factory.hpp:77] Creating layer conv1_2_D_bn
I1006 22:27:25.480190 26506 net.cpp:86] Creating Layer conv1_2_D_bn
I1006 22:27:25.480193 26506 net.cpp:408] conv1_2_D_bn <- conv1_2_D
I1006 22:27:25.480216 26506 net.cpp:369] conv1_2_D_bn -> conv1_2_D (in-place)
I1006 22:27:25.480517 26506 net.cpp:124] Setting up conv1_2_D_bn
I1006 22:27:25.480523 26506 net.cpp:131] Top shape: 1 64 128 128 (1048576)
I1006 22:27:25.480525 26506 net.cpp:139] Memory required for data: 121831424
I1006 22:27:25.480531 26506 layer_factory.hpp:77] Creating layer conv1_2_D_scale
I1006 22:27:25.480554 26506 net.cpp:86] Creating Layer conv1_2_D_scale
I1006 22:27:25.480557 26506 net.cpp:408] conv1_2_D_scale <- conv1_2_D
I1006 22:27:25.480562 26506 net.cpp:369] conv1_2_D_scale -> conv1_2_D (in-place)
I1006 22:27:25.480656 26506 layer_factory.hpp:77] Creating layer conv1_2_D_scale
I1006 22:27:25.480823 26506 net.cpp:124] Setting up conv1_2_D_scale
I1006 22:27:25.480829 26506 net.cpp:131] Top shape: 1 64 128 128 (1048576)
I1006 22:27:25.480832 26506 net.cpp:139] Memory required for data: 126025728
I1006 22:27:25.480836 26506 layer_factory.hpp:77] Creating layer conv1_2_D_relu
I1006 22:27:25.480859 26506 net.cpp:86] Creating Layer conv1_2_D_relu
I1006 22:27:25.480862 26506 net.cpp:408] conv1_2_D_relu <- conv1_2_D
I1006 22:27:25.480867 26506 net.cpp:369] conv1_2_D_relu -> conv1_2_D (in-place)
I1006 22:27:25.481267 26506 net.cpp:124] Setting up conv1_2_D_relu
I1006 22:27:25.481276 26506 net.cpp:131] Top shape: 1 64 128 128 (1048576)
I1006 22:27:25.481278 26506 net.cpp:139] Memory required for data: 130220032
I1006 22:27:25.481281 26506 layer_factory.hpp:77] Creating layer conv1_1_D
I1006 22:27:25.481307 26506 net.cpp:86] Creating Layer conv1_1_D
I1006 22:27:25.481324 26506 net.cpp:408] conv1_1_D <- conv1_2_D
I1006 22:27:25.481328 26506 net.cpp:382] conv1_1_D -> conv1_1_D
F1006 22:27:25.481375 26506 base_conv_layer.cpp:36] Check failed: num_kernel_dims == 1 || num_kernel_dims == num_spatial_axes_ kernel_size must be specified once, or once per spatial dimension (kernel_size specified 0 times; 2 spatial dims).
*** Check failure stack trace: ***
Aborted (core dumped)

did i do something wrong?

help me, thanks.

In the test images from ISPRS images, the predicted image is 96% success. However I have screenshoted a satellite image for the testing, as shown below.

after using the inference_patches, the result is shown below, it seems that at 50% area are failing to predict the crowed vehicles in the parking lot. The up-left side is correctly predicted but can't tell the instances of the vehicles.

I wonder is it because of the training data not enough to predict this case or the vehicles here are very small objects, any approach to improve ?

Additionally, does each image put into the network has to be stick to 128x128 because of our training data is 128x128 size, currently in this test the testing image is roughly 200x200

@nshaud
I just run the default PyTorch code SegNet_PyTorch_v2.ipynb. However, the accuracy of Vaihingen is about 86%, not as good as your paper. Then I read your paper again, and I found that you have made some change. Do you have any plan to share the code in the BeyondRGB paper?

hi ,
this may be a bit silly but , I got an ftp link how to download the datasets from it?

Hello Sir,
Please I tried to apply the Segnet-Pytorch code you developed using the SEMCITY TOULOUSE dataset
(http://rs.ipb.uni-bonn.de/data/semcity-toulouse/)

I want to inform you that I have changed only the classes at the code level.
Please find below the results I obtained:
impervious surface = nan
building = 0.99
pervious surface = nan
high vegetation = nan
car = nan
water = nan
sport venues = nan

Please I would like to know why the nan in the all classes except the Building class
Thank you for your help

Hello

Please could you provide a little explanation of how to get the images from DSM, NDSM, and NDVI.
We can get the nDSM from as you said: The nDSM for the Vaihingen dataset is available here (courtesy of Markus Gerke, see also his webpage).

But from where we can get NDSM and NDVI.

Also, combining those images is the same as doing an RGB image.

Many thanks

I downloaded the data.
when i create the label lmdb i got error because the code assume that the label are 1dim and i have 3dim
you make something to the original label ?
i use this path for the label - "~/ISPRS/Vaihingen/ISPRS_semantic_labeling_Vaihingen/gts_for_participants"
so i change the code for read the label to lmdb.
But of course i have a problem in the loss layer because againe the net expect for 1dim and not 3dim .
you have solution for me?
thenk you

Good morning,
How did you manage to get the DataSet of Vahingen and Postdam ?
Did you contact the webmaster of ISPRS website ?
thank you

$ cd caffe/docker
$ docker build -t gei/caffe-eo:cpu standalone/cpu

produces error:

Traceback (most recent call last):
File "/usr/bin/pip", line 5, in
from pkg_resources import load_entry_point
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 70, in
import packaging.version
ImportError: No module named packaging.version
The command '/bin/sh -c git clone -b ${CLONE_TAG} --depth 1 https://github.com/BVLC/caffe.git . && for req in $(cat python/requirements.txt) pydot; do pip install $req; done &amp;&amp; mkdir build &amp;&amp; cd build &amp;&amp; cmake -DCPU_ONLY=1 .. &amp;&amp; make -j"$(nproc)"' returned a non-zero code: 1

$ docker run -i caffe-eo bash
$ cd /opt/caffe
$ ll
yields no build dir to run make runtest

using BVLC/caffe image =>
$ cd /opt/caffe/build
$ make runtest
...........
[100%] Built target runtest

Hello sir, the data set present in the https://drive.google.com/drive/folders/0B8XVGOkhuqDTdGNibWJPeTcxLVE is not matching with the code.Please provide the exact dataset used for the code!

In the Readme, it is mentioned that:

Data

Our example models are trained on the ISPRS Vaihingen dataset. We use the IRRG tiles (8bit format). The ground truth files are color-encoded and should be converted to the numerical labels, e.g. {0,1,2,3,4,5} instead of {[255,255,255],[0,0,255],[0,255,255],[0,255,0],[255,0,255],[255,0,0]}.

This doesn't appear to be happening in any of the notebooks.

Hi @nshaud
I have a troblem in downloading datasets with ftp,
may you share a google driver link of the dataset?
thank you!

Yours sincerely!

I was having some problems when running the following line ,

Write the train prototxt in a file

f = open(MODEL_FOLDER + 'train_segnet.prototxt', 'w')
f.write(str(net_arch.to_proto()))

I got the error:
'LayerParameter' object has no attribute 'scale'

I don't know what cause the error,how can i resolve this problem?
the following error message is showed:
File "training.py", line 213, in
f.write(str(net_arch.to_proto()))
File "/home/tukrin1/caffe/python/caffe/net_spec.py", line 189, in to_proto
top._to_proto(layers, names, autonames)
File "/home/tukrin1/caffe/python/caffe/net_spec.py", line 97, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "/home/tukrin1/caffe/python/caffe/net_spec.py", line 158, in _to_proto
assign_proto(layer, k, v)
File "/home/tukrin1/caffe/python/caffe/net_spec.py", line 64, in assign_proto
is_repeated_field = hasattr(getattr(proto, name), 'extend')
AttributeError: 'LayerParameter' object has no attribute 'scale'

Could anyone help me with this?
I appreciate your help!

Dear nshaud first i would like to thank you for the code you gave us in github. I am trying to execute this code. i am facing some error when executing training.py. I have attached the screenshot of the error i got. Kindly go through and guide me in solving this issue. I thank you for the effort you have put in.

Good morning,
First, I encountered a problem while executing your code. Didn't figure out why it always return 'invalid syntax' for 'flip' as an argument for Data_augmentation.

/Documents$ python code_Nicolas.py
File "code_Nicolas.py", line 229
def data_augmentation(cls, *arrays, flip=True, mirror=True):
^
SyntaxError: invalid syntax

I tried to remove it. The code is well compiled.
Second problem, I did the training last night, I got these user Warnings :

code_Nicolas.py:573: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
losses[iter_] = loss.data[0]
/.local/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.
out=out, **kwargs)
/.local/lib/python2.7/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
code_Nicolas.py:583: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number

But the program works. The accuracy is always low for all the epoches.
Even after the training, I obtained a total accuracy of about 16% !!!!!
Do you think that both problems are related (could't solve the first problem, I simply removed (flip and mirror))
Thank you

hi,
my mission is to do building segmentation.
i want to ask what you think i should change in the code or my data for make it work

1. my data is rgb for now, its matter? if yes in what way?
2. if i will get dsm data you think its will improve the result?
3. you think to take as label distance like the Yuan 2016 improve the result?

thanks alot for the great code

after the installation, I tried to run the inference_patches.py, an error "ImportError: No module named _caffe" occurred

sys.path.insert(0,CAFFE_ROOT+"/python")
import caffe

I read some other notes with similar problems, which says _caffe.so should be in the /python folder after make pycaffe

I can find the libcaffe.so and libcaffe.so.1.0.0-rc4 in the /caffe/build/lib, but in the /python folder i could find any _caffe.so. to make _caffe.so as suggested by some other, I tried ' make pycaffe ' inside /python folder, but a 'no rule to make target pycaffe ' error occored

I have successfully trained you network using irrg images and got pretty good results. However , in your original paper, you use the IRRG tiles (8bit format) and we build 8bit composite images using the DSM, NDSM and NDVI. Can you share the training prototxt? Or is it the same as the 'potsdam_irrg_osm_fusenet'?

Hi,

Thank you very much for sharing.
I am new to deep learning and Python but successfully setup the environment and
could produce the result.

Is it possible to merge prediction (classification) result as GeoTIFF?

Thanks.

Regards,

Firman.

Hello, I execute without problems convert_gt.py, extract_images.py.
When I execute create_lmdb.py irrg_train_lmdb work correctly, but the labels_train_lmdb give me that error:

`=== Creating LMDB for path/to/labels_train/ ===
0%| | 0/57549 [00:00<?, ?it/s]
Traceback (most recent call last):
File "create_lmdb.py", line 121, in
create_label_lmdb(target_folder, samples)
File "create_lmdb.py", line 94, in create_label_lmdb
label = label.transpose((2,0,1))
ValueError: axes don't match array

Hey,

I want to build composite patches using the DSM, NDSM and NDVI. However when I try to extract patches over normal DSM tiff images using "extract_images.py" script, I face an error saying:

=== PROCESSING DSM === 0%| | 0/3 [00:00<?, ?it/s] Traceback (most recent call last): | 0/57549 [00:00<?, ?it/s] File "extract_images.py", line 126, in <module> io.imsave('{}/{}.png'.format(DATASET_DIR + suffix + '_train', i), sample) File "/home/praveer/anaconda2/lib/python2.7/site-packages/skimage/io/_io.py", line 133, in imsave return call_plugin('imsave', fname, arr, plugin=plugin, **plugin_args) File "/home/praveer/anaconda2/lib/python2.7/site-packages/skimage/io/manage_plugins.py", line 211, in call_plugin return func(*args, **kwargs) File "/home/praveer/anaconda2/lib/python2.7/site-packages/skimage/io/_plugins/pil_plugin.py", line 259, in imsave img = ndarray_to_pil(arr, format_str=format_str) File "/home/praveer/anaconda2/lib/python2.7/site-packages/skimage/io/_plugins/pil_plugin.py", line 171, in ndarray_to_pil arr = img_as_uint(arr) File "/home/praveer/anaconda2/lib/python2.7/site-packages/skimage/util/dtype.py", line 325, in img_as_uint return convert(image, np.uint16, force_copy) File "/home/praveer/anaconda2/lib/python2.7/site-packages/skimage/util/dtype.py", line 205, in convert raise ValueError("Images of type float must be between -1 and 1.") ValueError: Images of type float must be between -1 and 1.
Thanks in advance.

Praveer

Hello, I just tried to implement your model. I compiled caffe successfully. i adapted the script to fit my tiles. I get that issue:

....
I0221 05:26:39.852553 1508 net.cpp:408] conv1_1 <- data
I0221 05:26:39.852568 1508 net.cpp:382] conv1_1 -> conv1_1
I0221 05:26:39.855118 1508 net.cpp:124] Setting up conv1_1
I0221 05:26:39.855144 1508 net.cpp:131] Top shape: 10 64 128 128 (10485760)
I0221 05:26:39.855150 1508 net.cpp:139] Memory required for data: 45875200
I0221 05:26:39.855170 1508 layer_factory.hpp:77] Creating layer conv1_1_bn
I0221 05:26:39.856801 1508 net.cpp:86] Creating Layer conv1_1_bn
I0221 05:26:39.856830 1508 net.cpp:408] conv1_1_bn <- conv1_1
I0221 05:26:39.856847 1508 net.cpp:369] conv1_1_bn -> conv1_1 (in-place)
F0221 05:26:39.856976 1508 batch_norm_layer.cpp:44] Check failed: this-> layer_param_.param(i).lr_mult() == 0.f (0.5 vs. 0) Cannot configure batch normalization statistics as layer parameters.

*** Check failure stack trace: ***

`

any idea about what to do, where to check?
thx
colin

Is it possible to share the code for the FuseNet technique of the paper which takes both the IRRGB images and the Lidar DSMs as input ?

Hi,

Thank you for sharing the source code.

I have set the folder accordingly but I got an error, 'DATA_FOLDER' is not defined

NameError Traceback (most recent call last)
in ()
26 # Dataset class
27
---> 28 class ISPRS_dataset(torch.utils.data.Dataset):
29 def init(self, ids, data_files=DATA_FOLDER, label_files=LABEL_FOLDER,
30 cache=False, augmentation=True):

in ISPRS_dataset()
27
28 class ISPRS_dataset(torch.utils.data.Dataset):
---> 29 def init(self, ids, data_files=DATA_FOLDER, label_files=LABEL_FOLDER,
30 cache=False, augmentation=True):
31 super(ISPRS_dataset, self).init()

NameError: name 'DATA_FOLDER' is not defined