lmb-freiburg / demon Goto Github PK

Hello,

I am trying to finetune your iterative network with my own dataset.

I understand the ground truth must be:

• image pair [bs, 6, 192, 256] in range [-0.5, 0.5]
• inverse depth [bs, 1, 48, 64] in range [0.0, inf]
• normals [bs, 3, 48, 64] in range [-1.0, 1.0]
• optical flow [bs, 2, 48, 64] in range [-1.0, 1.0] bc x/y components scaled by image width/height
• rotation [bs, 3] where norm is the angle
• translation [bs, 3] which is a unit vector

However, at the start of training, I'm comparing the ranges of the ground truth to predictions from the iterative net:

[name][min][max]

[gt depth][0.0015259022][0.25714538]
[pred depth][-4.7255404e+09][4.8389247e+10]

[gt flow][-0.33519724][0.31490907]
[pred flow][-1.160358][0.12831117]

[gt normals][-1][0.99998885]
[pred normals][-3502147.5][2898221.5]

The ranges look reasonable for flow, however the predictions for depth and normals are much larger than the gt. I am generating the gt normals using the depth_to_normals function from lmbspecialops with my gt inverse depth.

Am I missing something with my preprocessing? For reference, I'm using the virtual kitti dataset.

Thank you for your time,
Sam

Hello!

I want to finetune your network on my dataset. I have a couple questions about how you initially trained the network.

First you trained the four encoder-decoders in the bootstrap and iterative net sequentially for 250k iterations. This is straightforward.

Second you trained the iterative net for 1600k iterations. You appended the predictions of the iterative net to the inputs to the iterative net to emulate having 4x iterative nets while only having one net in memory.

• Is the loss only defined / backpropagated with the final predictions (after the 4x iterations)? Or is the loss backpropagated after each iteration?
• If the loss is only backpropagated after all four iterations of the iterative net, is a single minibatch iteration (in the sense that it was trained for 1600k iterations) defined as one forward-pass of the bootstrap and iterative nets or 4x forward passes of the bootstrap and iterative nets (a single backwards pass)?
• I'm unclear about what this means: "We also train the first iteration on its own, but then train all iterations jointly which avoids intermediate storage." Is this referring to when you trained the iterative net as part of the first 250k iterations?

Finally, do you see any need to finetune the refinement net or is it sufficient to finetune the iterative net with simulated iterations?

Thank you!

Sam Pepose

I generated solution file with cmake and tried building the code with visual studio (2010, 2012, 2013), but I am getting some errors:

Error	1	error MSB6006: "cmd.exe" exited with code 9009.	C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.CppCommon.targets	170	5	webp

Error	2	error D8021: invalid numeric argument '/Wabi-tag'	E:\Vijay Bhardwaj\github repos\demon-master-new\lmbspecialops\build\multivih5datareaderop\cl	multivih5datareaderop

When I looked at the output window, it shows that

'CFLAGS' is not recognized as an internal or external command,
operable program or batch file.
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.CppCommon.targets(170,5): error MSB6006: "cmd.exe" exited with code 9009.

Another line says:

cl : Command line error D8021: invalid numeric argument '/Wabi-tag'

Can you help me out with this??

hello，
when i make,i have some errors as follows.
thank you for your answer.

[ 37%] Building NVCC (Device) object lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_warp2d_cuda.cu.o
/home/oem/demon/lmbspecialops/src/warp2d_cuda.cu:20:49: fatal error: tensorflow/core/framework/op_kernel.h: No such file or directory
compilation terminated.
CMake Error at lmbspecialops_generated_warp2d_cuda.cu.o.Release.cmake:222 (message):
Error generating
/home/oem/demon/lmbspecialops/build/lib/CMakeFiles/lmbspecialops.dir//./lmbspecialops_generated_warp2d_cuda.cu.o

lib/CMakeFiles/lmbspecialops.dir/build.make:112: recipe for target 'lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_warp2d_cuda.cu.o' failed
make[2]: *** [lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_warp2d_cuda.cu.o] Error 1
CMakeFiles/Makefile2:127: recipe for target 'lib/CMakeFiles/lmbspecialops.dir/all' failed
make[1]: *** [lib/CMakeFiles/lmbspecialops.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2

I am trying to run demon against a set of images I extracted from a video passing them into bootstrap_net.eval. The depth images I get back are not smooth from one image to the next but have a lot of inconsistency between pairs. The project web site has a video of a road with a bicyclist that looks like it was webcam or video. I normalize the depth images but this did not help. How was the project video made?

While building lmbspecialops:

[ 33%] No install step for 'lz4'
[ 35%] Completed 'lz4'
[ 35%] Built target lz4
[ 37%] Building NVCC (Device) object lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_median3x3downsample_cuda.cu.o
CMake Error at lmbspecialops_generated_median3x3downsample_cuda.cu.o.cmake:207 (message):
  Error generating
  /home/sharkubuntu/demon/lmbspecialops/build/lib/CMakeFiles/lmbspecialops.dir//./lmbspecialops_generated_median3x3downsample_cuda.cu.o


lib/CMakeFiles/lmbspecialops.dir/build.make:112: recipe for target 'lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_median3x3downsample_cuda.cu.o' failed
make[2]: *** [lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_median3x3downsample_cuda.cu.o] Error 1
CMakeFiles/Makefile2:164: recipe for target 'lib/CMakeFiles/lmbspecialops.dir/all' failed
make[1]: *** [lib/CMakeFiles/lmbspecialops.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2

python3 -c 'import tensorflow as tf; print(tf.version)':
1.6.0

nvcc --version:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

lsb_release -a:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial

cmake --version:
cmake version 3.5.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

Hello.
Thanks for the code.
I have a question about the data reader.
Is the image distortion is pre-distorted when build the hdf5 file for rgbd-slam dataset ?
Thanks.

when I run python3 example.py,I get "module 'lmbspecialops' has no attribute 'leaky_relu'"

I think the scaling factor can be trained like rotation and translation, because the true scaling factor can be obtained by normalizing translation when training, is it right? But in the DeMoN paper, it seems the scale is only trained together with the inverse depth.

hello,
I wonder that if you use the Matlab toolbox provided by nyuv2 website to impaint holes in raw depth map, or you just use raw depth staightforward?
thank you~

tensorflow-gpu 1.0.0
python 3.5
...
All the libs satisfied with the requirements specified in readme.
But got errors in the make step.

'demon/lmbspecialops/src/decode_lz4_op.cc:43:29: error: ‘class tensorflow::shape_inference::InferenceContext’ has no member named ‘MakeShapeFromPartialTensorShape’
TF_RETURN_IF_ERROR(c->MakeShapeFromPartialTensorShape(expected_shape, &out));'

Hello,

Is there a simple way to apply DeMoN to batches of images? Recent GPUs have plenty of memory and inference could be sped up by spending less time moving data around. There is a hardcoded batch size of 1, for example here ff. So just pushing batches of data into DeMoN does not work, sadly.

I am wondering how complex the required changes are. Are there any implementation details that forbid batched inference? Maybe you can give some hints about where non trivial changes are neccessary

Thanks

File "create_dataset_and_use_readerop.py", line 79, in
reader_tensors = multi_vi_h5_data_reader(len(data_tensors_keys), json.dumps(reader_params))
TypeError: 'NoneType' object is not callable

Panorma has more context, but what is the best practice to get the depth map for 2 paired panorama? should we change it to cubemap and estimate each face and stitch them together.. NOt sure, if the individual face is consistent with each other's result. Or should we consider directly use the distorted panorama for training?

Hello @benjaminum ,
Thank you and your partners so much about the greate Computer Vision work.
Today, I have tried to build your branch master with Ubuntu 18.04, after some tweaks with CMake (I disabled CUDA as I don't have NVidia GPU available and link the Python path to Python 3.6.5, TENSORFLOW_LIB_DIR to tensorflow_framework.so as I use Tensorflow 1.8 binary with naive pip).
But the make stage show me error on
#include "tensorflow/core/framework/op.h"
Is .so file I linked is incorrect and if so, which file I have to link with?
And if not, could you please tell me the solution?
Thanks in advance.

Could you give an example on how to use the intrinsics to crop and image as stated in example. Not sure how to proceed.

I want to learn the training network of this model and make some improvements on it, but I do not know how to train. Can you open source the training network? Thanks~

Hi,

When I run python3 example.py, an errror message showed up:
Traceback (most recent call last):
File "example.py", line 12, in
from depthmotionnet.networks_original import *
File "../python/depthmotionnet/networks_original.py", line 18, in
from .blocks_original import *
File "../python/depthmotionnet/blocks_original.py", line 19, in
from .helpers import *
File "../python/depthmotionnet/helpers.py", line 19, in
import lmbspecialops as sops
File "/home/bingyao/demon/lmbspecialops/python/lmbspecialops.py", line 30, in
lmbspecialopslib = tf.load_op_library(_lib_path)
File "/home/bingyao/.local/share/virtualenvs/demon_venv/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
File "/home/bingyao/.local/share/virtualenvs/demon_venv/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /home/bingyao/demon/lmbspecialops/build/lib/lmbspecialops.so: undefined symbol: _ZTIN10tensorflow8OpKernelE

Any ideas?
Thanks,
Bingyao

After around 10 days of training, the script generates the snapshot-250000.data-00000-of-00001 of the refine stage. However, plugging these newly computed weights to the provided example.py script I am getting errors starting with Not found: Key netDM1/motion_conv1/kernel not found in checkpoint...:

The only suspicious thing I saw during training is lots of warnings on empty queues or something like that.. but these seem to go away after a couple of stages of training.

I'm currently using rgbd_dataset_freiburg1_xyz as input of the example code. However, the output of the depth map prediction is so dark that i cant even see the architecture of the scene comparing to the ground truth provided by the dataset. Is the output depth map of the example code inverted? or it is the actual depth? also if i wish to compare the depth map to the ground truth, i will need to scale up the output of the example, how can i obtain the scale between the output and the ground truth?

Hi, in the paper, you wrote "We resolve the scale ambiguity by normalizing translations and depth values such that ||t||=1."  Now, I want to obtain actual translation (not normalized). Is it possible?

To obtain the real depth map , I invert the pixel value of the output depth map, multiply it with the scaling factor s. However, the result is not very good. I wonder if I have made some mistakes here?

Hi,

When I run python3 create_dataset_and_use_readerop.py , an errror message showed up:
Traceback (most recent call last):
File "../python/depthmotionnet/dataset_tools/lz4.py", line 24, in
liblz4 = CDLL(_lib_path)
File "/usr/lib/python3.5/ctypes/init.py", line 347, in init
self._handle = _dlopen(self._name, mode)
OSError: /media/deep/文档/ubuntubf/test/demon/build/lz4/src/lz4-build/liblz4.so: cannot open shared object file: No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "../python/depthmotionnet/dataset_tools/lz4.py", line 28, in
liblz4 = CDLL('liblz4.so')
File "/usr/lib/python3.5/ctypes/init.py", line 347, in init
self._handle = _dlopen(self._name, mode)
OSError: liblz4.so: cannot open shared object file: No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "create_dataset_and_use_readerop.py", line 12, in
from depthmotionnet.dataset_tools import *
File "../python/depthmotionnet/dataset_tools/init.py", line 19, in
from .view_io import *
File "../python/depthmotionnet/dataset_tools/view_io.py", line 21, in
from .lz4 import lz4_uncompress, lz4_compress_HC
File "../python/depthmotionnet/dataset_tools/lz4.py", line 30, in
raise RuntimeError('Cannot load liblz4.so')
RuntimeError: Cannot load liblz4.so

Any ideas?
Thanks,
Zhang-jn

Did you do any data augmentation (DA) for training DeMoN just as FlowNet2? It seems a little difficult to performe DA because of different kinds of inputs, such as flow, warped image and depth. If you did DA, could you give me some advice about how to perform DA for DeMoN?

The image pairs used in training don't look like the stereo pairs. Are they consecutive frames ?
How are these 'unconstrained image pairs' considered for training?
Also for prediction, image pairs are required in the same manner?

Hi Just wonder what kind of data the release model is trained on ?

Is it also possible to have other models about your prediction of different scenes ? Or you use the same model for both indoor and outdoor images ?

Thanks

What is the physical unit of the final depth map? Is there a easy way to obtain a depth map e.g. in meters?

I guess one has to first invert the inverse depth back to the physical depth. But then it still needs to be converted into SI-units. I tried to eyeball the conversion factor, but could not find a consistent number.

I could imagine that the scale factor (the s from in the paper) cannot be estimated very accurately. Just imagine having two image pairs: one of a usual lego figure, and one of a human sized one. There would be no way to tell them apart right?

Just to note VTK for Python 3 is available with Anaconda

$ conda search vtk
Fetching package metadata ...........
pyvtk 0.5.18 py27_0 conda-forge
0.5.18 py35_0 conda-forge
0.5.18 py36_0 conda-forge
vtk 7.0.0 py27_0 conda-forge
7.0.0 py35_0 conda-forge
7.0.0 py27_1 conda-forge
7.0.0 py35_1 conda-forge
7.0.0 py27_2 conda-forge
7.0.0 py35_2 conda-forge
7.1.0 py27_0 conda-forge
7.1.0 py35_0 conda-forge
7.1.0 py27_1 conda-forge
7.1.0 py35_1 conda-forge
7.1.0 py27_2 conda-forge
7.1.0 py35_2 conda-forge
7.1.0 py27_3 conda-forge
7.1.0 py35_3 conda-forge
7.1.0 py36_3 conda-forge
7.1.0 py27_4 conda-forge
7.1.0 py35_4 conda-forge
7.1.0 py36_4 conda-forge
5.10.1 py27_0 defaults
5.10.1 py27_1 defaults
6.3.0 py27_0 defaults
6.3.0 py27_1 defaults

In the paper, it states one of output of the fully connected layers is a depth scale factor s. The scale factor s relates the scale of the depth values to the camera motion. Ref: Figure 3 in the paper.

Looking at the example code and then the source code, I am unable to figure out how I can get the scale factor value. What am I missing? Any help appreciated.

Thanks for this great work. Just one important question. I would like to try this with images using an intrinsic matrix, let's call this K. I see that the network was trained using a fixed intrinsic matrix, let's call it Khat.

The way to do this would seem to be to warp my images by a corrective homography H = Khat*inv(K). Then I can pass the warped images to the network. This seems fine, but the main problem with this is when K has a longer focal length than Khat, my warped image will need to be padded at the borders. Standard padding techniques (zero padding) gives very bad results.

Can you see any simple way that this can be resolved? It's related to you mentioning cropping as a way to use a different intrinsic matrix, however if the focal length is longer we can't deal with it by cropping.

Much appreciated for your great paper and code, Toby

I am wondering is there are training script for this as well. Thank you.

Hi, thanks for such an interesting work and making DeMoN open source.

I have a question regarding the output format of the relative poses predicted in the example.py file. It is clear that translation is a normalised unit vector with 3 elements. But for the rotation prediction, it is also a 3-element vector. What are these values? How does the vector represent the rotation, as mentioned in the paper " The rotation r = θv is an angle axis representation with angle θ and axis v". I just want to convert it to a rotation matrix. Would you please share more details? Thanks a lot!

Hi @benjaminum, have you ever successfully trained the network with your latest v2 training code? I've been experiment with it for a few days, finding it's extremely hard to converge.
This is my training loss status:

(please ignore TotalLoss_1)
This is trained within about 500k iterations with default start learning rate and batch size on the first evolution. I also tried other lr and optimizer configurations, the result is similar.

1. If you do have successfully trained the network, would you like to share the hyperparameters you used?
2. I noticed that the implementation of the network is changed a little bit in blocks.py, could that be a reason? Why would you make this change and what is v2 means exactly..
3. What are the remaining tasks for training code, would you like to share your progress?

Thanks for this amazing work!

Hi,
Thank you for your last help. Maybe this problem needs your help again this time.
When I run python training.py, an errror message showed up:
Traceback (most recent call last):
...
Tensor("tower_0/netFlow1/dense5/LeakyRelu:0", shape=(16, 4608), dtype=float32, device=/device:GPU:0)
Tensor("tower_1/netFlow1/dense5/LeakyRelu:0", shape=(16, 4608), dtype=float32, device=/device:GPU:1)
nothing to restore. no checkpoint found.
train evolution "0_flow1" ...
start iteration 0
2017-12-19 22:36:48.508170: I tensorflow/stream_executor/dso_loader.cc:139] successfully opened CUDA library libcupti.so.8.0 locally
Segmentation fault(core dumped)

Any ideas?
Thanks,
JUNZHANG

Dear，
I tried to evaluate DeMoN using l1-inv and l1-rel as stated in the paper. However, I found the value does not consist with the paper:
1, the output depth is scaled or not? it equals to the depth when the norm translation equals to 1.0?
2, Even using S_{log} as stated in the supplementary material, l1-inv is still bigger than paper reported.
One difference is that I evaluate all the pixels, instead of two-frame visible pixels. Is this a big difference?

In the examples there're files: sculpture_Rt1.txt and sculpture_Rt2.txt with camera positions.
I cannot find a place where you're actually calculating those positions - also they're not updating after running example.py.
Could you explain how to use function for calculating them properly ?
Also - I am planning to generate a series of depthmaps from the same scene - and would like to have a camera positions for each of them relatively to each other in one coordinate system - any tips, what's best approach to do that ?

Thanks for sharing the work, great and impressive piece of code.

Hello!

From the sculpture example, I was able to deduce that the network is predicting egomotion in a coordinate system where +x points left, +y points up, +z points backwards. Is this correct? If so, I'm curious why this system was chosen. I'm not sure if there's a "standard", I just noticed that vkitti uses +x right, +y down, +z forward.

Best,
Sam

I runed the example.py, and it can show the depth map, but cannot visualize the pointclound. I installed the vtk on python3. and no error on "import vtk".

So please help me solve this error. Its about Cython? OR other? THANKS A LOT~

Hi, when I run training.py, I got such error message, any idea please.
by the way, the tensorflow version is 1.4.0, not compiled.

<module 'depthmotionnet.datareader' from '/home/kylin/demon/python/depthmotionnet/datareader/init.py'>
Tensor("tower_0/netFlow1/dense5/LeakyRelu:0", shape=(32, 4608), dtype=float32, device=/device:GPU:0)
nothing to restore. no checkpoint found.
train evolution "0_flow1" ...
start iteration 0
2018-05-17 19:37:59.029835: E tensorflow/core/common_runtime/executor.cc:643] Executor failed to create kernel. Invalid argument: Conv2DCustomBackpropInputOp only supports NHWC.
[[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]]
stop iteration 0
Traceback (most recent call last):
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
return fn(*args)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
status, run_metadata)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC.
[[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "training.py", line 604, in
tf.app.run()
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "training.py", line 598, in main
trainer.coordinator().raise_requested_exception()
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/training/coordinator.py", line 407, in raise_requested_exception
six.reraise(*self._exc_info_to_raise)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/six.py", line 693, in reraise
raise value
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/training/coordinator.py", line 296, in stop_on_exception
yield
File "/home/kylin/Applications/tfutils/python/tfutils/simpletrainer.py", line 280, in mainloop
values = self._session.run(ops, options=run_options, run_metadata=run_metadata)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC.
[[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]]

Caused by op 'tower_0/netFlow1/refine4/upconv/conv2d_transpose', defined at:
File "training.py", line 604, in
tf.app.run()
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "training.py", line 193, in main
kernel_regularizer=_kernel_regularizer,
File "/home/kylin/demon/python/depthmotionnet/v2/blocks.py", line 231, in flow_block
**conv_params,
File "/home/kylin/demon/python/depthmotionnet/v2/blocks.py", line 108, in _refine
**kwargs,
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/convolutional.py", line 1423, in conv2d_transpose
return layer.apply(inputs)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/base.py", line 671, in apply
return self.call(inputs, *args, **kwargs)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/base.py", line 575, in call
outputs = self.call(inputs, *args, **kwargs)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/convolutional.py", line 1289, in call
data_format=utils.convert_data_format(self.data_format, ndim=4))
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 1211, in conv2d_transpose
name=name)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 779, in conv2d_backprop_input
use_cudnn_on_gpu=use_cudnn_on_gpu, data_format=data_format, name=name)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Conv2DCustomBackpropInputOp only supports NHWC.
[[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]]

I've been trying to run this with a webcam and have been getting very odd segment faults in tensorflow. I was wondering if you have tried anything like this.

the basic code loop is:

img1 = Image.open(os.path.join(examples_dir,'sculpture2.png'))
img2 = Image.open(os.path.join(examples_dir,'sculpture1.png'))
video = cv2.VideoCapture('/home/rjn/data/videos/city.mp4')

while(True):

ok, buf = video.read()  
buf = cv2.cvtColor(buf,cv2.COLOR_BGR2RGB)
img2 = Image.fromarray(buf)
  *
  *
  *

img1 = img2.copy()

HI Benjamin,

Bug in your depth2normal code ? I recently use your depth2normal code and find the tensorflow implementation.

    const T cx = 1/(intrinsics[4*z+2]*x_size);
    const T cy = 1/(intrinsics[4*z+3]*y_size);

you have cx, cy as inverse of the input. I think it might be a bug since when we compute cx, cy, we do not need to inverse it.

Hello, thanks for your sharing with demon.

I have already run your evaluation.py successfully, and result shows your method very effective.

Now I want to create a new dataset(such as rgbd slam download from tum) and train this new dataset use your project, what should I do?

Could you give whole source code of training ?

Could you give list of rgbd-slam dataset which you used for training and dataset select code ?

Thanks again !

Hi, I have test your vtk code and create point clouds of one single image (fr3_teddy).
But I cannot get quality result if input multi-images (use your rgbd_test.h5 file).

Have you reconstruction 3D point clouds use multi-images?
Can you send me your reconstruction code if you have done this work?
My email address [email protected]. Thks a lot

Here are my test results:

• 1. gts pointclouds use gts_rt and gts_depth

• 2.DeMoN esti. results
  

Are there some errors in my test?

I'm currently testing the tool using the example.py with sculpture1 and 2. I'm running it with CPU and getting the following error statement:

2017-09-05 11:01:05.625707: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-05 11:01:05.625729: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-05 11:01:05.625739: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
In file included from /usr/include/numpy/ndarraytypes.h:1761:0,
                 from /usr/include/numpy/ndarrayobject.h:17,
                 from /usr/include/numpy/arrayobject.h:4,
                 from /home/zurich/.pyxbld/temp.linux-x86_64-3.5/pyrex/depthmotionnet/vis_cython.c:485:
/usr/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
Traceback (most recent call last):
  File "example.py", line 112, in <module>
    translation=translation)
  File "../python/depthmotionnet/vis.py", line 280, in visualize_prediction
    pointcloud = compute_point_cloud_from_depthmap(depth, K, R1, t1, n, img)
  File "../python/depthmotionnet/vis.py", line 47, in compute_point_cloud_from_depthmap
    return _compute_point_cloud_from_depthmap(depth, K, R, t, normals, colors)
  File "vis_cython.pyx", line 171, in depthmotionnet.vis_cython.compute_point_cloud_from_depthmap
ValueError: shape mismatch: colors (3, 256, 6), depth (192, 256)

I'm using an Anaconda environment with:

• Python 3.5.4
• tensorflow 1.3.0
• VTK 7.1.1
• Cmake 3.9.1

The Depth map I get is correct, but when it gets to the point cloud the error statement appears.

Thank you in andvanced.
Best regards,
Zuria.

I try to follow the readme but cannot get it right. I am using tensorflow 1.4.0 with python 3.5.2, on ubuntu 16.04, here is the error message for python3:

demon_venv tianwei@tianwei:~/Documents/Projects/demon/examples$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.

import lmbspecialops
Traceback (most recent call last):
File "", line 1, in
File "/home/tianwei/Documents/Projects/demon/lmbspecialops/python/lmbspecialops.py", line 30, in
lmbspecialopslib = tf.load_op_library(_lib_path)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/load_library.py", line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /home/tianwei/Documents/Projects/demon/lmbspecialops/build/lib/lmbspecialops.so: undefined symbol: _ZN10tensorflow11GetNodeAttrERKNS_9AttrSliceENS_11StringPieceEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

I'm using Demon to compare it with other Depthmaps and I have some trouble with the units of the output of Demon. Could you tell me in which units is Demon operating?

MultiGPU training with netFlow1 works fine to me, while for the next evolution training some tensors are shown 'None values'.
I'm running on a machine installed with 4 GPUs, while only feed the program two of them by CUDA_VISIBLE_DEVICES=2,3. I noticed that get_gpu_count() did return 4 even two gpus are fed into. I tried set _num_gpus = 2 manually but this didn't solve the problem.

BTW: Did you not divide loss by batch size for some reason on purpose? or this is just another normal way to deal with loss and batch size..

Thanks!

➜  v2 git:(devel) ✗ CUDA_VISIBLE_DEVICES=2,3 ipython training-testDM-multiGPU.py
Using /home/jiamingsun/Repositories/demon/build/multivih5datareaderop/multivih5datareaderop.so
Using /home/jiamingsun/Repositories/demon/lmbspecialops/build/lib/lmbspecialops.so
2017-10-11 17:01:19.298226: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-11 17:01:19.298265: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-11 17:01:19.298273: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-10-11 17:01:19.298281: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-11 17:01:19.298288: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-10-11 17:01:19.875586: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:83:00.0
Total memory: 11.90GiB
Free memory: 11.75GiB
2017-10-11 17:01:19.875754: W tensorflow/stream_executor/cuda/cuda_driver.cc:485] creating context when one is currently active; existing: 0x3e92220
2017-10-11 17:01:20.206920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 1 with properties:
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:84:00.0
Total memory: 11.90GiB
Free memory: 11.75GiB
2017-10-11 17:01:20.208079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 1
2017-10-11 17:01:20.208094: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y Y
2017-10-11 17:01:20.208100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 1:   Y Y
2017-10-11 17:01:20.208119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:83:00.0)
2017-10-11 17:01:20.208128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:84:00.0)
Tensor("tower_0/netFlow1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:0)
Tensor("tower_0/netDM1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:0)
[<tf.Tensor 'tower_0/gradients/tower_0/netDM1/predict_depthnormal2/split_grad/concat:0' shape=(8, 4, 48, 64) dtype=float32>, None, None]
[<tf.Tensor 'tower_0/gradients/tower_0/netDM1/split_grad/concat:0' shape=(8, 7) dtype=float32>, None, None]
Tensor("tower_1/netFlow1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:1)
Tensor("tower_1/netDM1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:1)
[<tf.Tensor 'tower_1/gradients/tower_1/netDM1/predict_depthnormal2/split_grad/concat:0' shape=(8, 4, 48, 64) dtype=float32>, None, None]
[<tf.Tensor 'tower_1/gradients/tower_1/netDM1/split_grad/concat:0' shape=(8, 7) dtype=float32>, None, None]
Tensor("tower_2/netFlow1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:2)
Tensor("tower_2/netDM1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:2)
[<tf.Tensor 'tower_2/gradients/tower_2/netDM1/predict_depthnormal2/split_grad/concat:0' shape=(8, 4, 48, 64) dtype=float32>, None, None]
[<tf.Tensor 'tower_2/gradients/tower_2/netDM1/split_grad/concat:0' shape=(8, 7) dtype=float32>, None, None]
Tensor("tower_3/netFlow1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:3)
Tensor("tower_3/netDM1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:3)
[<tf.Tensor 'tower_3/gradients/tower_3/netDM1/predict_depthnormal2/split_grad/concat:0' shape=(8, 4, 48, 64) dtype=float32>, None, None]
[<tf.Tensor 'tower_3/gradients/tower_3/netDM1/split_grad/concat:0' shape=(8, 7) dtype=float32>, None, None]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in apply_op(self, op_type_name, name, **keywords)
    490                 as_ref=input_arg.is_ref,
--> 491                 preferred_dtype=default_dtype)
    492           except TypeError as err:

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype)
    703         if ret is None:
--> 704           ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    705

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in _constant_tensor_conversion_function(v, dtype, name, as_ref)
    112   _ = as_ref
--> 113   return constant(v, dtype=dtype, name=name)
    114

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name, verify_shape)
    101   tensor_value.tensor.CopyFrom(
--> 102       tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
    103   dtype_value = attr_value_pb2.AttrValue(type=tensor_value.tensor.dtype)

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py in make_tensor_proto(values, dtype, shape, verify_shape)
    359     if values is None:
--> 360       raise ValueError("None values not supported.")
    361     # if dtype is provided, forces numpy array to be the type

ValueError: None values not supported.

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in apply_op(self, op_type_name, name, **keywords)
    504               observed = ops.internal_convert_to_tensor(
--> 505                   values, as_ref=input_arg.is_ref).dtype.name
    506             except ValueError as err:

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype)
    703         if ret is None:
--> 704           ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    705

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in _constant_tensor_conversion_function(v, dtype, name, as_ref)
    112   _ = as_ref
--> 113   return constant(v, dtype=dtype, name=name)
    114

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name, verify_shape)
    101   tensor_value.tensor.CopyFrom(
--> 102       tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
    103   dtype_value = attr_value_pb2.AttrValue(type=tensor_value.tensor.dtype)

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py in make_tensor_proto(values, dtype, shape, verify_shape)
    359     if values is None:
--> 360       raise ValueError("None values not supported.")
    361     # if dtype is provided, forces numpy array to be the type

ValueError: None values not supported.

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
/home/jiamingsun/Repositories/demon/training/v2/training-testDM-multiGPU.py in <module>()
    619
    620 if __name__ == '__main__':
--> 621     tf.app.run()
    622
    623

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py in run(main, argv)
     46   # Call the main function, passing through any arguments
     47   # to the final program.
---> 48   _sys.exit(main(_sys.argv[:1] + flags_passthrough))
     49
     50

/home/jiamingsun/Repositories/demon/training/v2/training-testDM-multiGPU.py in main(argv)
    558
    559     # combine gradients from all towers
--> 560     avg_grads = average_gradients(tower_grads)
    561
    562     optimize_op = optimizer.apply_gradients(grads_and_vars=avg_grads, global_step=trainer.global_step())

/home/jiamingsun/Repositories/demon/python/tfutils/helpers.py in average_gradients(tower_grads)
    276     for g, _ in grad_and_vars:
    277       # Add 0 dimension to the gradients to represent the tower.
--> 278       expanded_g = tf.expand_dims(g, 0)
    279
    280       # Append on a 'tower' dimension which we will average over below.

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py in expand_dims(input, axis, name, dim)
    168       raise ValueError("can't specify both 'dim' and 'axis'")
    169     axis = dim
--> 170   return gen_array_ops._expand_dims(input, axis, name)
    171 # pylint: enable=redefined-builtin,protected-access
    172

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py in _expand_dims(input, dim, name)
    898     dimension of size 1 added.
    899   """
--> 900   result = _op_def_lib.apply_op("ExpandDims", input=input, dim=dim, name=name)
    901   return result
    902

/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in apply_op(self, op_type_name, name, **keywords)
    507               raise ValueError(
    508                   "Tried to convert '%s' to a tensor and failed. Error: %s" %
--> 509                   (input_name, err))
    510             prefix = ("Input '%s' of '%s' Op has type %s that does not match" %
    511                       (input_name, op_type_name, observed))

ValueError: Tried to convert 'input' to a tensor and failed. Error: None values not supported.

When I was trying to run cmake .. to create makefile.
I ge the following error:

CMake Error at /usr/share/cmake-3.5/Modules/ExternalProject.cmake:1888 (message):
Could not extract tarball filename from url:

https://sourceforge.net/projects/half/files/latest/download?source=files

Call Stack (most recent call first):
/usr/share/cmake-3.5/Modules/ExternalProject.cmake:2459 (_ep_add_download_command)
multivih5datareaderop/CMakeLists.txt:55 (ExternalProject_Add)

-- Configuring incomplete, errors occurred!
See also "/mnt/work/Rui/demon/build/CMakeFiles/CMakeOutput.log".

Any suggestions for how to solve this?

Hello @benjaminum,

I want to train the network from scratch, is it possible to get the train code?

Thank you for making the code available! Is there some example code for training the network available?

Hi, Is it possible to train the DEMON using CPU??

If yes, what changes I need to do in the code??