sampepose / flownet2-tf Goto Github PK

View Code? Open in Web Editor NEW

407.0 16.0 197.0 16.76 MB

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

License: MIT License

Makefile 1.57% Python 50.53% Shell 0.46% C++ 47.43%

optical-flow cnn tensorflow flownet flownet2

flownet2-tf's People

Contributors

Stargazers

Watchers

Forkers

shangbuhuan13 marcylee sunshinezhe jperezrua nitinjsanket ondrejbiza donghaoye xue1liu2 vladpaunescu zachluo smallcorgi zc08 matthewd1993 lraxue swq-1993 chifang yunyouh ericsqxd strivejin wzmsltw shakeddunsky simalpha karolmajek kkk324 deshpandeshrinath bityangke soomanco coderzbx wyf-learning dachengxiaocheng liuguoyou wpfhtl jidai-code edharry trueyellow hanszone peteryej lynnlyn yiweichen04 tokb23 huangpu1 johannes-graeter einsteinliu hinpeng mehrdadshoeiby zed-wu menandro giulio-zhou geroge-gao jonganej onaggar zengzhi2015 hyang428 leeyangg yibo-chen jiyongma gongbudaizhe tangtangchx milletzcz kubafyi ml-lab hxl1990 yutinghu hss1737 deepdriving lanaelsanyoura rabitdash laoyoutiaotiao xiaojifeng huliyu1203 yonghoonkwon vitionxp linyia01 topgun666 zjsincere huipengzhang btujack pesser mobenqi biubiulsm666 nguyenhongchau 3togo nwestlake ck853178967 nomiscientist stonlimart viody maikuraky tianbartek qingsong99 fcinter loseall cxvista kevintrannz liran13 eternalsunshine1314 forestryprince xiongjiecheng bgtwoigu taowzzz

flownet2-tf's Issues

Corrupted OF output

Hello

I am using tensorflow 1.4 , I was able to build your code successfully no problem and its running but the output optical flow is completely corrupted on the sample images you provide even. Its not like what you provide with the samples at all. What could be the issue?

Thanks

undefined symbol: _ZN10tensorflow16CorrelationGrad

Env:
cuda: 9.0
cudnn: 9.0
tensorflow: 1.8.0
Ubuntu 16.04

I followed the instructions in Wxjwjj's post. Still get errors.

File "/home/charles/Libraries/anaconda2/envs/tensorflow1.2/lib/python2.7/site-packages/tensorflow/python/framework/load_library.py", line 64, in load_op_library
None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: src/ops/build/correlation.so: undefined symbol: _ZN10tensorflow16CorrelationGradAERKN5Eigen9GpuDeviceEiiiiiiiiiiiiiiiiiPKfS5_Pf

This is after commenting out -DGOOGLE_CUDA=1. With that, I get the following error during the make process:

48 errors detected in the compilation of "/tmp/tmpxft_000076b3_00000000-7_data_augmentation.cu.cpp1.ii".

How to convert the result to Gray Scale Image

Hi!
I wanna convert the result to Gray Scale Image whose number of channel is 1, rather than rgb which conains 3 channels. How can I do this?

TypeError: init() got an unexpected keyword argument 'reader_kwargs'

/home/ne5/anaconda2/envs/tensorflow/lib/python2.7/site-packages/matplotlib/colors.py:680: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
not cbook.is_string_like(colors[0]):
Traceback (most recent call last):
File "/home/ne5/anaconda2/envs/tensorflow/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/home/ne5/anaconda2/envs/tensorflow/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/ne5/test/flownet2-tf-master/src/flownet_css/train.py", line 10, in
input_a, input_b, flow = load_batch(FLYING_CHAIRS_DATASET_CONFIG, 'sample', net.global_step)
File "src/dataloader.py", line 235, in load_batch
reader_kwargs=reader_kwargs)
TypeError: init() got an unexpected keyword argument 'reader_kwargs'

Input preprocessing to use pretrained weights

I have FlowNetS running with its pretrained weights but I was wondering wheter I should normalize the input images in the range (0,1) or in the range (-1,1). As far as I know, I need to normalize the input images in the same range that it was done while training the weights provided.

Any help would be highly appreciated. Thanks in advance!

Not producing correct result

After successfully compiled the program with tensorflow 1.3.0, I tried running the sample using
python -m src.flownet2.test --input_a data/samples/0img0.ppm --input_b data/samples/0img1.ppm --out ./

However， the program is not producing expected flow result, instead, it produces something as attached, and the result is different for each run.

Has anyone encountered this? Thanks in advance!

Produced results (all running on the same sample images):

nsync_cv.h

I'm using TF 1.3.0 GPU support on Ubuntu 16.04. CUDA = 8.0, cudnn = 6.0

When I try make all finally, I have nsync_cv.h no such file error. I wonder what's causing this. I understand this repository is for TF1.2.1. But I had it working with 1.3 with the previously posted issue. I just changed my ssd and trying the same step once more. But this looks a new issue or very stupid one. I'd appreciate if you can share a bit of insight.

nvcc -g -std=c++11 -Ipython -c "import tensorflow; print(tensorflow.sysconfig.get_include())" -I"/usr/local/cuda/include" -DGOOGLE_CUDA=1 -D_MWAITXINTRIN_H_INCLUDED -D_FORCE_INLINES -D__STRICT_ANSI__ -D_GLIBCXX_USE_CXX11_ABI=0 -c src/ops/preprocessing/kernels/data_augmentation.cu.cc -x cu -Xcompiler -fPIC -o src/ops/build/data_augmentation.o
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
In file included from /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/platform/mutex.h:31:0,
from /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/framework/variant.h:31,
from /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/framework/allocator.h:26,
from /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/framework/op_kernel.h:23,
from src/ops/preprocessing/kernels/data_augmentation.h:4,
from src/ops/preprocessing/kernels/data_augmentation.cu.cc:6:
/usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/platform/default/mutex.h:25:22: fatal error: nsync_cv.h: No such file or directory
compilation terminated.
Makefile:62: recipe for target 'preprocessing' failed
make: *** [preprocessing] Error 1

from ..net import Mode ValueError: Attempted relative import in non-package

from ..net import Mode
ValueError: Attempted relative import in non-package
I can successfully run test.py code in flownet2 in terminal. But when I step through this code in pycharm, there is an error.

error when use _flow_warp_grad

Please review code about _flow_warp_grad() in src/flow_warp.py :
whether this function should be calling _flow_warp_ops.flow_warp_grad(), but not _correlation_ops.correlation_grad() ？

Estimated flow boarder is showing abnormal results

Hello,

Thanks for the code and it is helping me a lot in my project. While training flownet_s model , I am getting weird boarder with the predicted flow. I am attaching the result image down. Please let me know the reason if you have any idea.

Train only one batch?

Hi,

I got a question about the missing of for loop in the flownet_s train code. I found that input's shape is (8,320,448,3), which means only 8 images are used for training. I wonder if i've missed any detail or i need to add a for loop to fetch batch images during training?

Thanks!

Data conversion code

Hello,

First of all, thanks for your efforts. Could you please share data conversion code from original dataset to tfrecords? Thank you very much.

Haichao

test a batch

Hi,
Thanks a lot for providing the open source code!
How to test a batch?
Highly appreciate your time and help!

show the Image after processed by preprocessing.so

I want to see the image after it processed by preprocessing.so ,So I just got the preprocessing.so and compiled.

I read the image by cv2,and send it to the imageprocess function,but it doesn't work.
It successfully invokes the library file(preprocessing.so), When I want to get it by sess.run[image_as,image_bs,flows], It stopped. and gave the Error message.

Error message:
process finished with code 139 (interrupted by signal 11: SIGSEVG)

Have you ever had this problem?

No OpKernel was registered to support Op 'FlowWarp' with these attrs.

python -m src.flownet2.test --input_a data/samples/0img0.ppm --input_b data/samples/0img1.ppm --out ./
2017-09-13 13:48:11.827500: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-13 13:48:11.827522: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-13 13:48:11.827526: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-09-13 13:48:11.827529: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-13 13:48:11.827533: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/chuchienshu/tensorflowcode/newflownet/flownet2-tf/src/flownet2/test.py", line 51, in <module>
    main()
  File "/home/chuchienshu/tensorflowcode/newflownet/flownet2-tf/src/flownet2/test.py", line 18, in main
    out_path=FLAGS.out,
  File "src/net.py", line 68, in test
    saver.restore(sess, checkpoint)
  File "/home/chuchienshu/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1548, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/chuchienshu/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
    run_metadata_ptr)
  File "/home/chuchienshu/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "/home/chuchienshu/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "/home/chuchienshu/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'FlowWarp' with these attrs.  Registered devices: [CPU], Registered kernels:
  device='GPU'

	 [[Node: FlowNet2/FlowWarp = FlowWarp[](ExpandDims_1, FlowNet2/FlowNetSD/ResizeBilinear)]]

Caused by op u'FlowNet2/FlowWarp', defined at:
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/chuchienshu/tensorflowcode/newflownet/flownet2-tf/src/flownet2/test.py", line 51, in <module>
    main()
  File "/home/chuchienshu/tensorflowcode/newflownet/flownet2-tf/src/flownet2/test.py", line 18, in main
    out_path=FLAGS.out,
  File "src/net.py", line 62, in test
    predictions = self.model(inputs, training_schedule)
  File "src/flownet2/flownet2.py", line 33, in model
    flow_warp_sd = flow_warp(inputs['input_b'], net_sd_predictions['flow'])
  File "src/flow_warp.py", line 8, in flow_warp
    return _flow_warp_ops.flow_warp(image, flow)
  File "<string>", line 30, in flow_warp
  File "/home/chuchienshu/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/chuchienshu/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/chuchienshu/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): No OpKernel was registered to support Op 'FlowWarp' with these attrs.  Registered devices: [CPU], Registered kernels:
  device='GPU'

	 [[Node: FlowNet2/FlowWarp = FlowWarp[](ExpandDims_1, FlowNet2/FlowNetSD/ResizeBilinear)]]

The version of tensorflow is 1.2.0.thanks a lot！

fine-tune

Hi Sam,
Thanks a lot for providing the open-source code.
How to use your code to fine tune Sentel dataset?
Highly appreciate your time and help!

weights and loss become nan in the training

Hello sir,
First of all, thank you very much for your codes which helped me a lot to understand FlowNet2, but when I try to train your model, I found often that the weights and loss become nan. Have you ever encountered this strange problem and if so, how did you solve this problem ? Thank you very much for your attention.

Yours sincerely,
Jinglei SHI

OutOfRangeError (see above for traceback): Read less bytes than requested

when i run the python -m src.flownet2.test --input_a data/samples/0img0.ppm --input_b data/samples/0img1.ppm --out flow_generate
the problem is :
OutOfRangeError (see above for traceback): Read less bytes than requested
[[Node: save/RestoreV2_130 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_130/tensor_names, save/RestoreV2_130/shape_and_slices)]]
[[Node: save/RestoreV2_176/_37 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_434_save/RestoreV2_176", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

i don't know how to solve it?
on mint18,cuda8.0,python2.7,with gpu

Segmentation fault (core dumped)

Has anyone meet this problem before?

****@*******:~/Documents/flownet2-tf$ python -m src.flownet2.test --input_a data/samples/0img0.ppm --input_b data/samples/0img1.ppm --out ./
Segmentation fault (core dumped)

Problem with data augmentation

Hi,
Sampepose,
Thanks for your great job. While I running the code, I found that it's very difficult for the model with augmentation process to converge. The training loss and test loss is very large(about 50). And if I block out the preprocess code, the model converge very fast, the training loss is about 2. Then I realize that if I set the 'scale' option in dataset_config to be True, the tensorboard can show correct image, while if I set it to be False, the tensorboard failed to show correct image.

What's more, if I choose to only do the 'translate', 'rotate' and 'zoom' operations, no matter how the 'scale' option is, the tensorboard can show the correct image.

So, I'm wondering if there is anything about the augmentation process should I pay attention to? And would you like put your converging curve here, so can I make sure I'm doing the same training as yours.

My system is ubuntu14.04, and gpu is TitanXP, and I compile your code with gpu compatibility as sm=61.

Expecting your reply. Thanks in advance!

Is there a good method or labeling tool to create ground truth data to fine tune flownet2 on new video datasets?

Would be happy about some hint, if a good process exists.

checkpoints corrupted in download file?

FlowNet2/
FlowNet2/flownet-2.ckpt-0.index
FlowNet2/flownet-2.ckpt-0.data-00000-of-00001

tar: Unexpected EOF in archive

unmatched shapes in tf.concat()

In my distribution, the example works well:

python -m src.flownet2.test --input_a data/samples/0img0.ppm --input_b data/samples/0img1.ppm --out ./

But when I tried to use my own data, there was a unmatching problem in tf.concat() between conv5 and deconv5. I pasted the error logs below:

> $ python -m src.flownet2.test --input_a data/examples/00000.jpg --input_b data/examples/00001.jpg --out ./ 

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/workspace/playground/flownet2-tf/src/flownet2/test.py", line 78, in <module>
    main()
  File "/home/workspace/playground/flownet2-tf/src/flownet2/test.py", line 39, in main
    out_path=FLAGS.out,
  File "src/net.py", line 63, in test
    predictions = self.model(inputs, training_schedule)
  File "src/flownet2/flownet2.py", line 22, in model
    net_css_predictions = self.net_css.model(inputs, training_schedule, trainable=False)
  File "src/flownet_css/flownet_css.py", line 18, in model
    net_cs_predictions = self.net_cs.model(inputs, training_schedule, trainable=False)
  File "src/flownet_cs/flownet_cs.py", line 18, in model
    net_c_predictions = self.net_c.model(inputs, training_schedule, trainable=False)
  File "src/flownet_c/flownet_c.py", line 70, in model
    concat5 = tf.concat([conv5_1, deconv5, upsample_flow6to5], axis=3)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 1048, in concat
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 495, in _concat_v2
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2508, in create_op
    set_shapes_for_outputs(ret)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1873, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1823, in call_with_requiring
    return call_cpp_shape_fn(op, require_shape_fn=True)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
    debug_python_shape_fn, require_shape_fn)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 676, in _call_cpp_shape_fn_impl
    raise ValueError(err.message)
ValueError: Dimension 1 in both shapes must be equal, but are 15 and 16 for 'FlowNet2/FlowNetCSS/FlowNetCS/FlowNetC/concat_1' (op: 'ConcatV2') with input shapes: [1,15,27,512], [1,16,28,512], [1,16,28,2], [] and with computed input tensors: input[3] = <3>.

The shape of input images are the same, and I pasted the output of command file below:

> $ file data/examples/*  
data/examples/00000.jpg: JPEG image data, JFIF standard 1.01, resolution (DPCM), density 94x94, segment length 16, baseline, precision 8, 854x480, frames 3
data/examples/00001.jpg: JPEG image data, JFIF standard 1.01, resolution (DPCM), density 94x94, segment length 16, baseline, precision 8, 854x480, frames 3

What's more, I tried to use the same image as --input_a and --input_b, the error still occurs.

Intermedia npy file

Hi, thanks for sharing this code. Can you also share the intermedia npy file of the model weights? Since installing Caffe on my cluster is a bit tricky, I wanna have the npy to feed my TensorFlow model.

Thank you!

correlation.so : Undefined symbols (after succesfull compilation)

Hello,

My setup is python 3.5.2 , TF 1.8.0 and Cuda 9.0.
I have successfully compiled flownet in order to use the correlation layer.
However, when loading the correlation.so, I get the following error :

from object_detection.utils.correlation import correlation
ImportError: /data/tensorflow/models/research/object_detection/utils/correlation.so: undefined symbol: _ZTIN10tensorflow8OpKernelE

After looking this up online, i found more than a couple of people suggesting this fix :
https://www.tensorflow.org/versions/master/extend/adding_an_op#compile_the_op_using_your_system_compiler_tensorflow_binary_installation

( adding -ltensorflow_framework to the CGPU Flags)

I have successfully compiled (make all) flownet with the added flag, but i still get the same error. This is the output of my compilation process (only included last several lines, but no errors occur, as you can see, and the flag is in place) :

g++ -g -std=c++11 -Ipython -c "import tensorflow; print(tensorflow.sysconfig.get_include())" -I"/usr/local/cuda/include" -DGOOGLE_CUDA=1 -D_MWAITXINTRIN_H_INCLUDED -D_FORCE_INLINES -D__STRICT_ANSI__ -D_GLIBCXX_USE_CXX11_ABI=0 "src/ops/downsample/downsample_kernel.cc" "src/ops/downsample/downsample_op.cc" src/ops/build/downsample_kernel_gpu.o -pthread -shared -fPIC -L/usr/local/cuda/lib -L/usr/local/cuda/lib64 -lcudart -L/home/andi/miniconda3/envs/tf18/lib/python3.5/site-packages/tensorflow -ltensorflow_framework -o src/ops/build/downsample.so
/home/andi/miniconda3/envs/tf18/lib/python3.5/site-packages/h5py/init.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from .conv import register_converters as register_converters
GLib-GIO-Message: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications.
nvcc -g -std=c++11 -Ipython -c "import tensorflow; print(tensorflow.sysconfig.get_include())" -I"/usr/local/cuda/include" -DGOOGLE_CUDA=1 -D_MWAITXINTRIN_H_INCLUDED -D_FORCE_INLINES -D__STRICT_ANSI -D_GLIBCXX_USE_CXX11_ABI=0 -c src/ops/correlation/correlation_kernel.cu.cc -x cu -Xcompiler -fPIC --expt-relaxed-constexpr -o src/ops/build/correlation_kernel_gpu.o
/home/andi/miniconda3/envs/tf18/lib/python3.5/site-packages/h5py/init.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
GLib-GIO-Message: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications.
/home/andi/miniconda3/envs/tf18/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(57): warning: integer conversion resulted in a change of sign

/home/andi/miniconda3/envs/tf18/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(304): warning: integer conversion resulted in a change of sign