dyelax / adversarial_video_generation Goto Github PK

When I download the model in https://drive.google.com/file/d/0Byf787GZQ7KvR2JvMUNIZnFlbm8/view
and run
python avg_runner.py -l ./Models/Adversarial/model.ckpt-500000
The program can't run

Caused by op u'save/RestoreV2_51', defined at:
File "avg_runner.py", line 185, in
main()
File "avg_runner.py", line 177, in main
runner = AVGRunner(num_steps, load_path, num_test_rec)
File "avg_runner.py", line 53, in init
self.saver = tf.train.Saver(keep_checkpoint_every_n_hours=2)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1040, in init
self.build()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1070, in build
restore_sequentially=self._restore_sequentially)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 675, in build
restore_sequentially, reshape)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 402, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 242, in restore_op
[spec.tensor.dtype])[0])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 668, in restore_v2
dtypes=dtypes, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2327, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1226, in init
self._traceback = _extract_stack()

NotFoundError (see above for traceback): Tensor name "generator/scale_0/setup/Variable_1/optimizer" not found in checkpoint files ../Models/Adversarial/model.ckpt-500000
[[Node: save/RestoreV2_51 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_51/tensor_names, save/RestoreV2_51/shape_and_slices)]]
[[Node: save/RestoreV2/_343 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_851_save/RestoreV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Hi,

Thanks for providing such a well-written and nicely commented code. We are trying to run your code on UCF101 and Sports1m dataset as the authors of the original paper report results on these datasets.
However, when we are training with: LRATE_G = 0.00004 and LRATE_D = 0.02, we are getting "nans" for Global Loss, PSNR Error and Sharpdiff Error in g_model.py

Is there any logical way to adjust the learning rate of Generator and Discriminator to prevent the above issue?

Hi Matt,
Thanks for the great code. According to the paper, the input to the discriminator is a sequence of frames (history of frames, and the next frame). If I understand your code correctly, the input to the discriminator is a single frame, which is only the next frame (either generated or from ground-truth). Is this right? If yes, wouldn't this prevent the discriminator to make use of the continuity in the video?
Thanks in advance.

I get the error:
'module' object has no attribute 'SummaryWriter'
which I found out is due to a deprecation in the code.
I recommend changing these lines or updating the FAQ to suggest a specific version of TensorFlow.

Thanks for your work! Hopefully I can get it up and running.

Traceback (most recent call last):
File "avg_runner.py", line 185, in
main()
File "avg_runner.py", line 177, in main
runner = AVGRunner(num_steps, load_path, num_test_rec)
File "avg_runner.py", line 40, in init
c.SCALE_FC_LAYER_SIZES_D)
File "C:\Users\tao-he\PycharmProjects\testing-pytorch\combustion\Adversarial_Video_Generation-master\Code\d_model.py", line 44, in init
self.define_graph()
File "C:\Users\tao-he\PycharmProjects\testing-pytorch\combustion\Adversarial_Video_Generation-master\Code\d_model.py", line 65, in define_graph
self.scale_fc_layer_sizes[scale_num]))
File "C:\Users\tao-he\PycharmProjects\testing-pytorch\combustion\Adversarial_Video_Generation-master\Code\d_scale_model.py", line 43, in init
self.define_graph()
File "C:\Users\tao-he\PycharmProjects\testing-pytorch\combustion\Adversarial_Video_Generation-master\Code\d_scale_model.py", line 76, in define_graph
self.conv_layer_fms[i + 1]]))
File "C:\Users\tao-he\PycharmProjects\testing-pytorch\combustion\Adversarial_Video_Generation-master\Code\tfutils.py", line 10, in w
return tf.int32(Variable(tf.random.truncated_normal(shape, stddev=stddev)))
NameError: name 'Variable' is not defined

C:\Users\tao-he\PycharmProjects\testing-pytorch\combustion\Adversarial_Video_Generation-master\Code>python avg_runner.py --test_dir=F:/zfg/Input/test/ --
name=F:/zfg/output/ --stats_freq=100 --summary_freq=100 --img_save_freq=100 --test_freq=2 --model_save_freq=100
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a
synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a
synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a
synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a
synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a
synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a
synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:541: FutureWarning: Passing (type, 1) or '1typ
e' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:542: FutureWarning: Passing (type, 1) or '1typ
e' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:543: FutureWarning: Passing (type, 1) or '1typ
e' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:544: FutureWarning: Passing (type, 1) or '1typ
e' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:545: FutureWarning: Passing (type, 1) or '1typ
e' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:550: FutureWarning: Passing (type, 1) or '1typ
e' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
2019-11-19 11:50:22.962937: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compi
led to use: AVX2
Init discriminator...
Traceback (most recent call last):
File "avg_runner.py", line 185, in
main()
File "avg_runner.py", line 177, in main
runner = AVGRunner(num_steps, load_path, num_test_rec)
File "avg_runner.py", line 40, in init
c.SCALE_FC_LAYER_SIZES_D)
File "C:\Users\tao-he\PycharmProjects\testing-pytorch\combustion\Adversarial_Video_Generation-master\Code\d_model.py", line 44, in init
self.define_graph()
File "C:\Users\tao-he\PycharmProjects\testing-pytorch\combustion\Adversarial_Video_Generation-master\Code\d_model.py", line 65, in define_graph
self.scale_fc_layer_sizes[scale_num]))
File "C:\Users\tao-he\PycharmProjects\testing-pytorch\combustion\Adversarial_Video_Generation-master\Code\d_scale_model.py", line 43, in init
self.define_graph()
File "C:\Users\tao-he\PycharmProjects\testing-pytorch\combustion\Adversarial_Video_Generation-master\Code\d_scale_model.py", line 95, in define_graph
self.fc_layer_sizes[i + 1]]))
File "C:\Users\tao-he\PycharmProjects\testing-pytorch\combustion\Adversarial_Video_Generation-master\Code\tfutils.py", line 10, in w
return tf.Variable(tf.random.truncated_normal(shape, stddev=stddev))
File "C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\random_ops.py", line 178, in truncated_normal
shape_tensor, dtype, seed=seed1, seed2=seed2)
File "C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\gen_random_ops.py", line 1063, in truncated_normal
name=name)
File "C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\op_def_library.py", line 626, in _apply_op_helper
param_name=input_name)
File "C:\Users\tao-he\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\op_def_library.py", line 60, in _SatisfiesTypeConstrain
t
", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
TypeError: Value passed to parameter 'shape' has DataType float32 not in list of allowed values: int32, int64

hello, thanks for sharing the repo for us,but when i have finished the 1-3 steps and begin to start train on my own datasets,TypeError: Value passed to parameter 'shape' has DataType float32 not in list of allowed values: int32, int64 is coming, even i changed all the dtype into int,the type error still exited. I will appreciate it very much if you can help me with the error.

When running the code I get multiple errors due to the use of deprecated functions.

The function tf.train.SummaryWriter is deprecated, and current installations of Tensorflow don't support it anymore. It should be renamed to tf.summary.FileWriter.

Same thing for:

• tf.pack, which should be renamed to tf.stack.
• tf.scalar_summary, which should be renamed to tf.summary.scalar.
• tf.merge_summary, which should be renamed to tf.summary.merge.

source: https://www.tensorflow.org/install/migration

Also, the order of the arguments of tf.concat is now swapped, so the call in file g_model.py, line 117
inputs = tf.concat(3, [inputs, last_gen_frames])
should be changed to
inputs = tf.concat([inputs, last_gen_frames], 3)

source: tensorflow/tensorflow#7031

If you want I can make a pull request with the changes above.

Hi,

Having read your code, I have some questions about your gdl_loss. In your implementation, it seems that you lost an inner abs ops which is different from the original equation. However, I have seen your result that performs well and I am just curious about that element's influence.

Thanks,

Hi dyelax,

I really appreciate for your work.
I tried to modified your code to Wgan and trained on AWS. But it seems like training pretty slow, it only trained 4000 iterations a day. Also it has high GPU Memory-Usage but zero volatile cpu-util. Do you have any idea why does this happening?

(tensorflow) ubuntu@ip-172-31-8-95:~$ nvidia-smi
Sun Apr 23 19:53:29 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.51 Driver Version: 375.51 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 Off | 0000:00:1E.0 Off | 0 |
| N/A 40C P0 71W / 149W | 10941MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 20379 C python 10937MiB |
+-----------------------------------------------------------------------------+

Thank you!

Hi Matt,

Thanks for the elegant code in Tensorflow. But why is the data processing step necessary?

It seems to me that it's possible to load the dataset into memory before training (at least for the PacMan dataset), and then randomly select 32*32 patches at runtime. Will that make I/O faster?

Thanks in advance

Losses are not normalized over the batch. Though it doesn't affect the results, if someone wants to experiment with different parameters, be aware that increasing batch size will increase loss as well. I also think normalization is necessary for a correct interpretation of the loss values.

Traceback (most recent call last):
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1628, in _create_c_op
c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 3 in both shapes must be equal, but are 12 and 3. Shapes are [?,8,8,12] and [?,8,8,3].
From merging shape 0 with other shapes. for 'generator/scale_1/calculation/concat/concat_dim' (op: 'Pack') with input shapes: [?,8,8,12], [?,8,8,3].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "avg_runner.py", line 187, in
main()
File "avg_runner.py", line 179, in main
runner = AVGRunner(num_steps, load_path, num_test_rec)
File "avg_runner.py", line 50, in init
c.SCALE_KERNEL_SIZES_G)
File "/home/guohaoyu/vedio_generation/Adversarial_Video_Generation/Code/g_model.py", line 48, in init
self.define_graph()
File "/home/guohaoyu/vedio_generation/Adversarial_Video_Generation/Code/g_model.py", line 159, in define_graph
last_scale_pred_train)
File "/home/guohaoyu/vedio_generation/Adversarial_Video_Generation/Code/g_model.py", line 123, in calculate
inputs = tf.concat(3, [inputs, last_gen_frames])
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1121, in concat
dtype=dtypes.int32).get_shape().assert_is_compatible_with(
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1050, in convert_to_tensor
as_ref=False)
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1146, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 971, in _autopacking_conversion_function
return _autopacking_helper(v, dtype, name or "packed")
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 923, in _autopacking_helper
return gen_array_ops.pack(elems_as_tensors, name=scope)
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 4875, in pack
"Pack", values=values, axis=axis, name=name)
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1792, in init
control_input_ops)
File "/home/guohaoyu/.conda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1631, in _create_c_op
raise ValueError(str(e))
ValueError: Dimension 3 in both shapes must be equal, but are 12 and 3. Shapes are [?,8,8,12] and [?,8,8,3].
From merging shape 0 with other shapes. for 'generator/scale_1/calculation/concat/concat_dim' (op: 'Pack') with input shapes: [?,8,8,12], [?,8,8,3].

Hello:
If I want to change the number of the input frames and the output frames. for example,
given 10 input frames and predict the next 3 frames. then, how to decide the matrix of SCALE_CONV_FMS, SCALE_KERNEL_SIZES and SCALE_FC_LAYER_SIZES.

Thank you!

I'm trying to use your code but I get a strange error with the avg_runner.
it seems like it has some issue with the size of some images.
this is however very unlickly as I exported all the images from a video with ffmpeg.

any idea about why it breaks?

gino:Code Lorenzo$ python2.7 avg_runner.py -n Test -s 1000 --model_save_freq=1000 --test_freq=1000
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
Init discriminator...
Init generator...
Init variables...
Training discriminator...
Training generator...
Training discriminator...
Training generator...
Training discriminator...
Training generator...
Training discriminator...
Training generator...
Training discriminator...
Training generator...
Training discriminator...
Training generator...
Training discriminator...
Training generator...
Training discriminator...
Training generator...
Training discriminator...
Training generator...
Training discriminator...
DiscriminatorModel: step 10 | global loss: 0.603209
Training generator...
GeneratorModel : Step  10
                 Global Loss    :  540.297
                 PSNR Error     :  5.45025
                 Sharpdiff Error:  4.7847
Training discriminator...
Training generator...
Training discriminator...
Training generator...
Training discriminator...
Traceback (most recent call last):
  File "avg_runner.py", line 185, in <module>
    main()
  File "avg_runner.py", line 181, in main
    runner.train()
  File "avg_runner.py", line 70, in train
    self.d_model.train_step(batch, self.g_model)
  File "/Users/Lorenzo/development/1-frame-in-the-future/Code/d_model.py", line 171, in train_step
    feed_dict = self.build_feed_dict(input_frames, gt_output_frames, generator)
  File "/Users/Lorenzo/development/1-frame-in-the-future/Code/d_model.py", line 132, in build_feed_dict
    resized_frame = resize(sknorm_img, [scale_net.height, scale_net.width, 3])
  File "/usr/local/lib/python2.7/site-packages/skimage/transform/_warps.py", line 119, in resize
    preserve_range=preserve_range)
  File "/usr/local/lib/python2.7/site-packages/skimage/transform/_geometric.py", line 1296, in warp
    image = _convert_warp_input(image, preserve_range)
  File "/usr/local/lib/python2.7/site-packages/skimage/transform/_geometric.py", line 1108, in _convert_warp_input
    image = img_as_float(image)
  File "/usr/local/lib/python2.7/site-packages/skimage/util/dtype.py", line 301, in img_as_float
    return convert(image, np.float64, force_copy)
  File "/usr/local/lib/python2.7/site-packages/skimage/util/dtype.py", line 205, in convert
    raise ValueError("Images of type float must be between -1 and 1.")

Hello,I'm trying to use your trained model to predict one frame on your dataset,however I encountered this problem.
NotFoundError (see above for traceback): Tensor name "generator/scale_3/setup/Variable_5/optimizer" not found in checkpoint files ../Models/Adversarial/model.ckpt-500000 [[Node: save/RestoreV2_153 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_153/tensor_names, save/RestoreV2_153/shape_and_slices)]] [[Node: save/RestoreV2_63/_147 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_651_save/RestoreV2_63", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

hi,dyelax
I found that in the original paper, each scale network is to learn Yk - uk(Yk-1). But in your implementation, it is to learn Yk. Did I misunderstand ?

Hello,

I have successfully completed your first three steps and am stuck on step 4 where I can train/test my own data. The data I am producing is not a .cpkt format and is a .npz format. What is the best way to take my .npz data and convert it into a .cpkt format? Is there something I am not doing?

Thank you,
Hao

Thanks for your sharing, I run the coda36 with my own dataset, the size of my data is (210,160,1), and i have changed the channel 3 to 1, but some errors below:

Traceback (most recent call last):
File "/home/dell/sly/reaver-master/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1576, in _create_c_op
c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimensions must be equal, but are 1 and 3 for 'generator/train/Conv2D' (op: 'Conv2D') with input shapes: [?,8,8,1], [1,2,3,3].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/dell/sly/AdversarialVideoGeneration-master/Code36/avg_runner.py", line 185, in
main()
File "/home/dell/sly/AdversarialVideoGeneration-master/Code36/avg_runner.py", line 177, in main
runner = AVGRunner(num_steps, load_path, num_test_rec)
File "/home/dell/sly/AdversarialVideoGeneration-master/Code36/avg_runner.py", line 50, in init
c.SCALE_KERNEL_SIZES_G)
File "/home/dell/sly/AdversarialVideoGeneration-master/Code36/g_model.py", line 48, in init
self.define_graph()
File "/home/dell/sly/AdversarialVideoGeneration-master/Code36/g_model.py", line 192, in define_graph
self.d_scale_preds)
File "/home/dell/sly/AdversarialVideoGeneration-master/Code36/loss_functions.py", line 27, in combined_loss
loss += lam_gdl * gdl_loss(gen_frames, gt_frames, alpha)
File "/home/dell/sly/AdversarialVideoGeneration-master/Code36/loss_functions.py", line 88, in gdl_loss
gen_dx = tf.abs(tf.nn.conv2d(gen_frames[i], filter_x, strides, padding=padding))
File "/home/dell/sly/reaver-master/venv/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 956, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "/home/dell/sly/reaver-master/venv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/dell/sly/reaver-master/venv/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/home/dell/sly/reaver-master/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/home/dell/sly/reaver-master/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1731, in init
control_input_ops)
File "/home/dell/sly/reaver-master/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1579, in _create_c_op
raise ValueError(str(e))
ValueError: Dimensions must be equal, but are 1 and 3 for 'generator/train/Conv2D' (op: 'Conv2D') with input shapes: [?,8,8,1], [1,2,3,3].

Process finished with exit code 1

I don't know whether i need to change other places??

Hi,

I've cloned the repo and grabbed your trained models in an attempt to quickly see the demo running on my computer, but I'm getting an error and I'm not 100% sure I've understood how to place the data correctly:

get_train_batch c.TRAIN_DIR_CLIPS ../Data/.Clips/ c.NUM_CLIPS 0
Traceback (most recent call last):
  File "avg_runner.py", line 186, in <module>
    main()
  File "avg_runner.py", line 182, in main
    runner.train()
  File "avg_runner.py", line 68, in train
    batch = get_train_batch()
  File "~/Adversarial_Video_Generation/Code/utils.py", line 127, in get_train_batch
    path = c.TRAIN_DIR_CLIPS + str(np.random.choice(c.NUM_CLIPS)) + '.npz'
  File "mtrand.pyx", line 1391, in mtrand.RandomState.choice (numpy/random/mtrand/mtrand.c:15381)
ValueError: a must be greater than 0

I'm running the script by first cding into Code then running python avg_runner.py -l ../Models/Adversarial/model.ckpt-500000. I've added a print statement before the error line to see what the variables hold and it looks like the .Clips folder is empty:
get_train_batch c.TRAIN_DIR_CLIPS ../Data/.Clips/ c.NUM_CLIPS 0

I've double checked and that seems to be the case:

> file ../Data/.Clips/
../Data/.Clips/: directory
> ls ../Data/.Clips/ | wc -w
0

I feel I'm missing something: should I have downloaded the contents of the .Clips folder (if so from where ?) or should the .Clips contents be generated ?

How can I double check and make sure I'm using the examples correctly ?

I am using tensorflow version '0.12.0' with gpu support in a virtual environment on OSX 10.11.5 with an nVidia GeForce GT 750M (2GB VRAM), CUDA 8.0 and CuDNN 5.1 installed.

The first 3 levels of the repo look like this:

├── Code
│   ├── avg_runner.py
│   ├── constants.py
│   ├── constants.pyc
│   ├── d_model.py
│   ├── d_model.pyc
│   ├── d_scale_model.py
│   ├── d_scale_model.pyc
│   ├── g_model.py
│   ├── g_model.pyc
│   ├── loss_functions.py
│   ├── loss_functions.pyc
│   ├── loss_functions_test.py
│   ├── process_data.py
│   ├── tfutils.py
│   ├── tfutils.pyc
│   ├── tfutils_test.py
│   ├── utils.py
│   └── utils.pyc
├── Data
│   └── Ms_Pacman
│       ├── Test
│       └── Train
├── DataOld
│   └── Ms_Pacman
│       ├── Test
│       └── Train
├── LICENSE
├── Models
│   ├── Adversarial
│   │   ├── checkpoint
│   │   ├── model.ckpt-500000
│   │   └── model.ckpt-500000.meta
│   └── NonAdversarial
│       ├── checkpoint
│       ├── model.ckpt-1020000
│       └── model.ckpt-1020000.meta
├── Models.zip
├── Ms_Pacman.zip
├── README.md
├── Results
│   ├── Gifs
│   │   ├── 4_Comparison.gif
│   │   ├── 5_Comparison.gif
│   │   └── rainbow_NonAdv.gif
│   └── Summaries
│       ├── Adv-1
│       └── NonAdv-1
├── Save
│   ├── Images
│   │   └── Default
│   ├── Models
│   │   └── Default
│   └── Summaries
│       └── Default
└── deep_multi-scale_video_prediction_beyond_mean_square_error.pdf

Full output:

python avg_runner.py -l ../Models/Adversarial/model.ckpt-500000
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.dylib locally
c.TEST_DIR ../Data/Ms_Pacman/Test/
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:901] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GeForce GT 750M
major: 3 minor: 0 memoryClockRate (GHz) 0.9255
pciBusID 0000:01:00.0
Total memory: 2.00GiB
Free memory: 1.21GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 750M, pci bus id: 0000:01:00.0)
WARNING:tensorflow:From avg_runner.py:30 in __init__.: __init__ (from tensorflow.python.training.summary_io) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.FileWriter. The interface and behavior is the same; this is just a rename.
Init discriminator...
WARNING:tensorflow:From ~/Adversarial_Video_Generation/Code/d_model.py:92 in define_graph.: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
WARNING:tensorflow:From ~/Adversarial_Video_Generation/Code/d_model.py:93 in define_graph.: merge_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.merge.
Init generator...
WARNING:tensorflow:From ~/Adversarial_Video_Generation/Code/g_model.py:199 in define_graph.: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
WARNING:tensorflow:From ~/Adversarial_Video_Generation/Code/g_model.py:219 in define_graph.: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
WARNING:tensorflow:From ~/Adversarial_Video_Generation/Code/g_model.py:221 in define_graph.: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
WARNING:tensorflow:From ~/Adversarial_Video_Generation/Code/g_model.py:226 in define_graph.: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
WARNING:tensorflow:From ~/Adversarial_Video_Generation/Code/g_model.py:228 in define_graph.: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
WARNING:tensorflow:From ~/Adversarial_Video_Generation/Code/g_model.py:232 in define_graph.: merge_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.merge.
WARNING:tensorflow:From ~/Adversarial_Video_Generation/Code/g_model.py:233 in define_graph.: merge_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.merge.
Init variables...
Model restored from ../Models/Adversarial/model.ckpt-500000
get_train_batch c.TRAIN_DIR_CLIPS ../Data/.Clips/ c.NUM_CLIPS 0
Traceback (most recent call last):
  File "avg_runner.py", line 186, in <module>
    main()
  File "avg_runner.py", line 182, in main
    runner.train()
  File "avg_runner.py", line 68, in train
    batch = get_train_batch()
  File "~/Adversarial_Video_Generation/Code/utils.py", line 127, in get_train_batch
    path = c.TRAIN_DIR_CLIPS + str(np.random.choice(c.NUM_CLIPS)) + '.npz'
  File "mtrand.pyx", line 1391, in mtrand.RandomState.choice (numpy/random/mtrand/mtrand.c:15381)
ValueError: a must be greater than 0

I appreciate any tips or advice you can share.

Thank you,
George

Hi Matt,

First, thanks for sharing your nice code which helped me a lot for figuring out details of the paper.
I came up with a question regarding the normalization step.
What is the reason for making images to be in the range [0,1] in line 128~133 of 'd_model.py'? Was it due to the undesired interpolation effect occurring during running skimage.transform.resize? I tried by myself with two options: with and without taking your normalization step before resizing; it seems like there is difference.

One more thing, when running your code with UCF101 dataset, there is an error "Images of type float must be between -1 and 1." I checked with the debugging mode, and found that sometimes skimage.transform.resize results violate [-1, 1] range. For a case you need a quick modification, I guess following code will help you some,

sknorm_img = np.minimum(sknorm_img, 1)  
sknorm_img = np.maximum(sknorm_img, -1)

Best,
Da-Hye

I think I've found another bug: In the BCE loss you calculate -sum targets · log(preds) + (1 - targets) · log(1 - preds), whereas in the paper it is defined like this:

Following the other notation in the paper, Ŷ appears to be the prediction and Y is the target, which also matches the real interval as domain for Ŷ. So it is the other way around in the paper. Is the BCE symmetric?

Hi, dyelax!
Thanks for sharing the awesome code!!! It helps me a lot, but I don't know whether the code is using gpu. I don't find any sentence of configuring gpu. When I run the code, I found it didn't use gpu to compute, I don't know why.
Please help me! Thank you very much!

Can you share the code that generates the gif file?

Hey all! There is a glaring bug with the data_proccess pipeline.

Currently, for each clip, the video generator sorts the data by filename
ep_frame_paths = (glob(os.path.join(ep_dir, '*'))). If your files are written as

filename1
filename2
...
filenameN

and N > 10 this will interlace earlier frames of the data into the clips used for training. With a large history window this adds some robustness to the model, but it really invalidates the modeling assumption!

Sortiing by the last integer fixes this (change line 80 to this)

ep_frame_paths = sorted(glob(os.path.join(ep_dir, '*')), key=lambda x: int(x.split("frame")[-1].split(".png")[0]))

As I can see here:

@dyelax

Hello, thanks for sharing your wonderful project
However, it seems that I have to downgrade all python, tf, packages,.. in order to use this model.
Which might lead to many issues. It would be kind of you if you could show any update code for this

TypeError: Value passed to parameter 'shape' has DataType float32 not in list of allowed values: int32, int64

I know PSNR is Peak signal to Noise Ratio and higher PSNR value implies better the image generated. Is PSNR error and PSNR same? Or am I missing something?

Traceback (most recent call last):
File "avg_runner.py", line 185, in
main()
File "avg_runner.py", line 181, in main
runner.train()
File "avg_runner.py", line 68, in train
batch = get_train_batch()
File "/Users/shreyans/Desktop/Hamburg_Assignments/Neural networks/version 2/Adversarial_Video_Generation/Code/utils.py", line 126, in get_train_batch
path = c.TRAIN_DIR_CLIPS + str(np.random.choice(c.NUM_CLIPS)) + '.npz'
File "mtrand.pyx", line 1120, in mtrand.RandomState.choice
ValueError: a must be greater than 0

Hi Matt,

I am trying to reproduce your results on Pacman dataset. When running the pre-process step, I am getting the following warnings: "Overflow encountered in square", "Invalid value encountered in subtract" in line 53 of utils.py ( diff += np.sum(np.square(next_frame - frame))). I am working with the master branch. When I tried: print(np.amax(frame) and print(np.amax(next_frame)) in clip_l2_diff function, the values are as high as 9.91442958312e+230. In addition, there are some nans too.

Is there an issue with the Pacman dataset itself?

Hi,

I was trying out the code, and obtained a scikit value error: float should be between -1 and 1. I found out in /utils.py of function get_train_batch(), Line 126, the clip is defined to be float32, but it apparently causes the range error, and changed it to float64. It works now. Is it due to some version of dependency or scikit, etc?

Thanks,

Vishal

This is not really an issue, but might be helpful for you:
While checking out your (awesome) code, I just discovered this TODO:

# TODO: Add SSIM error http://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf

I was looking for an SSIM/MS-SSIM implementation in TensorFlow for a while. Because I could not discovery anything, I ported some Matlab/C++ code to TensorFlow. I've posted the code on StackOverflow, so feel free to use and improve it:

Stack Overflow: SSIM / MS-SSIM for TensorFlow

Hi, I tried running avg_runner.py and I'm getting quite a few function deprecation errors. For example, the summary writer function here is getting an error:

File "avg_runner.py", line 191, in <module>
main()

File "avg_runner.py", line 183, in main
runner = AVGRunner(num_steps, load_path, num_test_rec)
File "avg_runner.py", line 32, in init
self.summary_writer = tf.train.SummaryWriter(c.SUMMARY_SAVE_DIR, graph=self.sess.graph)
AttributeError: 'module' object has no attribute 'SummaryWriter'

I'm using version 1.13.1 of tensorflow, but I noticed the code is only updated for version 0.12. I tried to install version 0.12 of tensorflow but it looks like the older versions are not available anymore. Are there any updated versions of this code using a newer version of tensorflow?

First of all, thanks for sharing this fantastic and clean code! I'm having trouble understanding this part of your code: Here you run the discriminator on the predicted frames to get the real/fake predictions per frame. You then pass these via the d_scale_preds placeholders back to the generator and finally you regress the generator to bring d_scale_preds closer to a tensor of ones. What I am wondering is how the gradients are backpropagated from the discriminator back to the generator. Can the gradients pass through sess.run statements?