tudelft / ssl_e2vid Goto Github PK

License: MIT License

Python 100.00%

ssl_e2vid's Introduction

Back to Event Basics: SSL of Image Reconstruction for Event Cameras

Minimal code for Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy, CVPR'21.

Usage

This project uses Python >= 3.7.3. After setting up your virtual environment, please install the required python libraries through:

pip install -r requirements.txt

Code is formatted with Black (PEP8) using a pre-commit hook. To configure it, run:

pre-commit install

Data format

Similarly to researchers from Monash University, this project processes events through the HDF5 data format. Details about the structure of these files can be found in datasets/tools/.

Inference

Download our pre-trained models from here.

Our HDF5 version of sequences from the Event Camera Dataset can also be downloaded from here for evaluation purposes.

To estimate optical flow from the input events:

python eval_flow.py <path_to_model_dir>

To perform image reconstruction from the input events:

python eval_reconstruction.py <path_to_model_dir>

In configs/, you can find the configuration files associated to these scripts and vary the inference settings (e.g., number of input events, dataset).

Training

Our framework can be trained using any event camera dataset. However, if you are interested in using our training data, you can download it from here. The datasets are expected at datasets/data/, but this location can be modified in the configuration files.

To train an image reconstruction and optical flow model, you need to adapt the training settings in configs/train_reconstruction.yml. Here, you can choose the training dataset, the number of input events, the neural networks to be used (EV-FlowNet or FireFlowNet for optical flow; E2VID or FireNet for image reconstruction), the number of epochs, the optimizer and learning rate, etc. To start the training from scratch, run:

python train_reconstruction.py

Alternatively, if you have a model that you would like to keep training from, you can use

python train_reconstruction.py --prev_model <path_to_model_dir>

This is handy if, for instance, you just want to train the image reconstruction model and use a pre-trained optical flow network. For this, you can set train_flow: False in configs/train_reconstruction.yml, and run:

python train_reconstruction.py --prev_model <path_to_optical_flow_model_dir>

If you just want to train an optical flow network, adapt configs/train_flow.yml, and run:

python train_flow.py

Note that we use MLflow to keep track of all the experiments.

Citations

If you use this library in an academic context, please cite the following:

@article{paredes2020back,
  title={Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy},
  author={Paredes-Vall{\'e}s, Federico and de Croon, Guido C. H. E.},
  journal={arXiv preprint arXiv:2009.08283},
  year={2020}
}

Acknowledgements

This code borrows from the following open source projects, whom we would like to thank:

ssl_e2vid's People

Contributors

Stargazers

Watchers

Forkers

evilperfectionist adarshkosta tlwzzy anywayany jamisonlu mrfisher97 mountains-high mapolinario94 bjpxyzt ahupujr

ssl_e2vid's Issues

Flow Evaluation on MVSEC

I was trying to evaluate the provided EV-FlowNet and FireFlowNet models for estimating optical flow on the MVSEC dataset and had some issues replicating the AEE results.

The provided models seem to perform well, visually on the ECD dataset, but there is no error metric to quantify the predictions due to lack of ground truth data.

The same models seem to perform not as well on MVSEC as on ECD and I could not find any codes which compute AEE for evaluation on MVSEC. According to Section 4.1 in the paper, the flow predictions were generated at each grayscale frame timestamp and scaled for the time duration between consecutive grayscale frames during evaluation. As per the binning and event window scheme, configs/train_flow.yaml uses window: 5000 and num_bins: 5. How are these changed when evaluating on MVSEC between grayscale frames? I understand that the window length can be changed to span over the duration between consectutive gray frames but the num_bins cannot be changed as it represents the number of input channels. Correct me if I am missing something.

It would be great if you could provide a bit more explanation on this and better yet update the repository with the evaluation code.

Thanks in advance!

Error reported on GPU, not on CPU

/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [0,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [1,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [2,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [3,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [4,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [5,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [6,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [7,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [8,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [9,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [10,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [11,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [12,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [13,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [14,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [15,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [16,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [17,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [18,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [19,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [20,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [21,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [22,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [23,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [24,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [25,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [26,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [27,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [28,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [29,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [30,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [31,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [0,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [1,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [2,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [3,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [4,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [5,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [6,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [7,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [8,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [9,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [10,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [11,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [12,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [13,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [14,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [15,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [16,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [17,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [18,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [19,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [20,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [21,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [22,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [23,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [24,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [25,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [26,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [27,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [28,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [29,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [30,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [31,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
Traceback (most recent call last):
File "/media/yons/F/code/ssl_e2vid-main/train_flow.py", line 157, in
train(args, YAMLParser(args.config))
File "/media/yons/F/code/ssl_e2vid-main/train_flow.py", line 104, in train
loss = loss_function(x["flow"], inputs["inp_list"].to(device), inputs["inp_pol_mask"].to(device))
File "/home/yons/anaconda3/envs/DCEIFlow/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/media/yons/F/code/ssl_e2vid-main/loss/flow.py", line 65, in forward
fw_iwe_neg = interpolate(fw_idx.long(), fw_weights, self.res, polarity_mask=pol_mask[:, :, 1:2])
File "/media/yons/F/code/ssl_e2vid-main/utils/iwe.py", line 87, in interpolate
iwe = torch.zeros((idx.shape[0], res[0] * res[1], 1)).to(idx.device)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

When using CPU instead of GPU, the error doesn't occur.

Question on the post-processing steps of reconstruction

Hi @fedepare, and thanks for this great work.

I have a question about the post-processing steps performed after the image reconstruction network. In the paper, it is stated that the output of the reconstruction network is first passed through an exponential function, and then a min/max normalization is applied. The relevant section from the paper can be seen in below image:

When I checked the codes in this repo, I saw that you applied the min/max normalization (here and here), without using an exponential function.

Is this a mistake in the code repo? I'm guessing that you were also using the exponential function while performing experiments in 4.2. Reconstruction Evaluation section of your paper. Is this correct?

Thanks again for sharing this amazing work!

Questions about the data augmentation

Hi, thanks for the amazing work.

But I have a question about the dataset. The events loaded uses data augmentations such as horizon flip, and random crop to 128x128 when training. But the codes only contain flip, no crop. When training you use 128x128 according to the config files. It is a error If just read the events without crop.

ssl_e2vid/dataloader/h5.py

Line 216 in 84f7c7e

xs, ys, ps = self.augment_events(xs, ys, ps, batch)

Question about the loss computing

In the code loss/flow.py
line 35
“param event_list: [batch_size x N x 4] input events (y, x, ts, p) ”
the input seems to be (ts,y, x, p) rather than (y, x, ts, p)

line 44-46
“ flow_idx = event_list[:, :, 1:3].clone()
flow_idx[:, :, 0] = self.res[1] # torch.view is row-major
flow_idx = torch.sum(flow_idx, dim=2)
”
does it mean flow_idx = 240y+x ？ I don't know if I misunderstood

Evaluate code and data

Hello, may I ask whether the code for quantitatively evaluating the effect of the model is considered public? We found that the data in the test set does not contain real images. How can I obtain the test data set with the original image?

Unable to reproduce the metrics in the paper

Since there is no code to evaluate the effect of reconstruction in this project, we refer to the implementation of the evaluation function of WarranWeng/ET-Net. However, we find that the metrics (the model is downloaded from there and the setting complies with the value given in the project) are different from the metrics published in the paper. For example, on the HQF data-set, we test E2VID model, the MSE value is only 0.0930 and the LPIPs is 0.5390, which are quite different from the 0.0611 and 0.4740 in the paper.

So, would you provide the test code or give a more detailed test configuration?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.