dachunkai / evtexture Goto Github PK
View Code? Open in Web Editor NEW[ICML 2024] EvTexture: Event-driven Texture Enhancement for Video Super-Resolution
Home Page: https://dachunkai.github.io/evtexture.github.io/
License: Apache License 2.0
[ICML 2024] EvTexture: Event-driven Texture Enhancement for Video Super-Resolution
Home Page: https://dachunkai.github.io/evtexture.github.io/
License: Apache License 2.0
Requirement already satisfied: setuptools in c:\users\miniconda3\envs\evtexture\lib\site-packages (from -r requirements.txt (line 17)) (65.6.3)
ERROR: Ignored the following versions that require a different python version: 0.20.0 Requires-Python >=3.8; 0.20.0rc2 Requires-Python >=3.8; 0.20.0rc3 Requires-Python >=3.8; 0.20.0rc4 Requires-Python >=3.8; 0.20.0rc5 Requires-Python >=3.8; 0.20.0rc6 Requires-Python >=3.8; 0.20.0rc7 Requires-Python >=3.8; 0.20.0rc8 Requires-Python >=3.8; 0.21.0 Requires-Python >=3.8; 0.21.0rc0 Requires-Python >=3.8; 0.21.0rc1 Requires-Python >=3.8; 0.22.0 Requires-Python >=3.9; 0.22.0rc1 Requires-Python >=3.9; 0.23.0 Requires-Python >=3.10; 0.23.0rc0 Requires-Python >=3.10; 0.23.0rc2 Requires-Python >=3.10; 0.23.1 Requires-Python >=3.10; 0.23.2 Requires-Python >=3.10; 0.23.2rc1 Requires-Python >=3.10; 0.24.0 Requires-Python >=3.9; 0.24.0rc1 Requires-Python >=3.9; 0.6.2rc0 Requires-Python >=3.8; 0.7.0 Requires-Python >=3.8; 0.7.0rc1 Requires-Python >=3.8; 0.7.0rc2 Requires-Python >=3.8; 0.8.0 Requires-Python >=3.8; 1.10.0 Requires-Python <3.12,>=3.8; 1.10.0rc1 Requires-Python <3.12,>=3.8; 1.10.0rc2 Requires-Python <3.12,>=3.8; 1.10.1 Requires-Python <3.12,>=3.8; 1.11.0 Requires-Python <3.13,>=3.9; 1.11.0rc1 Requires-Python <3.13,>=3.9; 1.11.0rc2 Requires-Python <3.13,>=3.9; 1.11.1 Requires-Python <3.13,>=3.9; 1.11.2 Requires-Python <3.13,>=3.9; 1.11.3 Requires-Python <3.13,>=3.9; 1.11.4 Requires-Python >=3.9; 1.12.0 Requires-Python >=3.9; 1.12.0rc1 Requires-Python >=3.9; 1.12.0rc2 Requires-Python >=3.9; 1.13.0 Requires-Python >=3.9; 1.13.0rc1 Requires-Python >=3.9; 1.13.1 Requires-Python >=3.9; 1.14.0 Requires-Python >=3.10; 1.14.0rc1 Requires-Python >=3.10; 1.14.0rc2 Requires-Python >=3.10; 1.4.0 Requires-Python >=3.8; 1.4.0rc0 Requires-Python >=3.8; 1.4.1 Requires-Python >=3.8; 1.4.2 Requires-Python >=3.8; 1.4.3 Requires-Python >=3.8; 1.4.4 Requires-Python >=3.8; 1.5.0 Requires-Python >=3.8; 1.5.0rc0 Requires-Python >=3.8; 1.5.1 Requires-Python >=3.8; 1.5.2 Requires-Python >=3.8; 1.5.3 Requires-Python >=3.8; 1.8.0 Requires-Python >=3.8,<3.11; 1.8.0rc1 Requires-Python >=3.8,<3.11; 1.8.0rc2 Requires-Python >=3.8,<3.11; 1.8.0rc3 Requires-Python >=3.8,<3.11; 1.8.0rc4 Requires-Python >=3.8,<3.11; 1.8.1 Requires-Python >=3.8,<3.11; 1.9.0 Requires-Python >=3.8,<3.12; 1.9.0rc1 Requires-Python >=3.8,<3.12; 1.9.0rc2 Requires-Python >=3.8,<3.12; 1.9.0rc3 Requires-Python >=3.8,<3.12; 1.9.1 Requires-Python >=3.8,<3.12; 1.9.2 Requires-Python >=3.8; 1.9.3 Requires-Python >=3.8; 2.0.0 Requires-Python >=3.8; 2.0.0rc0 Requires-Python >=3.8; 2.0.0rc1 Requires-Python >=3.8; 2.0.1 Requires-Python >=3.8; 2.0.2 Requires-Python >=3.8; 2.0.3 Requires-Python >=3.8; 2.1.0 Requires-Python >=3.9; 2.1.0rc0 Requires-Python >=3.9; 2.1.1 Requires-Python >=3.9; 2.1.2 Requires-Python >=3.9; 2.1.3 Requires-Python >=3.9; 2.1.4 Requires-Python >=3.9; 2.2.0 Requires-Python >=3.9; 2.2.0rc0 Requires-Python >=3.9; 2.2.1 Requires-Python >=3.9; 2.2.2 Requires-Python >=3.9; 2.32.0 Requires-Python >=3.8; 2.32.1 Requires-Python >=3.8; 2.32.2 Requires-Python >=3.8; 2.32.3 Requires-Python >=3.8; 3.10.0 Requires-Python >=3.8; 3.11.0 Requires-Python >=3.8; 3.6.0 Requires-Python >=3.8; 3.6.0rc1 Requires-Python >=3.8; 3.6.0rc2 Requires-Python >=3.8; 3.6.1 Requires-Python >=3.8; 3.6.2 Requires-Python >=3.8; 3.6.3 Requires-Python >=3.8; 3.7.0 Requires-Python >=3.8; 3.7.0rc1 Requires-Python >=3.8; 3.7.1 Requires-Python >=3.8; 3.7.2 Requires-Python >=3.8; 3.7.3 Requires-Python >=3.8; 3.7.4 Requires-Python >=3.8; 3.7.5 Requires-Python >=3.8; 3.8.0 Requires-Python >=3.9; 3.8.0rc1 Requires-Python >=3.9; 3.8.1 Requires-Python >=3.9; 3.8.2 Requires-Python >=3.9; 3.8.3 Requires-Python >=3.9; 3.8.4 Requires-Python >=3.9; 3.9.0 Requires-Python >=3.8; 3.9.0 Requires-Python >=3.9; 3.9.0rc2 Requires-Python >=3.9; 3.9.1 Requires-Python >=3.9
ERROR: Could not find a version that satisfies the requirement tb-nightly (from versions: none)
ERROR: No matching distribution found for tb-nightly
Thank you for sharing the data preparation details. I have created events for an image dataset, and the events.h5 file looks as follows:
events/ps/ : (7013772,) bool
events/ts/ : (7013772,) float64
events/xs/ : (7013772,) int16
events/ys/ : (7013772,) int16
Could you please share the code snippet that can convert these events into voxels in the format specified below?
voxels_b/000000/ : (5, 180, 320) float64
voxels_b/000001/ : (5, 180, 320) float64
voxels_b/000002/ : (5, 180, 320) float64
voxels_b/000003/ : (5, 180, 320) float64
voxels_b/000004/ : (5, 180, 320) float64
I tried using [events_contrast_maximization] but ended up generating voxels in a different format, which is incorrect. My dataset contains 26 images, but the voxel file only has 5 lines, and the tensor shape [bins, H, W] is also incorrect.
voxels_f/000000/ : (720, 1280) float64
voxels_f/000001/ : (720, 1280) float64
voxels_f/000002/ : (720, 1280) float64
voxels_f/000003/ : (720, 1280) float64
voxels_f/000004/ : (720, 1280) float64
The generated voxel file does not include all 26 images. Please assist. Thank you.
import random
import esim_py
import os
import matplotlib.pyplot as plt
import numpy as np
import torch
from abc import ABCMeta, abstractmethod
import h5py
import cv2 as cv
import numpy as np
from utils.event_utils import events_to_voxel_torch
image_folder = os.path.join(os.path.dirname(__file__), "data/frames/")
timestamps_file = os.path.join(os.path.dirname(__file__), "data/timestamps.txt")
config = {
'refractory_period': 1e-4,
'CT_range': [0.05, 0.5],
'max_CT': 0.5,
'min_CT': 0.02,
'mu': 1,
'sigma': 0.1,
'H': 720,
'W': 1280,
'log_eps': 1e-3,
'use_log': True,
}
Cp = random.uniform(config['CT_range'][0], config['CT_range'][1])
Cn = random.gauss(config['mu'], config['sigma']) * Cp
Cp = min(max(Cp, config['min_CT']), config['max_CT'])
Cn = min(max(Cn, config['min_CT']), config['max_CT'])
esim = esim_py.EventSimulator(Cp,
Cn,
config['refractory_period'],
config['log_eps'],
config['use_log'])
events = esim.generateFromFolder(image_folder, timestamps_file) # Generate events with shape [N, 4]
xs = torch.tensor(events[:, 0], dtype=torch.int16)
ys = torch.tensor(events[:, 1], dtype=torch.int16)
ts = torch.tensor(events[:, 2], dtype=torch.float32) # Use float32 for consistency
ps = torch.tensor(events[:, 3], dtype=torch.bool)
voxel_f = events_to_voxel_torch(xs, ys, ts, ps, 5, device=None, sensor_size=(720, 1280), temporal_bilinear=True)
output_h5_file = "voxel_fx.h5" # Replace with desired output path
B = 5 # Number of bins in voxel grids
n = 1000 # Number of events per voxel (not sure what to set)
temporal_bilinear = True
with h5py.File(output_h5_file, 'w') as output_file:
# Save each voxel grid to HDF5 file with the specified format
for i, voxel in enumerate(voxel_f):
print(list(voxel.shape))
voxel_name = f'voxels_f/{i:06d}'
output_file.create_dataset(voxel_name, data=voxel.numpy().astype(np.float64)) # Convert to float64
# Add metadata or attributes if needed
output_file.attrs['num_voxels'] = len(voxel_f)
output_file.attrs['B'] = B
output_file.attrs['n'] = n
print(f"Voxel data saved to {output_h5_file}")
Description
The model code assumes that event video data is included as part of the inputs. However, it's unclear how this was managed for the demonstration purposes, especially since the test datasets like Vid4 and REDS seem to be captured using traditional frame-based cameras.
Could you please provide details on the following:
Generation of Event Data: How was event data generated or simulated for the demonstration? Was there a specific method or tool used to convert traditional frame-based footage into event-based data?
Implementation Details: Any specific scripts or code examples used to achieve this conversion would be highly appreciated. Understanding the methodology would help in replicating the demo setup accurately.
Test Data Adaptation: If the event data was simulated, what adjustments or preprocessing steps were necessary to align this data with the model's requirements?
Hi how does this compare to video2x? And what is the VRAM/GPU req for running the models? thanks!
Thanks for your wanderful work!
I have some questions about the event data:
I appreciate your guidance on these queries as I am looking to better understand the processes involved in working with event data.
Thanks again!
PS: Is there any WeChat group or Discord channel where I can discuss these topics further?
I am trying to get this working under Windows without needing Docker. This should be possible right? Docker just complicates the setup. It should be create an environment, pip install the requirements, download models and go.
At this point I have a batch file to do all the setup and download the required models. This will get it all downloaded and setup on a Windows PC. Save this as install.bat into an empty directory and run it. For the downloads you also need wget.exe, 7z.ese and 7z.dll in the same directory.
@echo off
cd
echo *** Deleting EvTexture directory if it exists
if exist EvTexture\. rd /S /Q EvTexture
echo *** Cloning EvTexture repository
git clone https://github.com/DachunKai/EvTexture
cd EvTexture
echo *** Creating venv
python -m venv venv
call venv\scripts\activate.bat
echo *** Installing EvTexture requirements
python -m pip install --upgrade pip
pip install -r requirements.txt
echo *** Patching xformers
pip uninstall -y xformers
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts xformers==0.0.25 --index-url https://download.pytorch.org/whl/cu118
echo *** Installing GPU torch
pip uninstall -y torch
pip uninstall -y torch
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts torch==2.3.1+cu118 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
echo *** python setup.py develop
python setup.py develop
echo *** Downloading models
md experiments
md experiments\pretrained_models
md experiments\pretrained_models\EvTexture
..\wget "https://github.com/DachunKai/EvTexture/releases/download/v0.0/EvTexture_REDS_BIx4.pth" -O "experiments\pretrained_models\EvTexture\EvTexture_REDS_BIx4.pth" -nc
..\wget "https://github.com/DachunKai/EvTexture/releases/download/v0.0/EvTexture_Vimeo90K_BIx4.pth" -O "experiments\pretrained_models\EvTexture\EvTexture_Vimeo90K_BIx4.pth" -nc
echo *** Downloading test sets
..\wget "https://github.com/DachunKai/EvTexture/releases/download/v0.0/test_sets_REDS4_h5.zip" -O "datasets\test_sets_REDS4_h5.zip" -nc
..\wget "https://github.com/DachunKai/EvTexture/releases/download/v0.0/test_sets_Vid4_h5.zip" -O "datasets\test_sets_Vid4_h5.zip" -nc
echo *** Extracting test sets
cd datasets
..\..\7z x test_sets_REDS4_h5.zip
del test_sets_REDS4_h5.zip
..\..\7z x test_sets_Vid4_h5.zip
del test_sets_Vid4_h5.zip
cd..
call venv\scripts\deactivate.bat
cd..
echo *** Finished EvTexture install
echo.
echo *** Scroll up and check for errors. Do not assume it worked.
pause
Then to run the examples, once the venv is activated I run either
python basicsr\test.py -opt options/test/EvTexture/test_EvTexture_Vid4_BIx4.yml --launcher pytorch
or
python basicsr\test.py -opt options/test/EvTexture/test_EvTexture_REDS4_BIx4.yml --launcher pytorch
At that point I get the following error.
File "D:\Tests\EvTexture\EvTexture\basicsr\test.py", line 48, in <module>
test_pipeline(root_path)
File "D:\Tests\EvTexture\EvTexture\basicsr\test.py", line 16, in test_pipeline
opt, _ = parse_options(root_path, is_train=False)
File "d:\tests\evtexture\evtexture\basicsr\utils\options.py", line 122, in parse_options
init_dist(args.launcher)
File "d:\tests\evtexture\evtexture\basicsr\utils\dist_util.py", line 14, in init_dist
_init_dist_pytorch(backend, **kwargs)
File "d:\tests\evtexture\evtexture\basicsr\utils\dist_util.py", line 22, in _init_dist_pytorch
rank = int(os.environ['RANK'])
File "D:\Python\lib\os.py", line 680, in __getitem__
raise KeyError(key) from None
KeyError: 'RANK'
Is it possible to get a simpler example with a script like
python upscale_video.py --source myvideo.mp4 --output upscaled.mp4 --other --options --here
so it is easier to get working without needing Docker?
Thanks for any help.
Before spending my time on this to make it installed and run can you tell me? What is the resolution?
Like can we upscale 512x512 video to 1024x1024 or 2048x2048?
thank you
With all due respect to the author, dataset preparation is overly complicated even for someone who has been using computers for decades like myself. Is it really that complex? Could someone please create a detailed tutorial? I would really love to try this seemingly phenomenal open-source project, but not everything is explained properly. Really.
Thank you to everyone who can help!
When running basicsr/test.py for EvTexture on a macOS system without GPU support, the script fails due to CUDA-related operations despite setting use_gpu: False.
Error Message:
AssertionError: Torch not compiled with CUDA enabled
Details:
Steps to Reproduce:
Configure basicsr/test.py to use CPU (use_gpu: False).
Execute the script on macOS.
Expected Behavior: The script should run without attempting to initialize CUDA.
Actual Behavior: The script fails with a CUDA-related error.
Potential Solution: Modify the dist_validation method to ensure all operations are explicitly set to run on the CPU.
Code Snippet Causing Issue:
self.metric_results[folder] = torch.zeros(size, dtype=torch.float32, device=torch.device('cpu'))
Request for Assistance
Please provide guidance on ensuring the dist_validation method and related functions do not attempt to use CUDA when use_gpu: False.
Hi, first off, congratulations on your work. The use of event data for this is fascinating.
I'm trying to reproduce your work using simulated data and I have a couple of question
What is the FPS you recommend for frame interpolation? As you use a B=5, would it be a safe bet to assume 5 extra frames, between every 2 frames?
Am I understanding this correct? for calendar.h5, data looks like this:
dict_keys(['images/000000', 'images/000001', 'images/000002', 'images/000003', 'images/000004', 'images/000005', 'images/000006', 'images/000007', 'images/000008', 'images/000009', 'images/000010', 'images/000011', 'images/000012', 'images/000013', 'images/000014', 'images/000015', 'images/000016', 'images/000017', 'images/000018', 'images/000019', 'images/000020', 'images/000021', 'images/000022', 'images/000023', 'images/000024', 'images/000025', 'images/000026', 'images/000027', 'images/000028', 'images/000029', 'images/000030', 'images/000031', 'images/000032', 'images/000033', 'images/000034', 'images/000035', 'images/000036', 'images/000037', 'images/000038', 'images/000039', 'images/000040', 'voxels_b/000000', 'voxels_b/000001', 'voxels_b/000002', 'voxels_b/000003', 'voxels_b/000004', 'voxels_b/000005', 'voxels_b/000006', 'voxels_b/000007', 'voxels_b/000008', 'voxels_b/000009', 'voxels_b/000010', 'voxels_b/000011', 'voxels_b/000012', 'voxels_b/000013', 'voxels_b/000014', 'voxels_b/000015', 'voxels_b/000016', 'voxels_b/000017', 'voxels_b/000018', 'voxels_b/000019', 'voxels_b/000020', 'voxels_b/000021', 'voxels_b/000022', 'voxels_b/000023', 'voxels_b/000024', 'voxels_b/000025', 'voxels_b/000026', 'voxels_b/000027', 'voxels_b/000028', 'voxels_b/000029', 'voxels_b/000030', 'voxels_b/000031', 'voxels_b/000032', 'voxels_b/000033', 'voxels_b/000034', 'voxels_b/000035', 'voxels_b/000036', 'voxels_b/000037', 'voxels_b/000038', 'voxels_b/000039', 'voxels_f/000000', 'voxels_f/000001', 'voxels_f/000002', 'voxels_f/000003', 'voxels_f/000004', 'voxels_f/000005', 'voxels_f/000006', 'voxels_f/000007', 'voxels_f/000008', 'voxels_f/000009', 'voxels_f/000010', 'voxels_f/000011', 'voxels_f/000012', 'voxels_f/000013', 'voxels_f/000014', 'voxels_f/000015', 'voxels_f/000016', 'voxels_f/000017', 'voxels_f/000018', 'voxels_f/000019', 'voxels_f/000020', 'voxels_f/000021', 'voxels_f/000022', 'voxels_f/000023', 'voxels_f/000024', 'voxels_f/000025', 'voxels_f/000026', 'voxels_f/000027', 'voxels_f/000028', 'voxels_f/000029', 'voxels_f/000030', 'voxels_f/000031', 'voxels_f/000032', 'voxels_f/000033', 'voxels_f/000034', 'voxels_f/000035', 'voxels_f/000036', 'voxels_f/000037', 'voxels_f/000038', 'voxels_f/000039'])
where the shape of each individual voxel tensor is [B, H, W].
So in order to replicate this, should I use events_to_voxel_torch
on the Event data of each real frame, individually? (When I say real frame, I mean, the original frame plus all the interpolated frames between t and t+1)
Thank you
Dear team,
Great super-resolution work! It would be awesome if the model is hosted on the Hugging Face and it will increase the impact of your work too.
There are several advantages of hosting the models on HF compared to Google Drive/Baidu disk.
We also provide free A100 GPU for setting up demos via ZeroGPU: https://www.theverge.com/2024/5/16/24156755/hugging-face-celement-delangue-free-shared-gpus-ai
Let me know how you think and happy to invite you to our Slack channel for further discussion / assistant!
Thanks,
Tiezhen
Can you please share the inference script that takes the h5 data generated for a low resolution video (180 x 320) as an input and generate the SR output video (720, 1280)?
hope you guys setup a colab inference notebook where users can upload .mp4 and check the results.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.