Hi @NikolaZubic ,
Thanks for your wonderful job in event cameras.

Recently, I have been trying to replicate your work. I have installed the corresponding PyTorch and other libraries according to your readme file. However, with four 4090s, the training speed on the dataset gen1 is only 0.8 it/s. Is this speed normal?
What is the training speed on an A100?

Following is my train script.

#!/bin/bash
DATA_DIR="/home/data/gen1"
MDL_CFG=small
GPU_IDS=[0,1,2,3]
BATCH_SIZE_PER_GPU=4
TRAIN_WORKERS_PER_GPU=8
EVAL_WORKERS_PER_GPU=4
python RVT/train.py model=rnndet dataset=gen1 dataset.path=${DATA_DIR} wandb.project_name=ssms_event_cameras \
wandb.group_name=gen1 +experiment/gen1="${MDL_CFG}.yaml" hardware.gpus=${GPU_IDS} \
batch_size.train=${BATCH_SIZE_PER_GPU} batch_size.eval=${BATCH_SIZE_PER_GPU} \
hardware.num_workers.train=${TRAIN_WORKERS_PER_GPU} hardware.num_workers.eval=${EVAL_WORKERS_PER_GPU}

Hi @NikolaZubic

Hello, recently I carefully studied the paper and source code. There are a few aspects I don't quite understand.

Firstly, in the process of Output masking, why is only 'C' masked and not 'A' and 'B'?

Secondly, in your code, if only adjusting the 'step_scale', the mask of 'C' remains unaffected.

            step = step_scale * torch.exp(self.log_step)

            freqs = step / step_scale * self.Lambda[:, 1].abs() / (2 * math.pi)
            mask = torch.where(freqs < bandlimit * 0.5, 1, 0)  # (64, )

Since the freqs are not linked with the value of step_scale. (freqs = step / step_scale * self.Lambda[:, 1].abs() / (2 * math.pi) -> freqs = torch.exp(self.log_step) * self.Lambda[:, 1].abs() / (2 * math.pi)). Is the code wrong?

Question 2 is accompanied by another question, Question 3. In the paper, it's only mentioned that generalization from low frequency to high frequency is achieved by masking 'C', but in the code, the discretization of 'A' and 'B' is also related to 'step_scale'. So, I wonder, for generalization from low frequency to high frequency, is it necessary to adjust all three values 'A', 'B', and 'C'?

       if not torch.is_tensor(step_scale) or step_scale.ndim == 0:
            # step_scale = torch.ones(signal.shape[-2], device=signal.device) * step_scale
            step = step_scale * torch.exp(self.log_step)

Thanks for sharing this work! Looking forward to the code release~

Thank you for your wonderful work. How can I calculate the FLOPs of the model?

Hello! I have been interested in RVT for a long time . I think you have done a great job! I have a question and I hope you can help me answer it. After I reproduce RVT, I have the following reasoning time:
Loading and preparing results DONE (t=0.41s) creating index Index created!
Running per image evaluation Evaluate annotation type * bbox * DONE (t=7.23s)
Accumulating evaluation results DONE (t=2.31s)
May I ask how to infer the time velocity in ms, just like 7.81 ms in your paper on the gen1 dataset.
I am using a 3090TIGPU with a batch_size of 1, so what should be the speed in ms?

Hi, thank you for publishing great work!

I have been trying to reproduce the results on the Gen1 dataset, but my findings are significantly different from those reported in the paper. Below are the details of my experiment setup and the results I obtained.

Environment Configuration:

GPU_IDS: [0,1]         
BATCH_SIZE_PER_GPU: 4  
TRAIN_WORKERS_PER_GPU: 12
EVAL_WORKERS_PER_GPU: 4
DATA_DIR: /root/data/gen1
MDL_CFG: base

I have used an A6000 40GB GPU (two units) instead of the A100 80GB (one unit) as mentioned in the GitHub repository. Since the total memory is the same, I divided the configurations accordingly to allocate the memory across the two GPUs.

I have not modified anything in RVT/config/experiment/gen1/default.yaml.

Observed Results:
During training, I noticed that the max_step was set to 400,000, causing the process to stop after one epoch. The total training time was approximately 67 hours (2 days and 19 hours).

Here are the validation metrics I obtained:

- val/AP: 0.4406683404220787
- val/AP_50: 0.6706345137471073
- val/AP_75: 0.47589861083495194
- val/AP_L: 0.3979635967270493
- val/AP_M: 0.5222804973442176
- val/AP_S: 0.3784906607967766

These results are considerably lower than those reported in the paper.

Questions:
1. Why are the results different from the paper?
2. Is the training time normal? Is it expected to only run for 1 epoch?

Additionally, in the supplementary material, there are detection results images for the DSEC dataset. Could you please let me know if there are any available results or code for training on the DSEC dataset?

Thank you for your assistance.

Hi @NikolaZubic

Thanks for your nice work and opening source!
I have a question when testing in a higher frequency, which parameters we should change?

I have found two places about the 'step_scale':

class S5SSM(torch.nn.Module):
    def __init__(
        self,
        lambdaInit: torch.Tensor,
        V: torch.Tensor,
        Vinv: torch.Tensor,
        h: int,
        p: int,
        dt_min: float,
        dt_max: float,
        liquid: bool = False,
        factor_rank: Optional[int] = None,
        discretization: Literal["zoh", "bilinear"] = "bilinear",
        bcInit: Initialization = "factorized",
        degree: int = 1,
        bidir: bool = False,
        step_scale: float = 1.0,
        bandlimit: Optional[float] = None,
    ):

and

    def forward(self, signal, prev_state, step_scale: float | torch.Tensor = 1.0): 

in S5SSM moudle.

If I want two testing in frequecy 200Hz (training in frequency 20), how should I adjust the parameter?
Should I change the above step_scale from 1.0 to 0.1 both?

Hello, I am unable to download the dataset from your server right now. Could you please check if it is a server issue? Thanks!

Hey there,

I'm the author of s5-pytorch, and noticed you used added the code to the project. I'm happy it's been of use to you, but I would appreciate it if the license for that directory was indicated as having original license (MPL-2.0) it had, or a link to the original source so license can be found there.

Thanks in advance