Thank you for your outstanding work and valuable open-source contributions.
I have observed that the mAOE metric of Sparse4D is slightly inferior in r50 configurations compared to recent release methods (StreamPETR, SparseBEV and DynamicBEV), but it holds a great advantage on other configurations (r101, v2-99).
Do you have an explanation for this phenomenon? Anticipating your reply.

I was wondering whether you have tested with different batch sizes, in particular do you have any training config suggestions if I'm using total batch size of 8 or 16 or 32?

Hi, I'm fresh to nuscenes. I am just curious about how to generate the nuscenes_infos_train.pkl, nuscenes_infos_val.pkl, etc. Is there any difference bettween them and the nuscenes2d_temporal_infos_train.pkl in StreamPETR?

Btw, could you provide the script about how to generate nuscenes_kmeans900.npy, thanks so much!

Hi, when will the tracing module in the code be released?

作者您好，很感谢您出色的工作，我想请教一下是否有关注过该算法输出的预测框在深度上的误差？另外v2版本中增加了深度的监督后，深度是否更加准确呢？

Using this while training seems to just drop all of gts and predictions with mask being basically 0s.

This parameter is defined as 0.05 in configs:

But while running the training, this line causes problem:

Basically, all of the cls scores are below the threshold and thus mask's elements are set to 0s by this torch.logical_and gate.

Hi,
The grads are turned off for instance features. How are they updated?

初始的通过聚类得到的anchor有什么好处？ 可以随机初始化吗？

In Sparse4Dv3, is the tracking pipeline trained jointly with detection, or is it only used in inference to extend to tracking with Algorithm 1？

I wonder why we don't learn local yaw but yaw, will local yaw be better?

Hello, I am impressed with your papers.

I can check to provide only the config setting and model trained files for ResNet50 in Sparse4D v2.
So, can I get config files and ckpt files for ResNet-101 and VoVNet-99 for Sparse4D v2??

Additionally, I look forward to the Sparse4D v3 released.
Thank you!

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/A9dCpjHPfE or add me on WeChat (ID: van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

Attention: please create a new virtual environment for OpenMMLab 2.0.

Dear authors,

Thanks for the excellent work at first.
I'd like to know the nuScenes val set performance result of Sparse4D v1 on ResNet-50 because many methods compare the result of ResNet-50 on val set but there is only Resnet-101 in the paper.

I'll appreciate very much for your instant reply.

nuscenes_cam
│ │ ├── nuscenes_infos_test.pkl
│ │ ├── nuscenes_infos_train.pkl
│ │ ├── nuscenes_infos_val.pkl
│ │ └── nuscenes_infos_trainval_with_inds.pkl

Trainval_with_inds is valid at: https://github.com/linxuewu/Sparse4D/releases/download/v0.0/nuscenes_infos_trainval_with_inds.pkl
Train is valid at: https://github.com/linxuewu/Sparse4D/releases/download/v0.0/nuscenes_infos_train.pkl. BUT, mmcv load this pkl will encounter a problem:

File "/opt/miniconda3/envs/py38k/lib/python3.8/site-packages/mmcv/fileio/io.py", line 57, in load
raise TypeError(f'Unsupported format: {file_format}') 

Val is not valid at the web url.

Furthermore, can we prepare pkl files myself ?

可以提供模型训练的pth吗？

Can we futher improve mAP or NDS of valset and testset, if we use temporal information during inference?Thanks!

在resnet50版本中，instance_feature的requires_grad是False，如何更新self.instance_feature ？

Hello, thank you for your great work.

I am writing this issue to ask about the ablation study conducted on ego motion and object motion.
I believe that ego motion compensation follows similar process as BEVDet and BEVDepth, which aligns past frame t to current frame t_0.

For object motion compensation, I think it follows the equation 4 in the paper: .

And my question is how is it possible to conduct both ego and object motion compensation at the same time as in the last row of Table 2?

Thank you for your reply in advance.

感谢您非常棒的工作！这里想问一些问题。

问题1:对于跟踪任务而言，history query和det query是是有可能检测到同一个目标的吧？照论文的说法，会分配两个不同的track id，这个是如何解决的呢？

问题2:我在实验过程中，由于history query和det query有可能会命中相同的目标的，由于self attn会有抑制重复框的功能，会造成history query的score普遍偏低，所有检测结果基本都来源于当前帧的query，请问您在实验过程中有遇到过这种问题吗？

我理解InfiniteGroupEachSampleInBatchSampler 是每个batch单独跑一个stream/clip？在两个clip交界的时候， 请问怎么处理第一帧的，第一帧并没有上一帧进行叠帧。训练的时候哪里进行判断第一帧

I reviewed the code but didn't find the code about "Camera Parameter Encoding" noticed in Sec 3.4 in original paper, could you figure it out, thanks!

I tried training with projects/configs/sparse4dv2_r50_HInf_256x704.py and got NaN in grad_norm and loss in all attempts? Is this a known issue and is there a workaround?

I don't quite understand the purpose of cross attention here.
It seems that in this way , it will lose the information of new born object in current frame ?

The current state of the codebase is not well-organized and requires further cleanup. This issue aims to address the lack of clarity in the code structure and improve its overall cleanliness. While the code may still have some unresolved issues, your feedback is greatly appreciated, and I will make every effort to assist in resolving them. Rest assured, the code's cleanliness will continue to improve over time.

Thanks for your good job!

There is a small question for me, why put GridMask augmentation here rather than train_pipeline? This seems not that elegant, thanks.

Hi xuewu，
Sparse4D v3中关于yawness和centerness的训练的Y_pred和C_pred具体是？

Do you intend to add license for this project?

can't wait to try this SOTA Algorithm out XD

这是因为MMdet3D plugin中的插件没有注册成功吗，我应该如何解决。而且我能否使用Nuscenes的v1.0mini数据集进行实验？我应该在哪些部分进行更改？谢谢您！！

I trained the model using the unmodified code and data, expecting to reproduce sparse4d. However, sparse4dv2 failed to converge, and the grad_norm encountered NaN or Infinity, resulting in abnormal results. Do you have any suggestions on how to resolve this issue? Additionally, when can we expect the release of the Sparse4D-v3 code? I'm eagerly awaiting your response. Thank you!

Hello, in the paper and code the meaning of anchor[:, [w, h, l]] is lnw, lnh and lnl, but here in the output part, you just add output[:, [w, h, l]] and anchor[:, [w, h, l]] to get output anchor's lnw, lnh and lnl. I think the better formular should be output[:, [w, h, l]] = ln(output[:, [w, h, l]].exp() + anchor[:, [w, h, l]].exp()).

Dear author @linxuewu ,
This is really a nice work! However, when training "sparse4d_r101_H1" I will always meet a bug at these lines of code.

Where, the resulting mask is all False, causing an empty "reg_target".

感谢你非常棒的工作，看了你的代码后有一些疑惑请教你一下：
在V2版本中，训练过程中，instance_bank会在训练迭代的过程训练缓存当前的feature和anchor，训练集是按照sense划分的，整个数据集的数据不是连续的，如何保证从instance_bank获取的cache的feature和anchor与当前batch数据的连贯性呢？

谢谢！！！

Thank you for your great work.

How did you set the score_threshold of decoder to extract output from the results of 3D detection on nuScenes test data table in the sparse4D v2.0 paper?

Hello, I found that the version of mmdet3d is 0.17 < 1.0, while the latest version is >1.0 with some changes. Is the code suitable for the >1.0 version? (In my observation based on the bbox-coder code, I think yes.) If not, where should I revise, such as dataloader or boxcoder?

Thank you for your great work!
I wonder if you have released the code related to tracking, since the 'tracking for NuScenes3DDetTrackDataset' is False for all configs.

If you have released it, is there an example config?

will release the waymo challenge code?

I can't find it. Thank you

Will the hyperparameter noramalize_yaw in the refine module be helpful to pred sin & cos?

Check if your dataset permissions are executable, otherwise the training results will be very poor

作者您好，readme中图像预训练权重ImageNet的链接失效了，能否更新一下呢，感谢！

Thank for your sharing work! I don't know how to produce the ann file nuscenes_infos_train.pkl, nuscenes_infos_val.pkl and nuscenes_infos_test.pkl. Can you give me some guidelines?

Hi!
Are you planning to open-source your code?

Thanks

您好，请问Sparse4Dv2代码中，

1. 为什么DenseDepthNet模块使用回归任务来做监督，而不是普通使用的分类任务？
2. 回归损失函数为何这样设计，分母为什么是len(gt) * len(depth_preds) ？
   在我的模型里，error = torch.abs(depth_preds - gt_depths).sum()，error的值是338249.6469， 而分母 len(gt_depths) * len(depth_preds)的值是 30220*30220=913248400；使得最终得到的loss=338249.6469 / 913248400 = 0.00037038077143086153;这个值非常的小，同时乘以0.2的loss_weight以后就更小了，所以想问一下，这个depth_loss是否是符合预期的呢？

非常感谢~！