Hi, thanks for sharing your great work. I encounter some issues during creating data by running create_data.py
First
create reduced point cloud for training set
[ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last):
File "tools/create_data.py", line 247, in
out_dir=args.out_dir)
File "tools/create_data.py", line 24, in kitti_data_prep
kitti.create_reduced_point_cloud(root_path, info_prefix)
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/kitti_converter.py", line 374, in create_reduced_point_cloud
_create_reduced_point_cloud(data_path, train_info_path, save_path)
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/kitti_converter.py", line 314, in _create_reduced_point_cloud
count=-1).reshape([-1, num_features])
ValueError: cannot reshape array of size 461536 into shape (6)

It seems to set the num_features=4 and front_camera_id=2?
in this line:

I assume doing this can solve the problem but encounter another problem when
Create GT Database of KittiDataset
[ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last):
File "tools/create_data.py", line 247, in
out_dir=args.out_dir)
File "tools/create_data.py", line 44, in kitti_data_prep
with_bbox=True) # for moca
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/create_gt_database.py", line 275, in create_groundtruth_database
P0 = np.array(example['P0']).reshape(4, 4)
KeyError: 'P0'

Can you help me figure out how to solve these issues?

Hi, I like your good work!
I am wondering have you tried not to crop the gradient of $f^{\alpha}$ in eq7?
If you crop the gradient, it seems like the pertaining of the point branch cannot learn anything from the image branch.

Thanks for your inspiring work.
I have some wonder about eq5 and eq6.
As far as I know, After eq5, f should be a tensor which is a global feature with shape (batchsize * 2048 * 1 * 1), how can you sample corresponding image features by projection location? After all, there's no spatial information in f anymore.
Or maybe you got features from a previous layer of ResNet?
Looking forward to your reply.

Thanks for your insightful paper and clear code repo!

Hi, I met with the ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query' when run the command bash tools/dist_train.sh project_cl/configs/simipu/simipu_kitti.py 1 --work_dir ./

Do you know how to solve it?

Traceback (most recent call last):
File "tools/train.py", line 16, in
from mmdet3d.apis import train_model
File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/apis/init.py", line 1, in
from .inference import (convert_SyncBN, inference_detector,
File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/apis/inference.py", line 10, in
from mmdet3d.core import (Box3DMode, DepthInstance3DBoxes,
File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/init.py", line 2, in
from .bbox import * # noqa: F401, F403
File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/init.py", line 4, in
from .iou_calculators import (AxisAlignedBboxOverlaps3D, BboxOverlaps3D,
File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/iou_calculators/init.py", line 1, in
from .iou3d_calculator import (AxisAlignedBboxOverlaps3D, BboxOverlaps3D,
File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/iou_calculators/iou3d_calculator.py", line 5, in
from ..structures import get_box_type
File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/structures/init.py", line 1, in
from .base_box3d import BaseInstance3DBoxes
File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/structures/base_box3d.py", line 5, in
from mmdet3d.ops.iou3d import iou3d_cuda
File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/init.py", line 5, in
from .ball_query import ball_query
File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/init.py", line 1, in
from .ball_query import ball_query
File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/ball_query.py", line 4, in
from . import ball_query_ext
ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query' (/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/init.py)

I noticed that you once met with the same error.
open-mmlab/mmdetection3d#503 (comment)

So, I would like to ask for your help~ Hopefully you have a good solution. :)

[Question] Is this paper aims at pretraining a better camera backbone?
Actually, when I take a look at the given pth., it seems it is the pretrained parameter of a Resnet?

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/amFNsyUBvm or add me on WeChat (van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

Attention: please create a new virtual environment for OpenMMLab 2.0.

Hi, I'm a little confused about the data augmentation.

1. How did you set img_aug when img_moco=True? It seems that we need an 'img_pipeline' in 'simipu_kitti.py', right?
2. For 3D augmentation, it seems that it is done in this line. So the 3D augmentation is done based on the point features instead the raw points, right? If I want to try moco=True, how to set 3D augmentation? should I do this in the dataset building part?
   

   SimIPU/project_cl/decorator/inter_intro_decorator_moco_better.py

   Line 394 in 5b346e3

   loc_to_ori = apply_3d_transformation(loc_t[i], 'LIDAR', img_metas[i], reverse=True, points_center=self.cl_cfg["points_center"])

Looking forward to your reply. Many thanks.

Hello author, in the SimIPU paper, I find that the inter-modal module loss is gradient truncation in the point cloud features. When gradient backpropagation is considered, this step does not contribute to the update of the intra-modal module parameters. Why does adding this loss improve the point cloud 3D object detection performance?

Thanks for your excellent work first! I have a question about Tab.5 in Ablation Study. Why "Scratch" equals "SimIPU w/o inter-module ", which means that the intra-module is useless?

Could you tell me how to train on a single GPU？