dvlab-research / vff Goto Github PK

View Code? Open in Web Editor NEW

95.0 3.0 11.0 2.81 MB

Voxel Field Fusion for 3D Object Detection (CVPR2022)

License: Apache License 2.0

Python 84.47% C++ 5.55% Cuda 9.32% C 0.44% Shell 0.22%

3d-detection

vff's Introduction

VFF

Voxel Field Fusion for 3D Object Detection

Yanwei Li, Xiaojuan Qi, Yukang Chen, Liwei Wang, Zeming Li, Jian Sun, Jiaya Jia

[arXiv] [BibTeX]

This project provides an implementation for the CVPR 2022 paper "Voxel Field Fusion for 3D Object Detection" based on OpenPCDet. VFF aims to maintain cross-modality consistency by representing and Image fusing augmented image features as a ray in the voxel field.

Installation

This project is based on OpenPCDet, which can be constructed as follows.

Install OpenPCDet following the instructions.
Prepare the dataset following the structure.

Training

You can train the model following the instructions. You can find the pretrained DeepLab V3 here if you want to train the model from scratch. For example, to launch PVRCNN-VFF training on multi GPUs, one should execute:

cd /path/to/vff/tools
bash scripts/dist_train.sh ${NUM_GPUS} --cfg_file cfgs/kitti_models/VFF_PVRCNN.yaml

or train with a single GPU:

python3 train.py --cfg_file cfgs/kitti_models/VFF_PVRCNN.yaml

Evaluation

You can evaluate the model following the instructions. For example, to launch PVRCNN-VFF evaluation with a pretrained checkpoint on multi GPUs, one should execute:

bash scripts/dist_test.sh ${NUM_GPUS} \
    --cfg_file cfgs/kitti_models/VFF_PVRCNN.yaml --batch_size ${BATCH_SIZE} --ckpt ${CKPT}

or evaluate with a single GPU:

python3 test.py --cfg_file cfgs/kitti_models/VFF_PVRCNN.yaml --batch_size ${BATCH_SIZE} --ckpt ${CKPT}

KITTI 3D Object Detection Results

We provide results on KITTI val set with pretrained models. All models are trained and evaluated on 8 V100 GPU.

	Car@R40	Pedestrian@R40	Cyclist@R40	download
PVRCNN-VFF	85.50	65.30	73.30	GoogleDrive
VoxelRCNN-VFF	85.72	-	-	GoogleDrive

Acknowledgement

We would like to thank the authors of OpenPCDet and CaDDN for their open-source release.

License

VFF is released under the Apache 2.0 license.

Citing VFF

Consider cite VFF in your publications if it helps your research.

@inproceedings{li2022vff,
  title={Voxel Field Fusion for 3D Object Detection},
  author={Li, Yanwei and Qi, Xiaojuan and Chen, Yukang and Wang, Liwei and Li, Zeming and Sun, Jian and Jia, Jiaya},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2022}
}

vff's People

Contributors

Stargazers

Watchers

Forkers

lianlau quotation2520 yeminglang jlqzzz vehxianfish freysteinctap unknowone 0nedawn mryu001 alvinti whuhxb

vff's Issues

python3 train.py --cfg_file cfgs/kitti_models/VFF_PVRCNN.yaml

when I run python3 train.py --cfg_file cfgs/kitti_models/VFF_PVRCNN.yaml, there is an error that:
File "VFF/pcdet/utils/transform_utils.py", line 44, in project_to_image
points_depth = points_t[..., -1] - raw_project[..., 2, 3]
RuntimeError: The size of tensor a (1760) must match the size of tensor b (2) at non-singleton dimension 1
I don't know how to fix it.

Graphics card type?

Can you tell the Graphics card type or vedio memory?

How can this problem be solved?

File "train.py", line 198, in
main()
File "train.py", line 114, in main
model = build_network(model_cfg=cfg.MODEL, num_class=len(cfg.CLASS_NAMES), dataset=train_set)
File "/home/abc/下载/VFF-main/pcdet/models/init.py", line 18, in build_network
model_cfg=model_cfg, num_class=num_class, dataset=dataset
File "/home/abc/下载/VFF-main/pcdet/models/detectors/init.py", line 30, in build_detector
model_cfg=model_cfg, num_class=num_class, dataset=dataset
File "/home/abc/下载/VFF-main/pcdet/models/detectors/voxel_rcnn_fusion.py", line 7, in init
self.module_list = self.build_networks()
File "/home/abc/下载/VFF-main/pcdet/models/detectors/detector3d_template.py", line 49, in build_networks
model_info_dict=model_info_dict
File "/home/abc/下载/VFF-main/pcdet/models/detectors/detector3d_template.py", line 64, in build_vfe
depth_downsample_factor=model_info_dict['depth_downsample_factor']
File "/home/abc/下载/VFF-main/pcdet/models/backbones_3d/vfe/image_point_vfe.py", line 18, in init
self.build_modules()
File "/home/abc/下载/VFF-main/pcdet/models/backbones_3d/vfe/image_point_vfe.py", line 25, in build_modules
module = getattr(self, 'build_%s' % module_name)()
File "/home/abc/下载/VFF-main/pcdet/models/backbones_3d/vfe/image_point_vfe.py", line 64, in build_f2v
disc_cfg=self.disc_cfg
File "/home/abc/下载/VFF-main/pcdet/models/backbones_3d/vfe/image_vfe_modules/f2v/voxel_field_fusion.py", line 34, in init
fuse_layer=model_cfg.LAYER_CHANNEL.keys())
File "/home/abc/下载/VFF-main/pcdet/models/backbones_3d/vfe/image_vfe_modules/f2v/point_to_image_projection.py", line 45, in init
device=device,dtype=torch.float32)
TypeError: create_meshgrid3d() got an unexpected keyword argument 'dtype'

More details about the training implementation

Hi, thanks for your neat and valuable work! By the way, I would like to know about the training batch size for KITTI and Waymo.

According to the paper, it seems that the batch_size for PV-RCNN on KITTI and Waymo are set as 8 and 64 respectively. I would like to make sure these parameters are right as I can not find them in the configuration files.

By the way, have you tested the influence of batch size on KITTI or Waymo?

Request for visualization on KITTI 3D OD

Could you provide some suggestions on the visualization on KITTI 3D object detection? Specifically, is there any useful related visualization repos ? : )

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16000, 16]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead

Hi, thanks for sharing such a good project~
I have some problems when I tried to train with
bash tools/scripts/dist_train.sh 2 --cfg_file /public/chenrunze/xyy/VFF-main/tools/cfgs/kitti_models/VFF_PVRCNN.yaml
Here is the error:

Traceback (most recent call last):
  File "tools/train.py", line 205, in <module>
    main()
  File "tools/train.py", line 160, in main
    train_model(
  File "/public/chenrunze/xyy/VFF-main/tools/train_utils/train_utils.py", line 88, in train_model
    accumulated_iter = train_one_epoch(
  File "/public/chenrunze/xyy/VFF-main/tools/train_utils/train_utils.py", line 41, in train_one_epoch
    loss.backward()
  File "/public/chenrunze/miniconda3/envs/bevfusion/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/public/chenrunze/miniconda3/envs/bevfusion/lib/python3.8/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: 
[torch.cuda.FloatTensor [16000, 16]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the 
backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or 
anywhere later. Good luck!
                                                                                                                                
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 100876 closing signal SIGTERM                               
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 100877) of binary: /public/chenrunze/miniconda3/envs/bevfusion/bin/python3
Traceback (most recent call last):
  File "/public/chenrunze/miniconda3/envs/bevfusion/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/public/chenrunze/miniconda3/envs/bevfusion/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/public/chenrunze/miniconda3/envs/bevfusion/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/public/chenrunze/miniconda3/envs/bevfusion/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/public/chenrunze/miniconda3/envs/bevfusion/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/public/chenrunze/miniconda3/envs/bevfusion/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
    elastic_launch(
  File "/public/chenrunze/miniconda3/envs/bevfusion/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/public/chenrunze/miniconda3/envs/bevfusion/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
tools/train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-02-12_16:28:26
  host      : 8265f0d3bcdf
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 100877)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

can u give some advice?
lots of thanks!

The size of tensor a (16000) must match the size of tensor b (2) at non-singleton dimension 1

Traceback (most recent call last):
File "/home/wistful/work/VFF/tools/train.py", line 199, in
main()
File "/home/wistful/work/VFF/tools/train.py", line 170, in main
merge_all_iters_to_one_epoch=args.merge_all_iters_to_one_epoch
File "/home/wistful/work/VFF/tools/train_utils/train_utils.py", line 93, in train_model
dataloader_iter=dataloader_iter
File "/home/wistful/work/VFF/tools/train_utils/train_utils.py", line 38, in train_one_epoch
loss, tb_dict, disp_dict = model_func(model, batch)
File "/home/wistful/work/VFF/pcdet/models/init.py", line 44, in model_func
ret_dict, tb_dict, disp_dict = model(batch_dict)
File "/home/wistful/.conda/envs/vff/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/wistful/work/VFF/pcdet/models/detectors/pv_rcnn_fusion.py", line 11, in forward
batch_dict = cur_module(batch_dict)
File "/home/wistful/.conda/envs/vff/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/wistful/work/VFF/pcdet/models/backbones_3d/vfe/image_point_vfe.py", line 104, in forward
batch_dict = self.backbone_3d(batch_dict, fuse_func=self.f2v)
File "/home/wistful/.conda/envs/vff/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/wistful/work/VFF/pcdet/models/backbones_3d/spconv_backbone.py", line 511, in forward
x_conv1, batch_dict = self.fusion(fuse_func, batch_dict, x_conv1, "layer1")
File "/home/wistful/work/VFF/pcdet/models/backbones_3d/spconv_backbone.py", line 480, in fusion
layer_name=layer_name)
File "/home/wistful/.conda/envs/vff/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/wistful/work/VFF/pcdet/models/backbones_3d/vfe/image_vfe_modules/f2v/voxel_field_fusion.py", line 164, in forward
layer_name=layer_name)
File "/home/wistful/.conda/envs/vff/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/wistful/work/VFF/pcdet/models/backbones_3d/vfe/image_vfe_modules/f2v/point_to_image_projection.py", line 163, in forward
layer_name=layer_name)
File "/home/wistful/work/VFF/pcdet/models/backbones_3d/vfe/image_vfe_modules/f2v/point_to_image_projection.py", line 115, in transform_grid
image_grid, image_depths = transform_utils.project_to_image(project=I_C, points=camera_grid, bmm=True)
File "/home/wistful/work/VFF/pcdet/utils/transform_utils.py", line 46, in project_to_image
points_depth = points_t[..., -1] - raw_project[..., 2, 3]
RuntimeError: The size of tensor a (16000) must match the size of tensor b (2) at non-singleton dimension 1

It is estimated that the shape is not correct, how to modify it?

Error about using dist_train.sh

I execute the following command: 'bash scripts/dist_train.sh 2 --cfg_file cfgs/kitti_models/VFF_PVRCNN.yaml', it always gives an error: 'FileExistsError: [Errno 17] File exists: '../checkpoints'', but I deleted this folder, still the same problem.
And when I use 'python train.py --cfg_file cfgs/kitti_models/VFF_PVRCNN.yaml' to train, is works fine.
Why is this? My English is not very good, hope you can understand. Thank you!

Error about using dist_train.sh

How about the inference speed?

@yanwei-li Thank you so much for your sharing! I am reading your paper now. I am curious about your model's inference speed.

kitti_data_beam_32

What is the setting for extracting 32 line LiDAR data from the Kitti dataset?
How should I replicate the experiment on 32 lines in my paper?
In the code, it appears that the data is directly read from the velodyne-beam-32 dataset, but the official version is not provided. The four line LiDAR extracted from pesolidar++provides specific angle settings, which is not provided in this article. I am a bit confused and look forward to your answer.

requires problems

hi, thanks for sharing such a great work!
I have some problems about install.
I installed spconv by ⓔ bevfusion ~/xyy/VFF-main pip install spconv-cu113
here is the result:

ⓔ bevfusion  ~/xyy/VFF-main   pip list | grep spconv
spconv-cu113             2.3.3

Python 3.8.15 (default, Nov 24 2022, 15:19:38) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import spconv
>>>

it looks good
however, spconv can not be found when i ran pcdet ⓔ bevfusion ~/xyy/VFF-main python setup.py develop
it went to like this:

Processing dependencies for pcdet==0.3.0+0
Searching for spconv
Reading https://pypi.org/simple/spconv/

and it will reached out because my device is offline.
It seems that the "spconv" in requires.txt is not "spconv-cu113", so I tried to modify the requires.txt in path ~/xyy/VFF-main/pcdet.egg-info/requires.txt to make the names consistent.
However, every time I run python setup.py develop the requires.txt would recover to original content which contains "spconv" instead of "spconv-cu113".
can you give me some advice?
lots of thanks!

About 2D pretrained model

Thanks for your excellent work! I notice that 2D pretrained deeplabv3 model on coco dataset is applied. I wonder if you have tested the model with the 2D pretrained model on Cityspaces dataset? Do you think it may lead to better performance? Thanks!
Looking for your reply!

GPU Memory-Usage

Hi,
Could you please provide the memory usage of GPU during training? Thanks!

RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

wen I run "python3 train.py --cfg_file cfgs/kitti_models/VFF_PVRCNN.yaml", there has a problem:

#Traceback (most recent call last): | 0/3712 [00:00<?, ?it/s]
File "train.py", line 198, in
main()
File "train.py", line 170, in main
merge_all_iters_to_one_epoch=args.merge_all_iters_to_one_epoch
File "/home/hu/VFF/tools/train_utils/train_utils.py", line 93, in train_model
dataloader_iter=dataloader_iter
File "/home/hu/VFF/tools/train_utils/train_utils.py", line 38, in train_one_epoch
loss, tb_dict, disp_dict = model_func(model, batch)
File "/home/hu/VFF/pcdet/models/init.py", line 44, in model_func
ret_dict, tb_dict, disp_dict = model(batch_dict)
File "/home/hu/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/hu/VFF/pcdet/models/detectors/pv_rcnn_fusion.py", line 11, in forward
batch_dict = cur_module(batch_dict)
File "/home/hu/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/hu/VFF/pcdet/models/backbones_3d/vfe/image_point_vfe.py", line 100, in forward
batch_dict = self.ffn(batch_dict)
File "/home/hu/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/hu/VFF/pcdet/models/backbones_3d/vfe/image_vfe_modules/ffn/pyramid_ffn.py", line 57, in forward
ifn_result = self.ifn(images)
File "/home/hu/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/hu/VFF/pcdet/models/backbones_3d/vfe/image_vfe_modules/ffn/ifn/seg_template.py", line 123, in forward
features = self.model.backbone(x)
File "/home/hu/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/hu/anaconda3/envs/pcdet/lib/python3.7/site-packages/torchvision/models/_utils.py", line 63, in forward
x = module(x)
File "/home/hu/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/hu/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 532, in forward
world_size = torch.distributed.get_world_size(process_group)
File "/home/hu/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 711, in get_world_size
return _get_group_size(group)
File "/home/hu/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 263, in _get_group_size
default_pg = _get_default_group()
File "/home/hu/anaconda3/envs/pcdet/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 347, in _get_default_group
raise RuntimeError("Default process group has not been initialized, "
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

It seems need to change multi GPU training to single GPU, like set "SyncBN" to "BN" or other, but I don't know where it is or some other solutions

RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

Hi there! I met a problem: when i ranpython3 train.py --cfg_file cfgs/kitti_models/PartA2_free.yaml, terminal reported RuntimeError: Default process group has not been initialized, please make sure to call init_process_group. How can i solve this problem? Thx

UnboundLocalError: local variable 'mv_height' referenced before assignment

Hi,
Thanks for your great work. However, I meet with some problems when trying training a model by myself. Which is shown as below

I prepared the dataset exactly the same way, although I set USE_ROAD_PLANE: False for my training. Is there something that I miss, or USE_ROAD_PLANE must be True? Thanks!

ValueError: could not broadcast input array from shape (164,109,4) into shape (164,109,3)

Traceback (most recent call last): | 413/3712 [06:18<49:02, 1.12it/s, total_it=413]
File "train.py", line 200, in
main()
File "train.py", line 172, in main
merge_all_iters_to_one_epoch=args.merge_all_iters_to_one_epoch
File "/hdd/code/VFF/tools/train_utils/train_utils.py", line 93, in train_model
dataloader_iter=dataloader_iter
File "/hdd/code/VFF/tools/train_utils/train_utils.py", line 19, in train_one_epoch
batch = next(dataloader_iter)
File "/opt/miniconda3/envs/clocs/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/opt/miniconda3/envs/clocs/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/opt/miniconda3/envs/clocs/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/opt/miniconda3/envs/clocs/lib/python3.7/site-packages/torch/_utils.py", line 425, in reraise
raise self.exc_type(msg)
ValueError: Caught ValueError in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/opt/miniconda3/envs/clocs/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/opt/miniconda3/envs/clocs/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/miniconda3/envs/clocs/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/hdd/code/VFF/pcdet/datasets/kitti/kitti_dataset.py", line 425, in getitem
data_dict = self.prepare_data(data_dict=input_dict)
File "/hdd/zhanglinjie/code/VFF/pcdet/datasets/dataset.py", line 130, in prepare_data
'gt_boxes_mask': gt_boxes_mask
File "/hdd/code/VFF/pcdet/datasets/augmentor/data_augmentor.py", line 135, in forward
data_dict = cur_augmentor(data_dict=data_dict)
File "/hdd/code/VFF/pcdet/datasets/augmentor/database_sampler.py", line 352, in call
total_valid_sampled_dict)
File "/hdd/code/VFF/pcdet/datasets/augmentor/database_sampler.py", line 276, in add_sampled_boxes_to_scene
data_dict = self.copy_paste_to_image(data_dict, gt_crops2d, gt_number, point_idxes)
File "/hdd/code/VFF/pcdet/datasets/augmentor/database_sampler.py", line 152, in copy_paste_to_image
image[_box2d[1]:_box2d[3],_box2d[0]:_box2d[2]] = crop_feat[_order]
ValueError: could not broadcast input array from shape (164,109,4) into shape (164,109,3)