How do you calculate the latency of FB-BEV, F-VTM + B-VTM + FRPN？Thanks!

Has anyone tried to export any of the FB-OCC or FB-BEV model to ONNX format? Is it even possible?

Will this open source 3D detection code and config?

Is the simulator that turns this source code on the nvidia drive sim? Is the nvidia drive sim the only thing that needs to be installed separately?

warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
/opt/conda/envs/fbocc/lib/python3.8/site-packages/torch/utils/checkpoint.py:25: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
/opt/conda/envs/fbocc/lib/python3.8/site-packages/torch/utils/checkpoint.py:25: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
[                                                  ] 4/6019, 0.4 task/s, elapsed: 9s, ETA: 14005s/opt/conda/envs/fbocc/lib/python3.8/site-packages/torch/utils/checkpoint.py:25: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 6084/6019, 21.4 task/s, elapsed: 284s, ETA:    -2sWARNING:torch.distributed.elastic.multiprocessing.api:Sending process 795432 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 795433 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 795434 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 3 (pid: 795435) of binary: /opt/conda/envs/fbocc/bin/python
Traceback (most recent call last):
  File "/opt/conda/envs/fbocc/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/fbocc/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/.vscode-server/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/root/.vscode-server/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/root/.vscode-server/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 317, in run_module
    run_module_as_main(options.target, alter_argv=True)
  File "/root/.vscode-server/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 238, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/.vscode-server/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/fbocc/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/opt/conda/envs/fbocc/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/opt/conda/envs/fbocc/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/opt/conda/envs/fbocc/lib/python3.8/site-packages/torch/distributed/run.py", line 752, in run
    elastic_launch(
  File "/opt/conda/envs/fbocc/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/conda/envs/fbocc/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
tools_mm/test.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-08-30_01:11:31
  host      : fd-taxi-f-houjiawei-1691719963348-985c0990-3609275187
  rank      : 3 (local_rank: 3)
  exitcode  : -9 (pid: 795435)
  error_file: <N/A>
  traceback : Signal 9 (SIGKILL) received by PID 795435
============================================================

The evaluation run 6084 samples rather than 6019 samples and closed unexpectly.

Is this model able to convert to tensorrt engine?

An error was reported when I ran the occ model given by o official, Loading error, detailed as follows:
run : python ./tools/test.py /zz/code/FB-BEV/occupancy_configs/fb_occ/fbocc-r50-cbgs_depth_16f_16x4_20e.py /zz/code/FB-BEV-main/fbocc-r50-cbgs_depth_16f_16x4_20e.pth

raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for FBOCC:
Unexpected key(s) in state_dict: "img_sem_seg_head.seg_net.0.weight", "img_sem_seg_head.seg_net.0.bias", "img_sem_seg_head.seg_net.3.weight", "img_sem_seg_head.seg_net.3.bias".

In backward projection, context is used as the transform key. Is there a reason to use context instead of image feature? I don't know what the difference is between image feature and context.

Thanks so much for sharing!

I would like to ask if the resnet50 used in Table 1 is the imagenet pre-trained or other data? How many epochs were trained for the models in Table 1?

When I use launch.json in Vscode to debug the code, I am training with a single card, and it will report an error: Default process group has not been initialized, please make sure to call init_process_group.
Can anyone here help me?

Exception has occurred: RuntimeError (note: full exception trace is shown but execution is paused at: _run_module_as_main)
Default process group has not been initialized, please make sure to call init_process_group.
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 429, in _get_default_group
raise RuntimeError(
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 325, in _get_group_size
default_pg = _get_default_group()
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 867, in get_world_size
return _get_group_size(group)
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 731, in forward
world_size = torch.distributed.get_world_size(process_group)
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/code_lcy/FB-BEV/mmdet3d/models/fbbev/detectors/fbocc.py", line 296, in fuse_history
feats_to_return = self.history_keyframe_time_conv(
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 226, in new_func
output = old_func(*new_args, **new_kwargs)
File "/code_lcy/FB-BEV/mmdet3d/models/fbbev/detectors/fbocc.py", line 364, in extract_img_bev_feat
bev_feat = self.fuse_history(bev_feat, img_metas, img[6])
File "/code_lcy/FB-BEV/mmdet3d/models/fbbev/detectors/fbocc.py", line 386, in extract_feat
results.update(self.extract_img_bev_feat(img, img_metas, **kwargs))
File "/code_lcy/FB-BEV/mmdet3d/models/fbbev/detectors/fbocc.py", line 432, in forward_train
results= self.extract_feat(
File "/code_lcy/FB-BEV/mmdet3d/models/detectors/base.py", line 60, in forward
return self.forward_train(**kwargs)
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 140, in new_func
output = old_func(*new_args, **new_kwargs)
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 248, in train_step
losses = self(**data)
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 75, in train_step
return self.module.train_step(*inputs[0], **kwargs[0])
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 62, in train
outputs = self.model.train_step(data_batch, self.optimizer, **kwargs)
File "/opt/conda/envs/mmlab/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 138, in run
iter_runner(iter_loaders[i], **kwargs)
File "/code_lcy/FB-BEV/mmdet3d/apis/train.py", line 331, in train_detector
runner.run(data_loaders, cfg.workflow)
File "/code_lcy/FB-BEV/mmdet3d/apis/train.py", line 357, in train_model
train_detector(
File "/code_lcy/FB-BEV/tools/train.py", line 360, in main
train_model(
File "/code_lcy/FB-BEV/tools/train.py", line 370, in
main()
File "/opt/conda/envs/mmlab/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/mmlab/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/opt/conda/envs/mmlab/lib/python3.8/runpy.py", line 265, in run_path
return _run_module_code(code, init_globals, run_name,
File "/opt/conda/envs/mmlab/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/mmlab/lib/python3.8/runpy.py", line 194, in _run_module_as_main (Current frame)
return _run_code(code, main_globals, None,

I conducted an ablation study where I set 'backward_projection' to None (while keeping other configurations at their default settings). The results showed that the availability of this module did not have a significant impact on the results. Here are the results for epochs 2 and 4:

Can you please share any ablation results you may have for this module?

Can the author provide a log file for training the model so that we can correspond to the training situation at each stage？

Hello,
I have completed the model training phase , but an error occurred during the evaluation. The specific information is as follows：

Traceback (most recent call last):
File "./tools/test.py", line 15, in
from mmdet3d.apis import single_gpu_test
File "/home/jl/FB-BEV/mmdet3d/apis/init.py", line 2, in
from .inference import (convert_SyncBN, inference_detector,
File "/home/jl/FB-BEV/mmdet3d/apis/inference.py", line 12, in
from mmdet3d.core import (Box3DMode, CameraInstance3DBoxes, Coord3DMode,
File "/home/jl/FB-BEV/mmdet3d/core/init.py", line 4, in
from .evaluation import * # noqa: F401, F403
File "/home/jl/FB-BEV/mmdet3d/core/evaluation/init.py", line 4, in
from .kitti_utils import kitti_eval, kitti_eval_coco_style
File "/home/jl/FB-BEV/mmdet3d/core/evaluation/kitti_utils/init.py", line 2, in
from .eval import kitti_eval, kitti_eval_coco_style
File "/home/jl/FB-BEV/mmdet3d/core/evaluation/kitti_utils/eval.py", line 5, in
import numba
File "/home/jl/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/numba/init.py", line 43, in
from numba.np.ufunc import (vectorize, guvectorize, threading_layer,
File "/home/jl/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/numba/np/ufunc/init.py", line 3, in
from numba.np.ufunc.decorators import Vectorize, GUVectorize, vectorize, guvectorize
File "/home/jl/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/numba/np/ufunc/decorators.py", line 3, in
from numba.np.ufunc import _internal
SystemError: initialization of _internal failed without raising an exception
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 34725) of binary: /home/jl/anaconda3/envs/open-mmlab/bin/python
Traceback (most recent call last):
File "/home/jl/anaconda3/envs/open-mmlab/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/jl/anaconda3/envs/open-mmlab/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/jl/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/home/jl/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/home/jl/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/home/jl/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/distributed/run.py", line 752, in run
elastic_launch(
File "/home/jl/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/jl/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

./tools/test.py FAILED

Hi Author，
Could you please give me a detailed introduction on how to visualization?

look forward to your reply.

Can you share a visual script?
thanks

Thank you for your great work! I noticed that all 3d heads are forced to use fp32. Will there be performance drop if we use amp instead?

The result is awesome, I'm wondering when the code of FB-BEV will be release? thank you

Hi,

Does FBOCC support execution with TensorRT? When I try to run benchmark_trt.py, I encounter the following error:

$ python3 tools/analysis_tools/benchmark_trt.py ./occupancy_configs/fb_occ/fbocc-r50-cbgs_depth_16f_16x4_20e.py ckpts/fbocc-r50-cbgs_depth_16f_16x4_20e.pth
08/08 16:10:48 - mmengine - INFO - Successfully loaded tensorrt plugins from /home/blaize/work/fb-bev/fb-bev-venv/lib/python3.8/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
Traceback (most recent call last):
  File "tools/analysis_tools/benchmark_trt.py", line 190, in <module>
    fps = main()
  File "tools/analysis_tools/benchmark_trt.py", line 133, in main
    model = build_model(cfg.model, test_cfg=cfg.get('test_cfg'))
  File "/home/blaize/work/fb-bev/FB-BEV/mmdet3d/models/builder.py", line 122, in build_model
    return build_detector(cfg, train_cfg=train_cfg, test_cfg=test_cfg)
  File "/home/blaize/work/fb-bev/FB-BEV/mmdet3d/models/builder.py", line 95, in build_detector
    return MMDET_DETECTORS.build(
  File "/home/blaize/work/fb-bev/mmcv/mmcv/utils/registry.py", line 234, in build
    return self.build_func(*args, **kwargs, registry=self)
  File "/home/blaize/work/fb-bev/mmcv/mmcv/cnn/builder.py", line 27, in build_model_from_cfg
    return build_from_cfg(cfg, registry, default_args)
  File "/home/blaize/work/fb-bev/mmcv/mmcv/utils/registry.py", line 58, in build_from_cfg
    raise KeyError(
KeyError: 'FBOCCTRT is not in the models registry'

thank you.

Hello zhiqi. How to train a depth pretrained r50+depthnet that is used in your occupancy model?

Hello,
When trying to evaluate the model I get the following error:

ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 887) of binary: /opt/conda/bin/python
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/run.py", line 755, in run
    )(*cmd_args)
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent
    failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
./tools/test.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-01-16_18:10:10
  host      : adas-server
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 887)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

As far as I know, this error happens due to insufficient memory in a computer. Right now, I have around 45 GBs of memory available and am not able to increase it for the time being, but I read it is possible to avoid it by changing the batch size. Unfortunately, I cannot seem to figure out how exactly to do that.
Any help would be greatly appreciated!

I have installed a Conda environment with the dependencies as written on the install page. However, when I try to run a test I get these errors:

(openmmlab) root@xyz:/FB-BEV# python tools/test.py --config log/weights/fbocc-r50-cbgs_depth_16f_16x4_20e.py --checkpoint log/weights/fbocc-r50-cbgs_depth_16f_16x4_20e.pth --out test/test.pkl
Traceback (most recent call last):
File "tools/test.py", line 303, in
main()
File "tools/test.py", line 155, in main
cfg = Config.fromfile(args.config)
File "/opt/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/utils/config.py", line 340, in fromfile
cfg_dict, cfg_text = Config._file2dict(filename,
File "/opt/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/utils/config.py", line 208, in _file2dict
mod = import_module(temp_module_name)
File "/opt/anaconda3/envs/openmmlab/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 843, in exec_module
File "", line 219, in _call_with_frames_removed
File "/tmp/tmp2fp83puh/tmpcfk_b6pr.py", line 1, in
NameError: name 'false' is not defined
Exception ignored in: <function _TemporaryFileCloser.del at 0x7f251c1f88b0>
Traceback (most recent call last):
File "/opt/anaconda3/envs/openmmlab/lib/python3.8/tempfile.py", line 440, in del
self.close()
File "/opt/anaconda3/envs/openmmlab/lib/python3.8/tempfile.py", line 436, in close
unlink(self.name)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp2fp83puh/tmpcfk_b6pr.py'

Do you have any idea what the solution could be?
This is my pip list:

Package Version Editable project location

absl-py 1.4.0
addict 2.4.0
anyio 4.0.0
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.2.3
asttokens 2.4.0
async-lru 2.0.4
attrs 23.1.0
Babel 2.12.1
backcall 0.2.0
beautifulsoup4 4.12.2
black 23.9.1
bleach 6.0.0
cachetools 5.3.1
ccimport 0.4.2
certifi 2023.7.22
cffi 1.15.1
charset-normalizer 3.2.0
click 8.1.7
cmake 3.27.4.1
comm 0.1.4
contourpy 1.1.0
cumm-cu114 0.4.11
cycler 0.11.0
debugpy 1.7.0
decorator 5.1.1
defusedxml 0.7.1
descartes 1.1.0
exceptiongroup 1.1.3
executing 1.2.0
fastjsonschema 2.18.0
filelock 3.12.3
fire 0.5.0
flake8 6.1.0
fonttools 4.42.1
fqdn 1.5.1
google-auth 2.23.0
google-auth-oauthlib 1.0.0
grpcio 1.58.0
idna 3.4
imageio 2.31.3
importlib-metadata 6.8.0
importlib-resources 6.0.1
iniconfig 2.0.0
ipykernel 6.25.2
ipython 8.12.2
ipython-genutils 0.2.0
ipywidgets 8.1.0
isoduration 20.11.0
jedi 0.19.0
Jinja2 3.1.2
joblib 1.3.2
json5 0.9.14
jsonpointer 2.4
jsonschema 4.19.0
jsonschema-specifications 2023.7.1
jupyter 1.0.0
jupyter_client 8.3.1
jupyter-console 6.6.3
jupyter_core 5.3.1
jupyter-events 0.7.0
jupyter-lsp 2.2.0
jupyter_server 2.7.3
jupyter_server_terminals 0.4.4
jupyterlab 4.0.5
jupyterlab-pygments 0.2.2
jupyterlab_server 2.24.0
jupyterlab-widgets 3.0.8
kiwisolver 1.4.5
lark 1.1.7
lit 16.0.6
llvmlite 0.36.0
lyft-dataset-sdk 0.0.8
Markdown 3.4.4
MarkupSafe 2.1.3
matplotlib 3.5.2
matplotlib-inline 0.1.6
mccabe 0.7.0
mistune 3.0.1
mmcls 0.25.0
mmcv-full 1.5.2
mmdet 2.24.0
mmdet3d 1.0.0rc4 /FB-BEV
mmsegmentation 0.24.0
mpmath 1.3.0
mypy-extensions 1.0.0
nbclient 0.8.0
nbconvert 7.8.0
nbformat 5.9.2
nest-asyncio 1.5.7
networkx 2.2
ninja 1.11.1
notebook 7.0.3
notebook_shim 0.2.3
numba 0.53.0
numpy 1.20.3
nuscenes-devkit 1.1.10
nvidia-cublas-cu11 11.10.3.66
nvidia-cuda-cupti-cu11 11.7.101
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11 8.5.0.96
nvidia-cufft-cu11 10.9.0.58
nvidia-curand-cu11 10.2.10.91
nvidia-cusolver-cu11 11.4.0.1
nvidia-cusparse-cu11 11.7.4.91
nvidia-nccl-cu11 2.14.3
nvidia-nvtx-cu11 11.7.91
oauthlib 3.2.2
opencv-python 4.8.0.76
overrides 7.4.0
packaging 23.1
pandas 2.0.3
pandocfilters 1.5.0
parso 0.8.3
pathspec 0.11.2
pccm 0.4.8
pexpect 4.8.0
pickleshare 0.7.5
Pillow 10.0.0
pip 23.2.1
pkgutil_resolve_name 1.3.10
platformdirs 3.10.0
plotly 5.16.1
pluggy 1.3.0
plyfile 1.0.1
portalocker 2.7.0
prettytable 3.9.0
prometheus-client 0.17.1
prompt-toolkit 3.0.39
protobuf 4.24.3
psutil 5.9.5
ptyprocess 0.7.0
pure-eval 0.2.2
pyasn1 0.5.0
pyasn1-modules 0.3.0
pybind11 2.11.1
pycocotools 2.0.7
pycodestyle 2.11.0
pycparser 2.21
pyflakes 3.1.0
Pygments 2.16.1
pyparsing 3.1.1
pyquaternion 0.9.9
pytest 7.4.2
python-dateutil 2.8.2
python-json-logger 2.0.7
pytz 2023.3.post1
PyWavelets 1.4.1
PyYAML 6.0.1
pyzmq 25.1.1
qtconsole 5.4.4
QtPy 2.4.0
referencing 0.30.2
requests 2.31.0
requests-oauthlib 1.3.1
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rpds-py 0.10.2
rsa 4.9
scikit-image 0.19.3
scikit-learn 1.3.0
scipy 1.10.1
Send2Trash 1.8.2
setuptools 68.0.0
Shapely 1.8.5
six 1.16.0
sniffio 1.3.0
soupsieve 2.5
spconv-cu114 2.3.6
stack-data 0.6.2
sympy 1.12
tenacity 8.2.3
tensorboard 2.14.0
tensorboard-data-server 0.7.1
termcolor 2.3.0
terminado 0.17.1
terminaltables 3.1.10
threadpoolctl 3.2.0
tifffile 2023.7.10
tinycss2 1.2.1
tomli 2.0.1
torch 2.0.1
torchvision 0.15.2
tornado 6.3.3
tqdm 4.66.1
traitlets 5.9.0
trimesh 2.35.39
triton 2.0.0
typing_extensions 4.7.1
tzdata 2023.3
uri-template 1.3.0
urllib3 1.26.16
wcwidth 0.2.6
webcolors 1.13
webencodings 0.5.1
websocket-client 1.6.3
Werkzeug 2.3.7
wheel 0.38.4
widgetsnbextension 4.0.8
yapf 0.40.1
zipp 3.16.2

I have tried exporting the onnx file of FB-OCC, but I face the following error during tracing at the custom op of QuickCumsumCuda specifically when torch.onnx.export while the feed-forward inference of the model does not have any issue:

File "/FB-BEV/mmdet3d/ops/bev_pool_v2/bev_pool.py", line 102, in forward_dummy
    x = QuickCumsumCuda.apply(depth, feat, ranks_depth, ranks_feat, ranks_bev, bev_feat_shape, interval_starts, interval_lengths)
RuntimeError: _Map_base::at

This is how the error can be reproduced:

1. I had isolated the custom op QuickCumsumCuda in a separate class function as showin the following for the ease of reproducibility:

class Bev_Pool_v2(torch.nn.Module):
    def __init__(self):
        super(Bev_Pool_v2, self).__init__()
    def forward(self, depth, feat, ranks_depth, ranks_feat, ranks_bev, bev_feat_shape, interval_starts, interval_lengths):
        x = QuickCumsumCuda.apply(depth, feat, ranks_depth, ranks_feat, ranks_bev, bev_feat_shape, interval_starts, interval_lengths)
        x = x.permute(0, 4, 1, 2, 3).contiguous()
        return x
    def forward_dummy(self, data):
        depth, feat, ranks_depth, ranks_feat, ranks_bev, bev_feat_shape, interval_starts, interval_lengths = data
        x = QuickCumsumCuda.apply(depth, feat, ranks_depth, ranks_feat, ranks_bev, bev_feat_shape, interval_starts, interval_lengths)    
        x = x.permute(0, 4, 1, 2, 3).contiguous()
        return x

2. I generate/feed-forward the random inputs, which does not yield any issue during model inference.

# Random Generations of Inputs
depth = torch.rand(1, 6, 80, 16, 44).cuda()
feat = torch.rand(1, 6, 80, 16, 44).cuda()
ranks_depth = torch.randint(0, 337522, (206988, )).to(dtype=torch.int32).cuda()
ranks_feat = torch.randint(0, 4223, (206988, )).to(dtype=torch.int32).cuda()
ranks_bev = torch.randint(0, 79972, (206988, )).to(dtype=torch.int32).cuda()
bev_feat_shape = (1, 8, 100, 100, 80)
interval_starts = torch.randint(0, 79972, (52815, )).to(dtype=torch.int32).cuda()
interval_lengths = torch.randint(0, 213, (52815, )).to(dtype=torch.int32).cuda()

# Define the model and the input
model = Bev_Pool_v2().eval().cuda()
model.forward = model.forward_dummy
input_ = [depth, feat, ranks_depth, ranks_feat, ranks_bev, 
    bev_feat_shape, interval_starts, interval_lengths]

# Feed the input to the model
model(input_)
print('feed-forward inference is done without errors.')

3. Yet, the error mentioned above appears when exporting the model.

with torch.no_grad():
    torch.onnx.export(
        model,
        input_,
        'bev_pool_v2_USE.onnx',
        # export_params=True,
        keep_initializers_as_inputs=True,
        do_constant_folding=False,
        verbose=True,
        opset_version=13
    )

Despite exploring various solutions, I have yet to resolve this error.

Congratulations and thanks for sharing this great work!
I wonder when the code will be requested! Could you please estimate the release date?
Thanks

Following the install instructions, numpy>=1.24.0 will be installed at this point. Which is somehow incompatible with numba==0.53.0.
Install earlier numpy version can solve this problem.

The detail of the error is:

mmdet3d/datasets/samplers/infinite_group_each_sample_in_batch_sampler.py", line 174, 
assert self.groups_num >= self.global_batch_size

The error message is located on line 174, which belongs to the initialization function of the InfiniteGroupEachSampleInBatchSamplerEval.

and the print info of relative variable is:

batch_size :  1,
world_size: 1,
global_batch_size: 1,
groups_num: 2

How can I slove the problem,
Thanks!

May I know the detailed results of the FB-OCC on the occ task with InternImage-H backbone?
Actually, the Version-K in your report, could u provide the result without test-time augmentation?
This will be very helpful to me, thank you!

Dear Author,

Thank you for your great work and sharing.
I have followed all the preparation steps and run the code through python tools/train.py ./configs/occupancy/bevdet4d-r50-cbgs_depth_cool.py and I got the following error message: KeyError: 'NewBEV4D is not in the models registry'

Which config files should I run?

Thank you.

FB-OCC (39.1 mIoU)所有类别的IoU可以分享下吗？谢谢~

I would like to express my gratitude for your excellent research. I had always been curious about whether using both forward and backward projections for BEV representation would enhance its performance. I had contemplated taking this up as a research topic. I was particularly astounded by the improvement in performance when a module for depth was added during the backward projection.

One aspect I'm curious about is the experiment with ResNet-50 mentioned in your paper. From what I observed, the FLOPS are lower compared to when BEVDepth is reproduced. I anticipate that to achieve this, one cannot use the BEVBackbone + neck portion utilized in BEVDepth. Could you confirm if my assumption is correct?

Thank you.

Thanks for sharing this wonderful work.

I wonder why setting do_history=False in the configs. Does this mean we have not used the temporal fusion strategy? And I also have tried to set do_history=True and train the same setting, the experimental results are almost the same. I don't know why.

Could anyone help me?

Hello,

I tried to run the inference code and faced this error "ModuleNotFoundError: No module named 'spconv'"
although I did the installation steps successfully. I would appreciate your input for solving this issue.

Thank you in advance for your help!

Thanks for your excellent works!

When I test with dist_test.sh, the memory continues to increase, causing the evaluation to stop.
The same thing happens when fbocc.py's pred_occcupancy_category = pred_occcupancy_category.cpu().numpy() is performed without changing to cpu.
I wonder if there were any problems like this, do you know how to solve them?

Thanks for great work，how to use tools/ vis_occupancy.py to show the result? it looks like has bug. I don't know where the problem is.
the inputs shape is 200 * 200 * 18

Anyone else having problems with the evaluation?

my results are really bad:
others,barrier,bicycle,bus,car,construction_vehicle,motorcycle,pedestrian,traffic_cone,trailer,truck,driveable_surface,other_flat,sidewalk,terrain,manmade,vegetation,Overall
0.0,0.01,0.0,0.0,0.0,0.04,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,3.78,0.0,0.23

I had to upgrade the mmcv-full to 1.6.0 because of the following error:

File "./tools/test.py", line 303, in <module>
    main()
  File "./tools/test.py", line 272, in main
    outputs = custom_multi_gpu_test(model, data_loader, args.tmpdir,
  File "/FB-BEV/mmdet3d/apis/test.py", line 164, in custom_multi_gpu_test
    result = model(return_loss=False, rescale=True, **data)
  File "/root/miniconda3/envs/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/miniconda3/envs/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/root/miniconda3/envs/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 969, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])
  File "/root/miniconda3/envs/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/miniconda3/envs/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 140, in new_func
    output = old_func(*new_args, **new_kwargs)
  File "/FB-BEV/mmdet3d/models/detectors/base.py", line 62, in forward
    return self.forward_test(**kwargs)
  File "/FB-BEV/mmdet3d/models/fbbev/detectors/fbocc.py", line 487, in forward_test
    if num_augs==1 and not img_metas[0][0
].get('tta_config', dict(dist_tta=False))['dist_tta']:
TypeError: 'DataContainer' object is not subscriptable

Here is my Dockerfile, maybe it helps:
Dockerfile.txt

感觉你的深度估计模块有点low ，把深度值作为类别搞感觉有问题，有没有类似heatmap那个有个突出点的

File "FB-BEV/mmdet3d/models/fbbev/detectors/fbocc.py", line 486, in forward_test
if num_augs==1 and not img_metas[0][0].get('tta_config', dict(dist_tta=False))['dist_tta']:
TypeError: 'DataContainer' object is not subscriptable

请问【LoadBEVMask】 具体是如何实现的

gt-mask画在128 x 128 的bev网格时，由于small-gt 在画mask时需要将float类型的bbox取整，出现偏移的问题，这个偏移对小目标而言是较大的。

请问作者是如何克服这个问题的。

有没有大佬知道，在测试的时候，有一个/data/nuscenes/gts/scene-0103/{sample_token}/labels.npz从哪获取啊？
如图片中加亮区域所示。

Does anyone know where to get the /data/nuscenes/gts/scene-0103/{sample_token}/labels.npz during testing? As shown in the highlighted area of the picture.

what does "fix_void" mean? In config, fix_void=num_class==19. And in model pred_occupancy = pred_occupancy[..., 1:] when fix_void is True. So what does it mean?

很高兴你们作出这样的工作，我在看文章时看到When using the default settings of LSS on the nuScenes dataset, only 50% of the grids can receive valid image features through projection，可否告知这个50%是怎么推算出来的？

Hi, I want to why we need to apply the below transformations in LoadOccupancy function?

# to BEVDet format
occupancy = occupancy.permute(2, 0, 1)  # to (16, 200, 200)
occupancy = torch.rot90(occupancy, 1, [1, 2])
occupancy = torch.flip(occupancy, [1])
occupancy = occupancy.permute(1, 2, 0)

Although the comment says they are used to transform the occupancy to BEVDet format, they are not exist in the BEVDet codebase.

Can anybody help to understand the purposes of these operations?