Hi Owen,

Great work! Thanks for uploading the code and providing very clear instructions to run it.

I have two issues that I wanted to ask -

First, I noticed that at the above line dt_annos and gt_annos params in the calculate_iou_partly function are swapped. I am not sure if it matters because IoU operation is commutative.

Second, I ran training for Mono3D with the example config provided in the repo. I was trying to reproduce the results but I am always getting results which are not similar to the expected results on the validation set.

Here are my results.

Car AP(Average Precision)@0.70, 0.70, 0.70:          
bbox AP:83.29, 70.23, 54.05                          
bev  AP:10.64, 8.42, 6.52  
3d   AP:6.76, 5.10, 3.73   
aos  AP:82.75, 69.28, 53.32                          
Car AP(Average Precision)@0.70, 0.50, 0.50:          
bbox AP:83.29, 70.23, 54.05                          
bev  AP:43.14, 32.86, 25.95                          
3d   AP:37.96, 27.91, 22.64                          
aos  AP:82.75, 69.28, 53.32                          
                                                     
/**** finish testing after training epoch 19 ******/ 

Car AP(Average Precision)@0.70, 0.70, 0.70:          
bbox AP:82.41, 67.77, 51.65                          
bev  AP:9.56, 6.98, 5.34                             
3d   AP:5.48, 3.93, 3.19                             
aos  AP:81.98, 66.96, 51.00
Car AP(Average Precision)@0.70, 0.50, 0.50:          
bbox AP:82.41, 67.77, 51.65
bev  AP:42.07, 32.54, 25.82                          
3d   AP:36.23, 27.81, 21.56
aos  AP:81.98, 66.96, 51.00                          
                             
/**** finish testing after training epoch 29 ******/ 

Here are my training steps -

1. Clone the repo and do make.sh.
2. Download image_2, image_3, calib and label_2 from Kitti official website and unzip them.
3. Do the exact steps from the Mono3D readme. While copying the config file. I just changed the paths to point to my directories and didn't alter any of the existing hyperparameters.
4. I used single 2080Ti gpu for training and it takes about 6-7 hours for 30 epochs to run.

Can you please tell me what I am doing wrong here?

Thanks!

Thanks for your code!

I want to inference the real time data which is from camera.

Do you have inference code ?

Thanks

Hi, Owen
I want to use your model to predict depth in a single image which is captured by a camera. I want to know the distance error rate(for example 50m position). I wonder how to test this.

Hi, thanks for your code it is amazing.
I want to mention that visualize_test_center.ipynb is missing a plt.show() in the end. Otherwise the image won't pop up just like in the jupyter notebook.

Hi,thanks for you nice work!
I've run sh ./make.sh and the program have created a iou3d_cuda.cpython-37m-x86_64-linux-gnu.so document.
But when I run sh ./launchers/det_precompute.sh config/Yolo3D_example.py train ,it turns out a traceback saying */iou3d/iou3d_cuda.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _Z17nmsNormalLauncherPKfPyif.
Could you please tell me how to solve this problem?Thank you!

I find the detailed detection results on web page : https://github.com/Owen-Liuyuxuan/visualDet3D/releases/tag/1.1. I wander to which data split is the detection results based on. Thank you.

Hi,

I am trying to run the validation but facing the following error. How can I solve it?

您好！模型测试时会使用两种标准，例如对于3D easy case， 第一种是21.90，第二种是57.11。为什么第二种比第一种性能高这么多？对于easy case， overlap的阈值都是0.7吗？
Car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:97.29, 84.61, 64.70
bev AP:29.40, 20.19, 15.59
3d AP:21.90, 14.54, 11.42
aos AP:96.11, 81.54, 62.28

Car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:97.29, 84.61, 64.70
bev AP:63.65, 44.70, 34.33
3d AP:57.11, 39.35, 30.90
aos AP:96.11, 81.54, 62.28

大佬你好，请问可以提供一下demo推理3d框的代码吗？感谢！

hi. Thank you for your work.
I use train, val splits below files.

train.txt
val.txt
train = 3712 / val = 3769

But when trained using Yolo3D_example, displayed in the terminal:

clean up the recorder directory of /home/user/repository/visualDet3D-master/exp/Mono3D/log/defaultconfig=/home/user/repository/visualDet3D-master/config/config.py
-1
number of trained parameters of the model: 58388994
Found evaluate function evaluate_kitti_obj
Num training images: 7424
Epoch: 0 | Iteration: 49 | Running loss: 0.84490 | eta:8.21h

Num training images: 7424 is fine?

+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
And i run imdb_precompute_3d.py, I got the result:

start reading training data
training split finished precomputing02s, eta:0.24s, total_objs:[10698], usable_objs:[10054]
start reading validation data
validation split finished precomputing00s, eta:0.01s, total_objs:[0], usable_objs:[0]
Preprocessing finished

this is fine? no vaildation objects -_-
myconfig.txt
<- my config.py
thx.

HELLO, Good job.
Is there any pre-trained model on KITTI depth that i can directly use to infer on a single image？

Dear authors,

thank you very much for your repository.
I would like to ask whether it would be possible for you to release the model from paper "Ground-aware Monocular 3D Object Detection for Autonomous Driving" on the training set only such that this model can be evaluated on the validation set.
Do you think that it would be also possible to release the codes that would reproduce the results of this model?

Also, would it work to train the GAC model for multi-class detection? Have you tried it?

Thank you very much in advance.

Thanks for the great work!
I tried to followed the instruction in the docs and train.
But I got the Error below

Precomputation for the training/validation split
Traceback (most recent call last):
File "scripts/imdb_precompute_3d.py", line 15, in
from visualDet3D.data.kitti.utils import generate_dispariy_from_velo
ImportError: cannot import name 'generate_dispariy_from_velo' from 'visualDet3D.data.kitti.utils' (/data/10_3D_Detection/visualDet3D/visualDet3D/data/kitti/utils.py)

I checked visualDet3D/data/kitti/utils.py, but found no 'generate_dispariy_from_velo'. Is it missing?

Thanks!

Hi, thanks for the code!
I am trying to follow your "visualize_test_3d_stereo.ipynb" demo, but when I run the first cell, it returns error "FileNotFoundError: [Errno 2] No such file or directory: '../config/kitti_stereo.py'"

I have checked the config folder, there is no kitti_stereo.py inside it, can you please share the file?
Please advise, thank you in advanced.

Regards,
Sona

Is there a parenthesis missing in the code.
y_shifts_base = F.relu( h_mean * (yy_grid - cy) / 2 * (self.relative_elevation - 0.5 * h_mean) ) / (yy_grid.shape[1] * 0.5) # [1, H, W]
https://github.com/Owen-Liuyuxuan/visualDet3D/blob/master/visualDet3D/networks/lib/look_ground.py

Precomputation for the training/validation split
Traceback (most recent call last):
File "scripts/imdb_precompute_3d.py", line 11, in
from visualDet3D.networks.heads.anchors import Anchors
File "/content/drive/MyDrive/3_Aug/visualDet3D/visualDet3D/networks/init.py", line 1, in
from .pipelines import *
File "/content/drive/MyDrive/3_Aug/visualDet3D/visualDet3D/networks/pipelines/init.py", line 3, in
from .evaluators import evaluate_kitti_obj
File "/content/drive/MyDrive/3_Aug/visualDet3D/visualDet3D/networks/pipelines/evaluators.py", line 16, in
from visualDet3D.data.kitti.utils import write_result_to_file
File "/content/drive/MyDrive/3_Aug/visualDet3D/visualDet3D/data/init.py", line 1, in
from .kitti.dataset import mono_dataset, depth_mono_dataset
File "/content/drive/MyDrive/3_Aug/visualDet3D/visualDet3D/data/kitti/init.py", line 1, in
from .dataset import KittiMonoDataset, KittiMonoTestDataset
File "/content/drive/MyDrive/3_Aug/visualDet3D/visualDet3D/data/kitti/dataset/init.py", line 4, in
from .KM3D_dataset import KittiRTM3DDataset
File "/content/drive/MyDrive/3_Aug/visualDet3D/visualDet3D/data/kitti/dataset/KM3D_dataset.py", line 19, in
from visualDet3D.networks.utils.rtm3d_utils import gen_hm_radius, project_to_image, gaussian_radius
File "/content/drive/MyDrive/3_Aug/visualDet3D/visualDet3D/networks/utils/rtm3d_utils.py", line 7, in
from visualDet3D.networks.lib.ops.iou3d.iou3d import boxes_iou3d_gpu
File "/content/drive/MyDrive/3_Aug/visualDet3D/visualDet3D/networks/lib/ops/init.py", line 1, in
from .dcn.deform_conv import ModulatedDeformConvPack, DeformConvPack
File "/content/drive/MyDrive/3_Aug/visualDet3D/visualDet3D/networks/lib/ops/dcn/deform_conv.py", line 50, in
from . import deform_conv_ext
ImportError: cannot import name 'deform_conv_ext' from 'visualDet3D.networks.lib.ops.dcn' (/content/drive/MyDrive/3_Aug/visualDet3D/visualDet3D/networks/lib/ops/dcn/init.py)

How can I get my prediction results on the test data? I try to run the testdata with eval.sh but it shows that there is no label. How can I solve it?

For the Car class, The validation results are:
bev AP: 73.51 48.49 37.47
3D AP: 62.28 39.16 30.51
Do you computing the disparity map using the point cloud or openCV BlockMatching?

Hi, thanks for your code first!

I got some question about the running time.
When I ran the test script and measured the inference time, it was about 0.18s per frame, much slower than 0.08s mentioned in the paper of YOLOStereo3D.
The GPU device I use is TITAN X (CUDA version 9.0) which has similar efficiency with 1080 Ti (I guess?).
The way I test is simply recording the start and end time and then computing the average time:

# visualDet3D/networks/pipelines/evaluators.py/evaluate_kitti_obj()
# inference on each sample in [dataset]
time_start = time.time()
for index in tqdm(range(len(dataset_val))):
    test_one(cfg, index, dataset_val, model, test_func, backprojector, projector, result_path)
time_end = time.time()
print('total inference time: ', time_end - time_start, ' s.')
print('average time: ', (time_end - time_start) / len(dataset_val))

I wonder if the way I test is correct. How did you test the running time? Could you please afford some scripts? Or maybe it's just because my device is slower?

Appreciate!

envs:
os: ubuntu18.04
cuda: nvcc 10.1
python: 3.7.10
torch: 1.8.1
tensorflow: 2.5.0

When run ./make.sh
error occured as follow:

(ground_aware) root@3b95744b2842:/zyf/code/visualDet3D# ./make.sh
/zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn /zyf/code/visualDet3D
running build_ext
building '..deform_conv_ext' extension
creating /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build
creating /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7
creating /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/src
creating /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/src/cuda
Emitting ninja build file /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/src/cuda/deform_conv_cuda_kernel.o.d -DWITH_CUDA -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/ground_aware/include/python3.7m -c -c /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/src/cuda/deform_conv_cuda_kernel.cu -o /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/src/cuda/deform_conv_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=deform_conv_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 -std=c++14
FAILED: /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/src/cuda/deform_conv_cuda_kernel.o
/usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/src/cuda/deform_conv_cuda_kernel.o.d -DWITH_CUDA -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/ground_aware/include/python3.7m -c -c /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/src/cuda/deform_conv_cuda_kernel.cu -o /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/src/cuda/deform_conv_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=deform_conv_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 -std=c++14
nvcc fatal : Unknown option '-generate-dependencies-with-compile'
[2/3] c++ -MMD -MF /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/src/cuda/deform_conv_cuda.o.d -pthread -B /root/anaconda3/envs/ground_aware/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/cuda/include -fPIC -DWITH_CUDA -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/ground_aware/include/python3.7m -c -c /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/src/cuda/deform_conv_cuda.cpp -o /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/src/cuda/deform_conv_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=deform_conv_ext -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/ATen/Parallel.h:140:0,
from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:13,
from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
from /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/src/cuda/deform_conv_cuda.cpp:4:
/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/ATen/ParallelOpenMP.h:83:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
#pragma omp parallel for if ((end - begin) >= grain_size)

[3/3] c++ -MMD -MF /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/src/deform_conv_ext.o.d -pthread -B /root/anaconda3/envs/ground_aware/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/cuda/include -fPIC -DWITH_CUDA -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/ground_aware/include/python3.7m -c -c /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/src/deform_conv_ext.cpp -o /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/build/temp.linux-x86_64-3.7/src/deform_conv_ext.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=deform_conv_ext -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/ATen/Parallel.h:140:0,
from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:13,
from /root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
from /zyf/code/visualDet3D/visualDet3D/networks/lib/ops/dcn/src/deform_conv_ext.cpp:4:
/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/include/ATen/ParallelOpenMP.h:83:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
#pragma omp parallel for if ((end - begin) >= grain_size)

ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1673, in _run_ninja_build
env=env)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "setup.py", line 198, in
zip_safe=False)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/setuptools/init.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/root/anaconda3/envs/ground_aware/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 79, in run
_build_ext.run(self)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 708, in build_extensions
build_ext.build_extensions(self)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/root/anaconda3/envs/ground_aware/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
_build_ext.build_extension(self, ext)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
depends=ext.depends)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 538, in unix_wrap_ninja_compile
with_cuda=with_cuda)
File "/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1359, in _write_ninja_file_and_compile_objects
error_prefix='Error compiling objects for extension')
File "/root/anaconda3/envs/ground_aware/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1683, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension

Hello,
I'd like to ask whether the network is pre-trained on a depth dataset.
Thanks.

Hi, I just get some error when I ran mono3d.

In this step,

Compute image database and anchors mean/std

You can run ./launcher/det_precompute.sh without arguments to see helper documents

./launcher/det_precompute.sh config/$CONFIG_FILE.py train
./launcher/det_precompute.sh config/$CONFIG_FILE.py test # only for upload testing

I got a message that :

Traceback (most recent call last):
File "scripts/imdb_precompute_3d.py", line 11, in
from visualDet3D.networks.heads.anchors import Anchors
File "/home/cv1/visualDet3D/visualDet3D/networks/init.py", line 1, in
from .pipelines import *
File "/home/cv1/visualDet3D/visualDet3D/networks/pipelines/init.py", line 3, in
from .evaluators import evaluate_kitti_obj
File "/home/cv1/visualDet3D/visualDet3D/networks/pipelines/evaluators.py", line 16, in
from visualDet3D.data.kitti.utils import write_result_to_file
File "/home/cv1/visualDet3D/visualDet3D/data/init.py", line 1, in
from .kitti.dataset import mono_dataset, depth_mono_dataset
File "/home/cv1/visualDet3D/visualDet3D/data/kitti/init.py", line 1, in
from .dataset import KittiMonoDataset, KittiMonoTestDataset
File "/home/cv1/visualDet3D/visualDet3D/data/kitti/dataset/init.py", line 4, in
from .KM3D_dataset import KittiRTM3DDataset
File "/home/cv1/visualDet3D/visualDet3D/data/kitti/dataset/KM3D_dataset.py", line 19, in
from visualDet3D.networks.utils.rtm3d_utils import gen_hm_radius, project_to_image, gaussian_radius
File "/home/cv1/visualDet3D/visualDet3D/networks/utils/rtm3d_utils.py", line 6, in
from visualDet3D.networks.lib.ops.iou3d.iou3d import boxes_iou3d_gpu
File "/home/cv1/visualDet3D/visualDet3D/networks/lib/ops/init.py", line 1, in
from .dcn.deform_conv import ModulatedDeformConvPack, DeformConvPack
File "/home/cv1/visualDet3D/visualDet3D/networks/lib/ops/dcn/deform_conv.py", line 50, in
from . import deform_conv_ext

ImportError: cannot import name 'deform_conv_ext' from 'visualDet3D.networks.lib.ops.dcn' (/home/cv1/visualDet3D/visualDet3D/networks/lib/ops/dcn/init.py)

So, How can I fix that error?

Checking the structure of the dataset, the training folder has a label.
But there is no label in the test folder.

I wonder how you used the test data (in what way you obtained the figures for the test data).

Thanks for your works.

Hi， I want to know the val results on recall11 or recall 40? Thank you for your great job!

Hello, LYX!
Thanks for your great work in mono3d detection. As in your paper, your work also gets competitative performance in ped/cyc classes. Can you release the config setting file of these two kind categories? Or can you tell me what should i rectify in your yolo3d_example to fit ped/cyc class? Thanks a lot!

用kitti数据和相机内参正常，换成自己的数据，图片大小进行crop，中心点保持不变，相机参数改成自己的内参，检测到的3D框都是上翘的有什么解决办法吗？

When I run ./launchers/train.sh config/monodepth_cfp.py 6 mono1, it meets error as below:

FileNotFoundError: [Errno 2] No such file or directory: '/data/sj_data/logdir/visualDet3D/MonoDepth/output/training/log_depth_sum.npy'

Maybe there is something missing in the "visualDet3D/docs/monoDepth.md"?
I guess it needs to run python scripts/depth_precompute.py config/monodepth_cfg.py 6 before training or testing?

Thanks for sharing the excellent work and the relevant code!

My question is:

After my own experiments and some slight changes, I found the performance (3D IOU) of the model was pretty bad if I directly regress the coordinate of the 3D bbox centre. However, the 2D Iou was better and reached 90% in the first few epochs. I am really confused about this result. Have you tried to directly regress these parameters? Do you have any idea about it?

These results are generated by resnet18 on Chen Split.

Evaluation result of regressing alpha
Car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:88.38, 71.46, 64.18
bev AP:8.49, 5.02, 4.39
3d AP:2.49, 1.81, 1.51
aos AP:88.33, 71.40, 64.11
Car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:88.38, 71.46, 64.18
bev AP:51.12, 37.06, 34.06
3d AP:31.00, 20.17, 19.07
aos AP:88.33, 71.40, 64.11

Evaluation result of regressing x, y, z and theta
Car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:96.50, 80.83, 63.48
bev AP:0.75, 0.43, 0.38
3d AP:0.22, 0.21, 0.08
Car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:96.50, 80.83, 63.48
bev AP:6.38, 4.43, 3.37
3d AP:4.52, 3.50, 2.40

Also, I noticed that the result on the validation set is pretty good using the full model. But it seems that there is a huge performance gap between the validation set and the test set on the KITTI server. So is it right that currently, the main problem is overfitting?

Hi, thanks for your great work!
I have trained GroundAwareYolo3D model and get results as below:
Car AP(Average Precision)@0.70, 0.70, 0.70
bbox AP: 97.29, 84.55, 64.65
bev AP: 29.53, 20.15, 15.53
3d AP: 22.90, 15.26, 11.33
aos AP: 96.52, 82.52, 63.05

seems comparable with paper report (23.63 16.16 12.06) in Car [email protected] validation set.

However if training with multi-gpu e.g. 4-GPU, we get poor result as below:
Car AP(Average Precision)@0.70, 0.70, 0.70
bbox AP: 97.08, 86.41, 66.67
bev AP: 20.56, 15.16, 11.22
3d AP: 15.17, 10.81, 8.22
aos AP: 95.50, 83.36, 64.24

training command:
bash ./launchers/train.sh config/$CONFIG_FILE.py 0,1,2,3 multi-gpu-train
bash ./launchers/train.sh config/$CONFIG_FILE.py 0 single-gpu-train

I trained twice with 'multi-gpu' and both results are similar and lower than 'single-gpu', so do you have some suggestions about this case? What about your multi-gpu training performance?

In detection_3d_head.py line 468 and 469, you use undefined variables: anchors_3d_mean_std;
I guess it should be anchor_mean_std_3d_j

您好！ 请问anchor filtering这部分的代码主要在哪个文件？ 非常感谢！

Could you provide your results in each config?
Thanks for your work. Because my results using your code is lower than that in papers. I would appreciate if you provide the results for each config. And if it's possible, could you provide a docker for easily install(some machine can't install well, due to conflict among packages?)

And my results as follows: (It seems something wrong in Yolo3D_example and KM3D_example)
In Ground-aware:)

Car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:91.76, 79.74, 62.81
bev  AP:22.39, 17.37, 13.54   
3d   AP:16.80, 12.73, 10.22    <----    In paper: 22.16 | 15.71 | 11.75
aos  AP:90.96, 78.19, 61.48
Car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:91.76, 79.74, 62.81
bev  AP:56.30, 41.20, 32.98
3d   AP:51.48, 37.65, 29.91
aos  AP:90.96, 78.19, 61.48

In monoflex:

Car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:97.04, 91.58, 81.64
bev  AP:30.90, 22.72, 19.22
3d   AP:22.91, 16.49, 13.59    <----    In paper:   23.64 | 17.51 | 14.83
aos  AP:96.92, 91.25, 81.16
Car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:97.04, 91.58, 81.64
bev  AP:66.93, 49.88, 43.21
3d   AP:62.03, 46.13, 39.80
aos  AP:96.92, 91.25, 81.16

In RTM3d:

Car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:96.98, 88.85, 78.72
bev  AP:16.05, 12.08, 9.98    <----    In paper:  27.83 | 23.38 | 21.69
3d   AP:10.20, 7.86, 6.26      <----    In paper:   22.50 | 19.60 | 17.12
aos  AP:96.45, 87.73, 77.50
Car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:96.98, 88.85, 78.72
bev  AP:50.02, 37.08, 30.29
3d   AP:43.69, 32.07, 26.78
aos  AP:96.45, 87.73, 77.50

Dear authors,

thank you very much for your repository. I would like to ask whether you use the ResNet-101 backbone that is initialized from ImageNet or from scratch?

Thank you very much in advance.

Why not using a denser disparity supervision(converted from lidar points) like PseudoLidar(++)?

I created synthetic data using by Carla simulator(Unreal Engine based).
And I transformed the data to Kitti data format.

Then, I faced the problem that

./launchers/det_precompute.sh config/mono3d.py train

Error :
Precomputation for the training/validation split
start reading training data
Traceback (most recent call last):
File "scripts/imdb_precompute_3d.py", line 213, in
Fire(main)
File "/home/cv1/miniconda3/envs/yolostereo/lib/python3.7/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/cv1/miniconda3/envs/yolostereo/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire
target=component.name)
File "/home/cv1/miniconda3/envs/yolostereo/lib/python3.7/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "scripts/imdb_precompute_3d.py", line 200, in main
read_one_split(cfg, train_names, data_root_dir, output_dict, 'training', time_display_inter)
File "scripts/imdb_precompute_3d.py", line 103, in read_one_split
image, P2, label = preprocess(original_image, p2=deepcopy(calib.P2), labels=deepcopy(data_frame.label))
File "/home/cv1/visualDet3D/visualDet3D/data/pipeline/augmentation_builder.py", line 49, in call
left_image, right_image, p2, p3, labels, image_gt, lidar = t(left_image, right_image, p2, p3, labels, image_gt, lidar)
File "/home/cv1/visualDet3D/visualDet3D/data/pipeline/stereo_augmentator.py", line 50, in call
left_image -= np.tile(self.mean, int(left_image.shape[2]/self.mean.shape[0]))
ValueError: operands could not be broadcast together with shapes (288,1280,4) (3,) (288,1280,4)

I think that problem is my data format.
I guessed the Expected Data format is (n,m,3) but my data format is (n,m,4).
Is my guess right?

Thanks your kind.

hi, Thanks for your aswesome job! And I am confused about the my eval results on kitti val datasets.
I trained YOLO3D_example with 8 2080TI， and the val results shows as fellows
epoch 10:
Car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:96.80, 83.32, 63.83
bev AP:15.41, 11.21, 8.66
3d AP:9.61, 7.17, 5.31
aos AP:93.90, 78.78, 60.36
Car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:96.80, 83.32, 63.83
bev AP:47.02, 33.31, 26.05
3d AP:39.81, 28.13, 21.42
aos AP:93.90, 78.78, 60.36
/**** finish testing after training epoch 9 ******/

epoch 20:
Car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:96.81, 85.85, 66.29
bev AP:21.13, 15.52, 12.39
3d AP:14.05, 10.35, 7.51
aos AP:94.29, 81.57, 62.97
Car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:96.81, 85.85, 66.29
bev AP:58.15, 41.16, 31.43
3d AP:51.35, 37.29, 28.19
aos AP:94.29, 81.57, 62.97
/**** finish testing after training epoch 19 ******/

epoch 30:
Car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:98.05, 84.85, 67.62
bev AP:12.92, 10.52, 7.81
3d AP:7.76, 6.14, 4.58
aos AP:95.21, 80.50, 63.99
Car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:98.05, 84.85, 67.62
bev AP:52.92, 38.00, 29.05
3d AP:45.49, 32.40, 25.20
aos AP:95.21, 80.50, 63.99
/**** finish testing after training epoch 29 ******/

It seems that epoth_20 is much better than epoch_30 in 3D detection. Could you tell me the reason. many thaks

Hello, author, when reproducing the code, I got the following results with regard to the evaluation indicators in simple, medium, and difficult cases. Why does the detection result on pedestrians appear to have higher evaluation indicators in simple cases? , Hope to get your answer, thank you.

Hello!

Awsome work on the theory, I aslo tried out the repo, works well. My problem is that I can't seem the find guidance on visualizing the 3D bounding box results. Can you give me some hints on this issue? I found the output txt files, and a draw_3D_box() function, but it requires corners and I'm not sure about the whole process.

Thanks in advance!

Hi Liu

I applied weight pruning on pretrained detector.core.backbone(resnet34) after training 80 epochs.

The process was training -> pruning -> finetuning.

However when I ran the eval.py.

The result was like this below

Can you guess what's the problem?

The result after 59 epochs training was like this below

Thanks

Hi LIU

I appreciate for your work.

I am curious about the results of validation set.

I got these results while training. Would you please explain the results of the image above?

Thanks

Hello, could you share the forluma derivation process or related reference? thanks .

Can this model export to onnx format?

Hi
Thanks to your code, I can learn many things. Thank you.

What I'm curious about is changing data.
I want to do the training only left image and the test data right image in a monocular 3D

Which part should I modify to execute the code?
I'll be waiting for the reply.
Thank you.

Hi, I want to test with a picture which size is 1280*720. Does the test support this image size? Thanks a lot!!!!

Is it possible to modify the code such that I can use my own dataset that has the same structure as the KITTI dataset? In addition, can I make the calibration files match with my setup if I only have two cameras and no need for Lidar or velodyne.

用kitti数据和相机内参正常，换成自己的数据，图片大小进行crop，中心点保持不变，相机参数改成自己的内参，检测到的3D框都是上翘的有什么解决办法吗？

@Owen-Liuyuxuan 你好，我在测试只有Car一个类别的时候没有问题，然后我想测试3个类别，就按照你上面的提示，在Yolo3D_example.py文件中修改了对应的三行代码，然后报错如下：

(yolov4) shl@zhihui-mint:~/Disk2/3_proj/visualDet3D$ ./launchers/eval.sh config/Yolo3D_example.py 0 workdirs/Mono3D/checkpoint/GroundAware_pretrained.pth test
CUDA available: True
Traceback (most recent call last):
  File "scripts/eval.py", line 55, in <module>
    fire.Fire(main)
  File "/home/shl/anaconda3/envs/yolov4/lib/python3.6/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/shl/anaconda3/envs/yolov4/lib/python3.6/site-packages/fire/core.py", line 471, in _Fire
    target=component.__name__)
  File "/home/shl/anaconda3/envs/yolov4/lib/python3.6/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "scripts/eval.py", line 37, in main
    detector = DETECTOR_DICT[cfg.detector.name](cfg.detector)
  File "/media/shl/2b7567a9-ed41-40af-8673-4d47b46a0f32/home/zhihui/Disk2/3_proj/visualDet3D/visualDet3D/networks/detectors/yolomono3d_detector.py", line 65, in __init__
    self.build_head(network_cfg)
  File "/media/shl/2b7567a9-ed41-40af-8673-4d47b46a0f32/home/zhihui/Disk2/3_proj/visualDet3D/visualDet3D/networks/detectors/yolomono3d_detector.py", line 137, in build_head
    **(network_cfg.head)
  File "/media/shl/2b7567a9-ed41-40af-8673-4d47b46a0f32/home/zhihui/Disk2/3_proj/visualDet3D/visualDet3D/networks/heads/detection_3d_head.py", line 32, in __init__
    self.anchors = Anchors(preprocessed_path=preprocessed_path, readConfigFile=read_precompute_anchor, **anchors_cfg)
  File "/media/shl/2b7567a9-ed41-40af-8673-4d47b46a0f32/home/zhihui/Disk2/3_proj/visualDet3D/visualDet3D/networks/heads/anchors.py", line 36, in __init__
    self.anchors_mean_original[i]  = np.load(npy_file) #[30, 2, 6] #[z,  sinalpha, cosalpha, w, h, l,]
ValueError: could not broadcast input array from shape (16,2,6) into shape (16,3,6)
(yolov4) shl@zhihui-mint:~/Disk2/3_proj/visualDet3D$ 

请问这个错误是需要修改anchor_mean_Car.npy和anchor_std_Car.npy 这两个文件吗，我目前不知道怎么解决这个问题，还希望你可以抽空帮忙一下，谢谢啦

Originally posted by @shliang0603 in #17 (comment)

Hi,

Thanks for providing the code. I have a question about the multi-bin classification loss. In the Mono3D paper, it is mentioned that the cross-entropy loss is used for the multi-bin classification of width, height, and length.

However, looking at the code in detection_3d_head.py, I find that it is being used only for angle alpha. Am I understanding the code wrong? Can you please clarify that?

Thank you