andreibarsan / dynslam Goto Github PK

Have you considered make this a ROS node so that depth, and segment classification can be integrated via ROS messages. This would allow switching between different implementation of these components, and would be language agnostic.

This issue keeps track of a series of issues on quantitative evaluation:

• LIDAR baseline implementation: reconstruct sequence, and then "rerun" through it comparing every frame with what the LIDAR saw; is this cheating because we're using future information? We could do it on the fly as well. (Implemented on the fly.)
• Reconstruction quality experiments
  • Show perf on static map (ELAS vs. DispNet), comparing against LIDAR
  • Show perf on dynamic scene with dynamic mode on or off (ELAS vs. DispNet)
  • Show results on first, say, 1.5k (or maybe all?) frames of all 10 kitti-odometry sequences; can't easily compare against other sequences because there are no corrected LIDAR pointclouds to use as GT...
• Memory usage and quality with decay on/off, and different decay weights.
  • Compare ELAS vs DispNet (should e.g., have higher impact on DispNet because of its noisier results)
• Tracklet experiments

• Show performance of just RANSAC vs. RANSAC+DA on sequences from object tracking bench (we need the 3D box ground truth so we can compute their relative poses).

• ATE if doing odometry
  • (Low-pri) but I could try to remove SF vectors from dyn objects before computing libviso2 VO and see if overall ATE is lower or not. Dunno yet. Was not the main focus of the thesis, so I focused on the other experiments.

It can be wasteful to allocate a HUGE hash map for tiny reconstruction.
However this refactor is nontrivial because the SDF_BUCKET_NUM define is referenced all over the place in the codebase...

This can be very useful in the writeup.

I got a fatal build error as the following:

~/DynSLAM/build$ make
Scanning dependencies of target InstRecLib
[ 0%] Building CXX object src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/SegmentationDataset.cpp.o
[ 1%] Building CXX object src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/SparseSFProvider.cpp.o
[ 2%] Building CXX object src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/InstanceTracker.cpp.o
In file included from /home/tutu/DynSLAM/src/DynSLAM/InstRecLib/Track.h:7:0,
from /home/tutu/DynSLAM/src/DynSLAM/InstRecLib/InstanceTracker.h:13,
from /home/tutu/DynSLAM/src/DynSLAM/InstRecLib/InstanceTracker.cpp:3:
/home/tutu/DynSLAM/src/DynSLAM/InstRecLib/../InfiniTamDriver.h:11:27: fatal error: gflags/gflags.h: No such file or directory
compilation terminated.
src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/build.make:110: recipe for target 'src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/InstanceTracker.cpp.o' failed
make[2]: *** [src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/InstanceTracker.cpp.o] Error 1
CMakeFiles/Makefile2:1247: recipe for target 'src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/all' failed
make[1]: *** [src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

This part seems to cause issues for lots of people. I should apply the fix posted by people in issues such as #33.

After working on a lot of CUDA-specific stuff, I disabled most of the CPU map reconstruction code. It shouldn't be too difficult to make it run again (low-priority, though).

I was getting the following bug

Track.cpp:73:30: error: ‘setw’ was not declared in this scope

but fixed it by adding the line at the top of the file.

#include

Now it fails with following errors....not sure what to do?

/home/rjn/opencv3-p3-code/classification_and_boxing/newstuff/DynSLAM/src/DynSLAM/InstRecLib/VisoSparseSFProvider.cpp: In function ‘Eigen::Matrix4d instreclib::VisoToEigen(const Matrix&)’:
/home/rjn/opencv3-p3-code/classification_and_boxing/newstuff/DynSLAM/src/DynSLAM/InstRecLib/VisoSparseSFProvider.cpp:9:28: error: passing ‘const Matrix’ as ‘this’ argument discards qualifiers [-fpermissive]
   return Eigen::Matrix4d((~viso_matrix).val[0]);
                            ^
In file included from /home/rjn/opencv3-p3-code/classification_and_boxing/newstuff/DynSLAM/src/DynSLAM/InstRecLib/../../libviso2/src/matcher.h:34:0,
                 from /home/rjn/opencv3-p3-code/classification_and_boxing/newstuff/DynSLAM/src/DynSLAM/InstRecLib/SparseSFProvider.h:7,
                 from /home/rjn/opencv3-p3-code/classification_and_boxing/newstuff/DynSLAM/src/DynSLAM/InstRecLib/VisoSparseSFProvider.h:7,
                 from /home/rjn/opencv3-p3-code/classification_and_boxing/newstuff/DynSLAM/src/DynSLAM/InstRecLib/VisoSparseSFProvider.cpp:3:
/home/rjn/opencv3-p3-code/classification_and_boxing/newstuff/DynSLAM/src/DynSLAM/InstRecLib/../../libviso2/src/matrix.h:106:11: note:   in call to ‘Matrix Matrix::operator~()’
   Matrix  operator~ ();                // transpose
           ^
[ 30%] Building CXX object src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/PrecomputedSegmentationProvider.cpp.o
src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/build.make:158: recipe for target 'src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/VisoSparseSFProvider.cpp.o' failed
make[2]: *** [src/DynSLAM/InstRecLib/CMakeFiles/InstRecLib.dir/VisoSparseSFProvider.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/home/rjn/opencv3-p3-code/classification_and_boxing/newstuff/DynSLAM/src/DynSLAM/InstRecLib/PrecomputedSegmentationProvider.cpp: In member function ‘virtual std::shared_ptr<instreclib::segmentation::InstanceSegmentationResult> instreclib::segmentation::PrecomputedSegmentationProvider::SegmentFrame(const Mat3b&)’:
/home/rjn/opencv3-p3-code/classification_and_boxing/newstuff/DynSLAM/src/DynSLAM/InstRecLib/PrecomputedSegmentationProvider.cpp:173:25: error: ‘imread’ is not a member of ‘cv’
   cv::Mat seg_preview = cv::imread(img_fpath);
                         ^
/home/rjn/opencv3-p3-code/classification_and_boxing/newstuff/DynSLAM/src/DynSLAM/InstRecLib/PrecomputedSegmentationProvider.cpp:174:3: error: ‘resize’ is not a member of ‘cv’
   cv::resize(seg_preview, *last_seg_preview_, cv::Size(), 1.0 / input_scale_, 1.0 / input_scale_, cv::INTER_LINEAR);
   ^
/home/rjn/opencv3-p3-code/classification_and_boxing/newstuff/DynSLAM/src/DynSLAM/InstRecLib/PrecomputedSegmentationProvider.cpp:174:99: error: ‘INTER_LINEAR’ is not a member of ‘cv’

src libviso2 @ c994d00 not found during git clone

Peidong mentioned it would be nice to have this; basically, to not rely on precomputed semantics.

Not hard, just lots of engineering; possible solution: use Flask webserver, and connect to it from C++, send the RGB image, and get back the masks.

Low priority, though.

It's currently a bit of a PITA to preprocess a demo sequence in order to check out DynSLAM. I prepared a trimmed preprocessed version of KITTI Odometry Sequence 06 to help people check out the system and give them an idea of what the folder structure should look like.

I should ensure DynSLAM is happy with the mini-dataset (i.e., I haven't deleted something necessary), and add that info to the README.

Link: http://www.cs.toronto.edu/~iab/dynslam/mini-kitti-odometry-seq-06-for-dynslam.7z

See Nießner et al., 2013, near end of Section §5: "In an idealized case, each depth sample would be modeled as an entire frustum rather than a single ray. We would then allocate all voxel blocks within the truncation region that intersect with this frus- tum. In practice however, this leads to degradation in performance (currently 10-fold). Our ray-based approximation provides a balance between performance and precision. Given the continuous nature of the reconstruction, the frame rate of the sensor, and the mobility of the user, this in practice leads to no holes appearing between voxel blocks at larger distances (see results and accompanying video)."

In their case, given the range of the kinect sensor, there are no artifacts in the map. For us, however, this may be the reason why we're seeing the artifacts in the ground, especially when we set the max depth relatively high (e.g., 20m).

To solve this, although nontrivial, we could try casting multiple rays when e.g., allocating the depth frame for z values > something.

Based on Peidong's provided code.

Comparing reconstructions based on the raw depth is not ideal and can read to problems as disparity goes to zero and Z becomes really noisy, since Z = bf / d.

We should compute disparity (from input, from LIDAR, from raycast) and compare disparities.

This would also allow us to actually use a meaningful unit of measurement instead of just and arbitrary pixel representation of a scaled depth map.

DON'T FORGET TO UPDATE THE DOCS ACCORDINGLY!

Suggestions:

• Check out https://www.eth3d.net and the associated paper to see how they eval depth
• Check out the paper assoc. with the KITTI 2015 stereo bench (Obj SF for autonomous driving, 2015) to see how they evaluate depth. Hint: they care about disparity, NOT the depth.

Hi, all,

I encounter the following build error

[ 55%] Building CXX object CMakeFiles/DynSLAMGUI.dir/src/DynSLAM/Evaluation/EvaluationCallback.cpp.o
[ 56%] Building CXX object CMakeFiles/DynSLAMGUI.dir/src/DynSLAM/DSHandler3D.cpp.o
In file included from /data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp:2:0:
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:15:10: error: ‘Eigen’ does not name a type
   static Eigen::Vector3d GetTranslation(const pangolin::OpenGlRenderState &stat
          ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:23:10: error: ‘Eigen’ does not name a type
   static Eigen::Vector3d GetDirection(const pangolin::OpenGlRenderState &state)
          ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:28:10: error: ‘Eigen’ does not name a type
   static Eigen::Vector3d GetEuler(const Eigen::Matrix3d &rot) {
          ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:77:3: error: ‘Eigen’ does not name a type
   Eigen::Vector3d eye;
   ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:78:3: error: ‘Eigen’ does not name a type
   Eigen::Vector3d direction;
   ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h: In constructor ‘dynslam::gui::DSHandler3D::DSHandler3D(pangolin::OpenGlRenderState*, pangolin::AxisDirection, float, float)’:
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:37:7: error: class ‘dynslam::gui::DSHandler3D’ does not have any field named ‘eye’
     : eye(GetTranslation(*cam_state)),
       ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:37:36: error: ‘GetTranslation’ was not declared in this scope
     : eye(GetTranslation(*cam_state)),
                                    ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:38:7: error: class ‘dynslam::gui::DSHandler3D’ does not have any field named ‘direction’
       direction(GetDirection(*cam_state)),
       ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:38:40: error: ‘GetDirection’ was not declared in this scope
       direction(GetDirection(*cam_state)),
                                        ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:45:5: error: ‘Eigen’ has not been declared
     Eigen::Matrix4d mv = cam_state->GetModelViewMatrix();
     ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:46:5: error: ‘Eigen’ has not been declared
     Eigen::Vector3d euler = GetEuler(mv.block(0, 0, 3, 3));
     ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.h:48:25: error: ‘euler’ was not declared in this scope
     yaw_accum_ = euler(1) + M_PI_2;
                         ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp: In member function ‘void dynslam::gui::DSHandler3D::UpdateModelViewMatrix()’:
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp:23:3: error: ‘direction’ was not declared in this scope
   direction = rot_quat * Eigen::Vector3d(1.0f, 0.0f, 0.0f);
   ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp:25:28: error: ‘eye’ was not declared in this scope
   Eigen::Vector3d target = eye + direction;
                            ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp: In member function ‘virtual void dynslam::gui::DSHandler3D::MouseMotion(pangolin::View&, int, int, int)’:
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp:45:36: error: ‘direction’ was not declared in this scope
       Eigen::Vector3d newMotionX = direction.cross(up_v).normalized() * dx * 0.
                                    ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp:47:7: error: ‘eye’ was not declared in this scope
       eye += newMotionX;
       ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp: In member function ‘virtual void dynslam::gui::DSHandler3D::Mouse(pangolin::View&, pangolin::MouseButton, int, int, bool, int)’:
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp:94:7: error: ‘eye’ was not declared in this scope
       eye += direction.normalized() * zoom_scale_;
       ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp:94:14: error: ‘direction’ was not declared in this scope
       eye += direction.normalized() * zoom_scale_;
              ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp:97:7: error: ‘eye’ was not declared in this scope
       eye -= direction.normalized() * zoom_scale_;
       ^
/data/code/DynSLAM/src/DynSLAM/DSHandler3D.cpp:97:14: error: ‘direction’ was not declared in this scope
       eye -= direction.normalized() * zoom_scale_;
              ^
CMakeFiles/DynSLAMGUI.dir/build.make:350: recipe for target 'CMakeFiles/DynSLAMGUI.dir/src/DynSLAM/DSHandler3D.cpp.o' failed
make[2]: *** [CMakeFiles/DynSLAMGUI.dir/src/DynSLAM/DSHandler3D.cpp.o] Error 1
CMakeFiles/Makefile2:74: recipe for target 'CMakeFiles/DynSLAMGUI.dir/all' failed
make[1]: *** [CMakeFiles/DynSLAMGUI.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

In my system, I already install eigen3 by:
sudo apt-get install libeigen3-dev

Any suggestion to fix this problem?

THX!

Hello,

Im trying DynSLAM on my own data. I used the ZED stereo camera to do this and calibrated it with OpenCV. I see that the kitti data set provides a calibration file in the form of:

P0: 7.215377000000e+02 0.000000000000e+00 6.095593000000e+02 0.000000000000e+00 0.000000000000e+00 7.215377000000e+02 1.728540000000e+02 0.000000000000e+00 0.000000000000e+00 0.000000000000e+00 1.000000000000e+00 0.000000000000e+00

P1: 7.215377000000e+02 0.000000000000e+00 6.095593000000e+02 -3.875744000000e+02 0.000000000000e+00 7.215377000000e+02 1.728540000000e+02 0.000000000000e+00 0.000000000000e+00 0.000000000000e+00 1.000000000000e+00 0.000000000000e+00

P2: 7.215377000000e+02 0.000000000000e+00 6.095593000000e+02 4.485728000000e+01 0.000000000000e+00 7.215377000000e+02 1.728540000000e+02 2.163791000000e-01 0.000000000000e+00 0.000000000000e+00 1.000000000000e+00 2.745884000000e-03

P3: 7.215377000000e+02 0.000000000000e+00 6.095593000000e+02 -3.395242000000e+02 0.000000000000e+00 7.215377000000e+02 1.728540000000e+02 2.199936000000e+00 0.000000000000e+00 0.000000000000e+00 1.000000000000e+00 2.729905000000e-03

R_rect 9.999239000000e-01 9.837760000000e-03 -7.445048000000e-03 -9.869795000000e-03 9.999421000000e-01 -4.278459000000e-03 7.402527000000e-03 4.351614000000e-03 9.999631000000e-01

Tr_velo_cam 7.533745000000e-03 -9.999714000000e-01 -6.166020000000e-04 -4.069766000000e-03 1.480249000000e-02 7.280733000000e-04 -9.998902000000e-01 -7.631618000000e-02 9.998621000000e-01 7.523790000000e-03 1.480755000000e-02 -2.717806000000e-01

Tr_imu_velo 9.999976000000e-01 7.553071000000e-04 -2.035826000000e-03 -8.086759000000e-01 -7.854027000000e-04 9.998898000000e-01 -1.482298000000e-02 3.195559000000e-01 2.024406000000e-03 1.482454000000e-02 9.998881000000e-01 -7.997231000000e-01

The calibration files of OpenCV are as follows:

#%YAML:1.0
camera_name: ZED2_left
image_width: 1920
image_height: 1080
camera_matrix:
rows: 3
cols: 3
data: [ 1.3325889551073612e+03, 0., 9.6462060792131365e+02, 0.,
1.3313556296229908e+03, 5.4097960043772673e+02, 0., 0., 1. ]

distortion_coefficients:
rows: 1
cols: 5
data: [ -1.6965238949210806e-01, 2.4916166768304802e-02,
-1.0860335812810083e-03, 3.5066287012128759e-03,
9.3440116918466895e-04 ]

distortion_model: plumb_bob

rectification_matrix:
rows: 3
cols: 3
data: [ 9.9814719429682419e-01, 4.1577454419303691e-03,
-6.0703308560720474e-02, -4.4027583777311967e-03,
9.9998269084855151e-01, -3.9030400917507819e-03,
6.0686029990808021e-02, 4.1630705171307736e-03,
9.9814822276444692e-01 ]

projection_matrix:
rows: 3
cols: 4
data: [ 1.3371802122767306e+03, 0., 1.0957167205810547e+03, 0., 0.,
1.3371802122767306e+03, 5.5516946792602539e+02, 0., 0., 0., 1.,
0. ]

Any idea to go from the opencv version to the kitti, or how to calibrate my ZED camera tot get the same calibration file as kitti ?

Thank you,
Kind Regards,
Dries Hulens

According to Nießner et al 2013, this can help prevent colors from getting washed out.

This is much easier to set up than the semantics or dispnet, but still low priority.

Ask the Prof for more info

Currently the Gauss-Newton algorithm optimizing relative pose inside RANSAC in libviso2 always starts from the identity transform (se(3) = {0, 0, 0, 0, 0, 0}).

I'd expect to see some improvements if that optimization were initialized to the known instance motion, if available. I'm not sure if doing this would improve the egomotion estimation, but I'm optimistic about the object pose estimation, especially for challenging cases with cars coming on the road from the opposite direction, resulting in very fast relative motion to the camera.

Link error....how to include pthread . Can someone help

[ 84%] Linking CXX executable InfiniTAM
/usr/bin/ld: ITMLib/libITMLib.a(ITMMainEngine.cpp.o): undefined reference to symbol 'pthread_create@@GLIBC_2.2.5'
/lib/x86_64-linux-gnu/libpthread.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
src/InfiniTAM/InfiniTAM/CMakeFiles/InfiniTAM_cli.dir/build.make:110: recipe for target 'src/InfiniTAM/InfiniTAM/InfiniTAM_cli' failed
make[2]: *** [src/InfiniTAM/InfiniTAM/InfiniTAM_cli] Error 1
CMakeFiles/Makefile2:873: recipe for target 'src/InfiniTAM/InfiniTAM/CMakeFiles/InfiniTAM_cli.dir/all' failed
make[1]: *** [src/InfiniTAM/InfiniTAM/CMakeFiles/InfiniTAM_cli.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: ITMLib/libITMLib.a(ITMMainEngine.cpp.o): undefined reference to symbol 'pthread_create@@GLIBC_2.2.5'
/lib/x86_64-linux-gnu/libpthread.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
src/InfiniTAM/InfiniTAM/CMakeFiles/InfiniTAM.dir/build.make:110: recipe for target 'src/InfiniTAM/InfiniTAM/InfiniTAM' failed
make[2]: *** [src/InfiniTAM/InfiniTAM/InfiniTAM] Error 1
CMakeFiles/Makefile2:914: recipe for target 'src/InfiniTAM/InfiniTAM/CMakeFiles/InfiniTAM.dir/all' failed
make[1]: *** [src/InfiniTAM/InfiniTAM/CMakeFiles/InfiniTAM.dir/all] Error 2
[100%] Built target DynSLAMGUI
Makefile:116: recipe for target 'all' failed
make: *** [all] Error 2

Hi,

Recently, I am interesting in your project and want to repeat it!

However, I face some problems about ELAS. (https://github.com/AndreiBarsan/libelas-tooling)
I just repeat the code from above, but get different depth value!

The min value of my depth is -10 (I think it's because of the variable "kInvalidDepth" which is defined in elas.h.), but the min depth you provided (KITTI Odometry Sequence 06, which is used in demo) is 0.
Also the max value is different.

I want to know how should I process the range of depth image!
Thanks so much :)

Will likely take longer than I think, because CMake.

Made some system changes ......but still not compiling

/home/rjn/opencv3-p3-code/tracking_and_slam/newstuff/DynSLAM/src/DynSLAM/InstRecLib/Track.cpp: In member function ‘std::__cxx11::string instreclib::reconstruction::Track::GetAsciiArt() const’:
/home/rjn/opencv3-p3-code/tracking_and_slam/newstuff/DynSLAM/src/DynSLAM/InstRecLib/Track.cpp:73:30: error: ‘setw’ was not declared in this scope
   out << "Object #" << setw(4) << id_ << " [";

It seems that much of the noisy voxels in the reconstruction which stem from the noise in the depth estimation can be singled out by looking at their weight. In the following image, red areas correspond to voxels with weight = 1, i.e., which were only "seen" once:

The blue voxels are those with the maximum allowed weight (in our case, just 10, since the camera is moving quite quickly; the original InfiniTAM had ~100).

A quick and dirty solution would be to simply not render low-weight voxels, but that would also require modifications to the mesh generation algorithm.

A better solution would be to (efficiently) trim all voxels with age > k but weight < l, effectively regularizing the resulting reconstruction. This is what I'll be implementing in the end.

Subtasks:

• Naive cleanup implementation: kernel to detect old, low-weight voxels, and clear them.
• Compute empty blocks
• Deallocate empty blocks
• Graph to visualize empty block count; hopefully it will no longer constantly go down if we do this!
• Clean up code (weird comments, TODOs, debug prints, etc.)
• Apply voxel decay to reconstructed object instances as well.
• Support "reap mode" which can operate on entire volumes.
• Clean up after getting reap mode working.
• Support deleting elements where collisions occur by looking up excess list entries correctly, and then
  performing a proper list deletion in the hash table, in addition to deallocating the VBA element.
• [ ] GUI button to "reap" currently active object.

Hi, I want to use your project but I face some problems now, and I really need help. :(

1. Before compiling DynSLAM project, I compile InstRecLib and get these errors:

/home/imlab-slam/DynSLAM-master/src/DynSLAM/InstRecLib/InstanceReconstructor.cpp: In member function ‘void instreclib::reconstruction::InstanceReconstructor::SaveObjectToMesh(int, const string&)’:
/home/imlab-slam/DynSLAM-master/src/DynSLAM/InstRecLib/InstanceReconstructor.cpp:750:30: error: expected type-specifier before ‘ITMMeshingEngine_CUDA’
auto *meshing_engine = new ITMMeshingEngine_CUDA<ITMVoxel, ITMVoxelIndex> (
^
/home/imlab-slam/DynSLAM-master/src/DynSLAM/InstRecLib/InstanceReconstructor.cpp:763:10: error: type ‘’ argument given to ‘delete’, expected pointer
delete meshing_engine;
^
make[2]: *** [CMakeFiles/InstRecLib.dir/InstanceReconstructor.cpp.o] Error 1
make[1]: *** [CMakeFiles/InstRecLib.dir/all] Error 2
make: *** [all] Error 2

I think the problem is that in InstanceReconstructor.cpp, there is no include related to ITMMeshingEngine_CUDA, so I just include it like this:

#include ITMMeshingEngine_CUDA.h

2. After include ITMMeshingEngine_CUDA.h file, I got another errors:
   /home/imlab-slam/DynSLAM-master/src/DynSLAM/InstRecLib/../../InfiniTAM/InfiniTAM/ITMLib/Engine/DeviceSpecific/CUDA/ITMMeshingEngine_CUDA.h:48:9: error: ‘__ global __’ does not name a type
   __ global __ void findAllocatedBlocks(Vector4s *visibleBlockGlobalPos,

and I also search for the solution... and found that terms related to CUDA(e.g. __ globlal __ in ITMMeshingEngine_CUDA.h) are not allowed in .cpp files, which is a compiler issue.

Because I am a beginner of CUDA, I really don't know how to solve this problem.
Can anyone help me please? really thanks!

Should be doable with nvidia docker, but I'll leave it for after the deadline.

download and add to CMakeList.txt
add_subdirectory(src/libuvc)

Although not completely trivial, we have to be able to do this.
It's also important to get working for the final evaluation, where (among other things) we're interested in seeing how many blocks we can "rescue" by killing noisy observations.

The readme in the InstRecLib folder is a litte redundant.
Also, we could probably rename it to instRec or something, or even dynslam::XXX, since currently it generates libinstreclib, which hurts my OCD.

I am trying to preprocess some extra KITTI odometry sequences for DynSLAM. I've gotten so far as to be able to generate the dispnet-based depths and the MNC-based semantic segmentation masks, and can run DynSLAM with the right combination of flags.

I'd like to be able to also generate and use ELAS-based depths. I have cloned https://github.com/AndreiBarsan/libelas-tooling and have been able to get to the point where I can have kitti2klg create a directory of .ppm and .pgm files. However, it appears that DynSLAM expects to find the ELAS depths in an XML format.

I have combed through the various processing scripts to find clues for the right way to perform that last step of the conversion, but haven't found anything. Any hints?

Currently a little annoying:

[Starting frame 172]
No more frames left in image source.
[Finished frame 171 in 0ms @ inf FPS (approx.)]

Try to see if MNC outputs per-pixel probabilities (may not be shown in the demo code I looked at, but the NN itself should do this) => propagate uncertainty and soft masking; suggested by Prof. Geiger.

Detailed Real-Time Urban 3D Reconstruction From Video by Pollefeys et al. apply sky removal (simple but effective). May be worth looking into this in the future.

Hi, @AndreiBarsan ,

Thanks for sharing such a good work. And now I'm practicing it.
After I had built it, I follow steps mentioned in readme:
Download the kitti seq (2011_09_26_drive_0001 synced+rectified) from here
cd DynSLAM/build
./DynSLAMGUI --dataset_root=/data/kitti_raw_data_seq/2011_09_26/2011_09_26_drive_0001_sync

root@milton-OptiPlex-9010:/data/code/DynSLAM/build# ./DynSLAMGUI  --dataset_root=/data/kitti_raw_data_seq/2011_09_26/2011_09_26_drive_0001_sync
terminate called after throwing an instance of 'std::runtime_error'
  what():  Could not open calibration file: [/data/kitti_raw_data_seq/2011_09_26/2011_09_26_drive_0001_sync/calib.txt]
Stack trace (most recent call last):
#11   Object ", at 0xffffffffffffffff, in 
#10   Object "./DynSLAMGUI, at 0x452e56, in _start
#9    Source "/build/eglibc-oGUzwX/eglibc-2.19/csu/libc-start.c", line 287, in __libc_start_main [0x7f9bb7c9af44]
#8    Source "/data/code/DynSLAM/src/DynSLAM/DynSLAMGUI.cpp", line 1307, in main [0x44d71d]
       1305:   dynslam::DynSlam *dyn_slam;
       1306:   dynslam::Input *input;
      >1307:   BuildDynSlamKittiOdometry(dataset_root, &dyn_slam, &input);
       1308: 
       1309:   dynslam::gui::PangolinGui pango_gui(dyn_slam, input);
       1310:   pango_gui.Run();
#7    Source "/data/code/DynSLAM/src/DynSLAM/DynSLAMGUI.cpp", line 1163, in dynslam::BuildDynSlamKittiOdometry(std::string const&, dynslam::DynSlam**, dynslam::Input**) [0x48bdcd]
       1160:   // HERE BE DRAGONS Make sure you're using the correct matrix for the grayscale and/or color cameras!
       1161:   ReadKittiOdometryCalibration(dataset_root + "/" + input_config.calibration_fname,
       1162:                                left_gray_proj, right_gray_proj, left_color_proj, right_color_proj,
      >1163:                                velo_to_left_gray_cam, downscale_factor);
       1164: 
       1165:   Eigen::Vector2i frame_size = GetFrameSize(dataset_root, input_config);
#6    Source "/data/code/DynSLAM/src/DynSLAM/DynSLAMGUI.cpp", line 1061, in dynslam::ReadKittiOdometryCalibration(std::string const&, Eigen::Matrix<double, 3, 4, 0, 3, 4>&, Eigen::Matrix<double, 3, 4, 0, 3, 4>&, Eigen::Matrix<double, 3, 4, 0, 3, 4>&, Eigen::Matrix<double, 3, 4, 0, 3, 4>&, Eigen::Matrix<double, 4, 4, 0, 4, 4>&, double) [0x48ba1a]
       1058:   static const string kRightColor = "P3:";
       1059:   ifstream in(fpath);
       1060:   if (! in.is_open()) {
      >1061:     throw runtime_error(utils::Format("Could not open calibration file: [%s]", fpath.c_str()));
       1062:   }
       1063: 
       1064:   left_gray_proj = ReadProjection(kLeftGray, in, downscale_factor);
#5    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6, at 0x7f9bb82bedf7, in __cxa_throw
#4    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6, at 0x7f9bb82bebe0, in std::terminate()
#3    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6, at 0x7f9bb82beb95, in std::rethrow_exception(std::__exception_ptr::exception_ptr)
#2    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6, at 0x7f9bb82c0bbc, in __gnu_cxx::__verbose_terminate_handler()
#1    Source "/build/eglibc-oGUzwX/eglibc-2.19/stdlib/abort.c", line 89, in __GI_abort [0x7f9bb7cb3027]
#0    Source "/build/eglibc-oGUzwX/eglibc-2.19/signal/../nptl/sysdeps/unix/sysv/linux/raise.c", line 56, in __GI_raise [0x7f9bb7cafc37]
Aborted (Signal sent by tkill() 25334 0)
Aborted (core dumped)
root@milton-OptiPlex-9010:/data/code/DynSLAM/build# 

I think that's the location inconsistency problem of calibration file and seg images, it 'll be more helpful if you could upload an example seq for testing, so that we could know the input directory structures for the code.
Thanks!

Despite (slightly) increasing its memory footprint, this would make the engine much more robust (witch cleaner code) to decaying voxel blocks.

Instead of referencing visible entries by their absolute hashTable location, which is brittle once we start moving things around as we're deleting entries, we can keep track of the full coords of the visible blocks, and simply re-look them up in the hash table when necessary, eliminating the problem of stale pointers to deallocated buckets, save for the case where a bucket is completely emptied out, a case which is trivial to detect in the code.

After running the experiments and while working on the improved viz, I seem to have introduced a tracking bug causing drift in the overall fused reconstruction. While the results can be visualized, the final 3D car positions are not quite right.

This did not occur with the old "dump" depth-only-camera-pov composited rendering code. I should look into this after working on the paper writeup.

Any idea how to solve this compile error:

/home/rjn/opencv3-p3-code/tracking_and_slam/newstuff/DynSLAM/src/InfiniTAM/InfiniTAM/Engine/LibUVCEngine.cpp: In function ‘void callback_depth(uvc_frame_t*, void*)’:
/home/rjn/opencv3-p3-code/tracking_and_slam/newstuff/DynSLAM/src/InfiniTAM/InfiniTAM/Engine/LibUVCEngine.cpp:57:42: error: ‘UVC_FRAME_FORMAT_GRAY16’ was not declared in this scope
  data->framebuffer_depth->frame_format = UVC_FRAME_FORMAT_GRAY16;
                                          ^
/home/rjn/opencv3-p3-code/tracking_and_slam/newstuff/DynSLAM/src/InfiniTAM/InfiniTAM/Engine/LibUVCEngine.cpp: At global scope:
/home/rjn/opencv3-p3-code/tracking_and_slam/newstuff/DynSLAM/src/InfiniTAM/InfiniTAM/Engine/LibUVCEngine.cpp:62:8: error: ‘uvc_frame_desc’ does not name a type
  const uvc_frame_desc *frame_desc;
        ^
/home/rjn/opencv3-p3-code/tracking_and_slam/newstuff/DynSLAM/src/InfiniTAM/InfiniTAM/Engine/LibUVCEngine.cpp: In function ‘FormatSpec findSuitableFormat(uvc_device_handle_t*, int, int, const char*)’:
/home/rjn/opencv3-p3-code/tracking_and_slam

• + no more depth generation code in ELAS tooling

• + no more branching to switch between depth-from-disp and depth inputs

• + can directly compare fused disparity with input disparity and LIDAR disparity

• + may improve evaluation accuracy a little

• - takes time to implement

• - need to regenerate ELAS cache

It would be nice to visualize the track of the camera through the 3D world. It should be doable by extending the rendering system to possibly allow extra stuff to get rendered over the output of the raytacer.

If we get the chance to also localize the cars, it would be nice to render vehicle tracks as well in the 3D visualization. Combined with the trimmed, high-resolution map, it should make for a badass visualization.

Hi ,all ,

I got the following build error regarding to Pangolin:

[ 64%] Building CXX object CMakeFiles/DynSLAMGUI.dir/src/DynSLAM/DynSLAMGUI.cpp.o
/data/code/DynSLAM/src/DynSLAM/DynSLAMGUI.cpp: In member function ‘void dynslam::gui::PangolinGui::PreviewSparseSF(const std::vector<instreclib::RawFlow, Eigen::aligned_allocator<instreclib::RawFlow> >&, const pangolin::View&)’:
/data/code/DynSLAM/src/DynSLAM/DynSLAMGUI.cpp:513:23: warning: unused variable ‘font’ [-Wunused-variable]
     pangolin::GlFont &font = pangolin::GlFont::I();
                       ^
[ 64%] Linking CXX executable DynSLAMGUI
/usr/bin/ld: cannot find -lbfd
collect2: error: ld returned 1 exit status
make[2]: *** [DynSLAMGUI] Error 1
make[1]: *** [CMakeFiles/DynSLAMGUI.dir/all] Error 2
make: *** [all] Error 2
root@milton-OptiPlex-9010:/data/code/DynSLAM/build

Any suggestion to fix it?

THANKS~

Right now the OpenCV buffers aren't zeroed out on launch causing ugly-looking artifacts before the system starts.

This issue does not occur when using ELAS, just dispnet; the filterDepth_device kernel fails (but without crashing the rest of the program) at the line tmpz = imageData_in[(x + j) + (y + i) * imgDims.x];

This may also be the root cause of the "floating" artifacts produced when using dispnet depth but not elas depth. Solving this would allow me to turn down the min depth back to 0.5 or so, yay!

Once again, cuda-memcheck is the real MVP.

Example for reconstructing a car:

Allocated voxel blocks in local GPU buffer: 10000
Last free block ID:       9999
Last free excess list ID: 524287
Allocated size:         5120000

The main hash map buffer has 10k entries, but the excess list has ~500k, even though it's almost never used for such a small volume.

The problem is that the excess list size is defined as a global constant. We need to fix that. This is particularly important for speeding up the cleanup for instance reconstruction, which should run on the entire volume in real time, but is made much slower by the pointlessly large excess list.

Warning: Note that the large excess list size does not imply we're actually allocating those blocks; just the hash map entries, which are still quite big, but not quite as big as the actual VBA entries, i.e., the actual blocks. This was something I originally got confused with. SDF_EXCESS_LIST_SIZE and SDF_BUCKET_NUM dictate the size of the hast table, NOT that of the VBA. There's just one VBA, of course, and elements from either the hash map itself or the excess list point to locations in the VBA.

Right now we're very naively fusing depth information into the maps. We want to compute the fusion weight
to reflect the stereo depth error function, which grows quadratically with distance.

If we do this, then perhaps we'll be able to get away with even more permissive depth clamping. Now we clamp depth values at 15, but if we discount the update weights for more distant points more appropriately we could even do 20 or more. In those cases, very distance values would simply have a very small weight, so we could simply remove them if they're bad once we have some map cleanup in place. But if measurements for distant objects stay consistent (e.g., a well-textured planar surface), we would still be able to integrate them into our map.

Important: consider the option of leaving the weight as it is, and adding another field to each voxel, called confidence, and use that maybe in the decay itself. Check out ITM v3's confidence and see how they implemented it; maybe it can be useful here.

General steps:

• Simple quadratic model
• Adapt map representation to use float weights.
• Adapt weight visualization.
• Adapt decay code accordingly.
• Support enabling/disabling this model. Using the vanilla int weights should still work OK if the weight fields are floats; I should just be careful I don't have any weird rounding errors in code expecting uchars.
• (Bonus) Gradient-based model (assumption: relevant for ELAS and dispnet; dicuss the details with Peidong first!)