zielon / 3dscanning Goto Github PK

Adapt the code from Exercise 3 PointCloud.h for generating a normal map from a depth map.
Check out the constructor of the class.

Task: Deployment of our first version project to iOS platforms using Unity on Mac OS.

The procedure is described on the website [0].

References:
[0] https://docs.unity3d.com/Manual/UnityCloudBuildiOS.html

Task: Implement RGB-D Stream from device (iPad camera) to server (unity) using the library Uplink.

References:
[0] https://structure.io/structure-sensor/getstarted/first-use
[1] https://github.com/occipital/uplink

Report

• Deadline is 31st January 23:55

• Two page report with the summary and result figures/tables.

• Same template as project proposals

• https://www.overleaf.com/5314334993ynbhvfrvygdb

Task: design an algorithm to generate mesh from a point cloud using Advancing Front approach.

Maybe we can use the PCL library [0].

References:
[0] http://pointclouds.org/
[1] Exercise 1: Camera Intrinsics - Back-projection - Meshes.

Task: Implement a complex AR animation on unity using the color map, the camera pose and the scene mesh given by the tracker.

The animation should consist on basic rigid body collision between virtual objects (e.g. spheres) and physical objects (e.g. table).

Unity modules:

• Graphics.
• Animation.
• Physics.

Task: Implement a basic tracker using ICP Algorithm.

The implementation will be based on the exercise 3 of the course [1].

References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP

Task: show the normal maps derived by the cloud points in a new test or an existing one.

Task: Compute 3-levels multi scale representation of the point cloud (vertex and normal map) in order to improve ICP point-to-plane with projective data association.

@juanraul8 support.

References:
[0] https://github.com/chrdiller/KinectFusionLib/blob/master/src/pose_estimation.cpp

References:
[0] Marching Cubes GPU implementation for Unity: https://github.com/pavelkouril/unity-marching-cubes-gpu/blob/master/Assets/MarchingCubes/Shaders/MarchingCubes.compute

Bug: mesh generated by reconstruction is mirrored on the x/z plane.

Task: Implement the sensor functionality using OpenNI 2 in a similar way to our XtionStreamReader class.

References:

• OpenNI 2: https://structure.io/openni

Task: Implement a surface integration to compute the SDF representation (voxels representation) of the world.

@Mirjang support.

References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 2: Implicit Surfaces - Marching Cubes.

Task: Define and implement the tracker skeleton, which will be implemented in C++.

The implementation will be based on the exercise 3 of the course [1].

References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP

Task: Appy bilateral filter to the raw depth map as it is proposed in the Kinect Fusion paper.

The depth values should be also scale to transform depth values to real scale in meters.

For example, the factor 5000 is iused in the datasets [1]. A value of 5000 in the depth map corresponds to 1 meter.

Reference:
[0] KinectFusion: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ismar2011.pdf
[1] RGB-D SLAM Dataset and Benchmark : https://vision.in.tum.de/data/datasets/rgbd-dataset

Task: Install Sensor software and libraries.

You can find instruction of the readme.

I am planning to integrate sensor to our project. So I need you install the dependencies so you can compile the project with sensor.

Ideally the Dataset reader should store already read files and return values from the storage for each request.

The function to fetch a trajectory for a given frame is needed.

Task: Integrate cuda ICP solution [0] to speed up further camera tracking .

References:
[0] ICPCUDA: https://github.com/mp3guy/ICPCUDA.

Task: Read current RGB camera from (DLL tracker) and set as texture for the sprite attached to the camera.

The screen should show the image created by the Tracker.

1- First slide explains the current level of progress
2- Second slide includes a road map for next two weeks

PS: For current status include everything before Fusion.
Second slide: Mesh integration to Unity, sensor integration with correct pose.

Task: Install sensor functionality on @Mirjang computer to first version demo.

We should create the necessary functionality to store our test in order to load it and execute it in realtime time unity (now our reconstruction is not realtime).

Experiment Dataset requires:

• RGB capture.
• Depth capture.
• Pose capture (Trajectory in SLAM datasets).
• Meshes.

@juanraul8 support.

Task: Implementing the optimization part of the ICP algorithm to estimate the camera pose of each frame.

The implementation will be based on the exercise 3 [1] and 4 [2] of the course.

References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP.
[2] Exercise 4: Feature Matching – Bundle Adjustment.

Task: Implementing the correspondence matching between frames in order to estimate the camera pose of each frame.

The implementation will be based on the exercise 3 [1] and 4 [2] of the course.

References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP.
[2] Exercise 4: Feature Matching – Bundle Adjustment.

Task: Implement a test (WindowTests) to check the point cloud previous to the volumetric fusion.

Similart to the Exercise 1 [0], we can generate a simple mesh from the point cloud. Then we can check the result in MeshLab [1]

@juanraul8 support.

References:
[0] Exercise 1: Camera Intrinsics, Back-projection, Meshes.
[1] MeshLab: http://www.meshlab.net/

In unity a user can select an option to record/scan the scene. (Lets say maximum a few minutes of recording)
Then the scene has to be processed offline. In the UI the user will see some kind of progress bar with no possibility do use any buttons. Once the processing is finished and the mesh is loaded to our scene the UI is available again. The user can use the camera and our ICP pose estimation to look at the constructed mesh from different angles.

Linked: #86

Task: Implement a basic AR animation on unity using the color map and the camera pose given by the tracker.

The animation should consist on placing virtual objects on the points selected by the user (e.g. mouse interaction).

Unity modules:

• Graphics.
• Animation.

@Mirjang support.

Task: Compute the camera intrinsics for the color map and depth map of the sensor.

References:
[0] Exercise 1: amera Intrinsics - Back-projection - Meshes.
[1] How to compute intrinsic camera matrix for a camera: https://codeyarns.com/2015/09/08/how-to-compute-intrinsic-camera-matrix-for-a-camera/

Task: Obtaining iPad from the 3D Scanning Group.

Relevant questions:
-Which is the policy to rent the devices?
-How many devices can we get?

Task: Implement efficient Marching Cubes solution in order to be able to extract surfaces in realtime.

I would recommend to check the current solutions (libraries, frameworks, etc). Then, we can use it directly.

It would nice though to check the code and try to understand it (at least generally).

NB: Tasks that were already implemented in the exercises can be replaced with efficient external code (said by professor).

The implementation should be a class so we can easily change between naive implementation and gpu implementation.

Library References:
[0] PCL: http://www.pointclouds.org/
[1] GPU-Marching-Cubes: https://github.com/smistad/GPU-Marching-Cubes
[2] GPU-accelerated data expansion for the Marching Cubes algorithm :https://www.nvidia.com/content/gtc-2010/pdfs/gtc10_session2020_hp5mc_dyken_ziegler.pdf

Task: Implement efficient Volumetric Fusion solution in order to be able to compute SDF in realtime.

I would recommend to check the current solutions (libraries, frameworks, etc) in order to adapt our current solultion to improve quality and performance.

References:
[0] CUDA TSDF: https://github.com/Scoobadood/TSDF
[1] Kinfu: http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php
[2] Open3D: http://www.open3d.org/docs/tutorial/Advanced/rgbd_integration.html#tsdf-volume-integration
[3] Elastic Reconstruction: https://github.com/qianyizh/ElasticReconstruction

Task: real time test on professor´s computer.

On Tuesday, we should install and run our solution in professor's laptop.

Task: implement option to record video of the basics physics scene in Unity.

Task: use depth map estimation of the model (sdf function) instead of previous depth maps.

The depth map estimation can be obtained by raycasting the sdf function with the previous pose.

Task: Implementing Marching Cubes algorithm to extract surface from a SDF, i.e. obtaining a semi-dense mesh from the scene.

@Zielon support.

References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 2: Implicit Surfaces - Marching Cubes.

Task: Implement skeleton of C++ to Unity interoperability by calling functions from dll.

References
[0] https://docs.unity3d.com/Manual/UsingDLL.html

Task: the mesh generated using volumetric fusion have important holes.

Possible solution:

• Filling holes implementation of volumetric fusion.
• Smoothing meshes as post processing task.

Task: Fix the bug related to the camera movement.

Up and down movement of the camera is working on the opposite direction.

The dataset has a few flaws.

1. It skips the first frame. The iterator is set to i + 1 in getSequentialFrame() where at the start i = 0. So we read the frame with number 1 in the first read call.
2. I want to be able just to pass an index of a frame which I am interested in. The whole logic for skipping, waiting, whatsoever is pointless because I cannot read for instance every 10th frame. Moreover, the read call is still in a loop so I have the index there.

It is a quick fix. Can you @Mirjang do those changes ASAP?

@juanraul8 just backup.

Task: Implement efficient ICP solution in order to be able to estimate pose in realtime.

I would recommend to check the current solutions (libraries, frameworks, etc). Then, we can use it directly.

It would nice though to check the code and try to understand it (at least generally).

NB: Tasks that were already implemented in the exercises can be replaced with efficient external code (said by professor).

We should try to integrate the ICPCUDA library to our Tracker solution. The implementation should be a class so we can easily change between naive implementation and cuda implementation.

Library References:
[0] PCL: http://www.pointclouds.org/
[1] ICPCUDA: https://github.com/mp3guy/ICPCUDA.

Task: Implementing the outliers pruning in order to estimate the camera pose of each frame.

The implementation will be based on the exercise 3 [1] and 4 [2] of the course.

References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP.
[2] Exercise 4: Feature Matching – Bundle Adjustment.

Task: choosing the software technologies and tools and deciding how to integrate them depending of the task.

The resolution 320 x 240 for color map and depth map works fine. However, the depth maps for the resolution 640 x 480 does not work properly. It seems depth map processing is significantly slower.

There is no problem when only depth maps are used.

@juanraul8 support.

The testing class has too many methods. We should split them into separate files.

1. Add a readme file
2. Create a testing class for each component