zielon / 3dscanning Goto Github PK
View Code? Open in Web Editor NEWKinect Fusion for Markerless Augumented Reality on CUDA
Kinect Fusion for Markerless Augumented Reality on CUDA
Task: Read current RGB camera from (DLL tracker) and set as texture for the sprite attached to the camera.
The screen should show the image created by the Tracker.
Task: Implement efficient ICP solution in order to be able to estimate pose in realtime.
I would recommend to check the current solutions (libraries, frameworks, etc). Then, we can use it directly.
It would nice though to check the code and try to understand it (at least generally).
NB: Tasks that were already implemented in the exercises can be replaced with efficient external code (said by professor).
We should try to integrate the ICPCUDA library to our Tracker solution. The implementation should be a class so we can easily change between naive implementation and cuda implementation.
Library References:
[0] PCL: http://www.pointclouds.org/
[1] ICPCUDA: https://github.com/mp3guy/ICPCUDA.
Task: Implement efficient Volumetric Fusion solution in order to be able to compute SDF in realtime.
I would recommend to check the current solutions (libraries, frameworks, etc) in order to adapt our current solultion to improve quality and performance.
References:
[0] CUDA TSDF: https://github.com/Scoobadood/TSDF
[1] Kinfu: http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php
[2] Open3D: http://www.open3d.org/docs/tutorial/Advanced/rgbd_integration.html#tsdf-volume-integration
[3] Elastic Reconstruction: https://github.com/qianyizh/ElasticReconstruction
Adapt the code from Exercise 3 PointCloud.h
for generating a normal map from a depth map.
Check out the constructor of the class.
Task: Compute 3-levels multi scale representation of the point cloud (vertex and normal map) in order to improve ICP point-to-plane with projective data association.
@juanraul8 support.
References:
[0] https://github.com/chrdiller/KinectFusionLib/blob/master/src/pose_estimation.cpp
Task: Install sensor functionality on @Mirjang computer to first version demo.
We should create the necessary functionality to store our test in order to load it and execute it in realtime time unity (now our reconstruction is not realtime).
Experiment Dataset requires:
@juanraul8 support.
@juanraul8 just backup.
Task: choosing the software technologies and tools and deciding how to integrate them depending of the task.
Task: Implement skeleton of C++ to Unity interoperability by calling functions from dll.
References
[0] https://docs.unity3d.com/Manual/UsingDLL.html
The testing class has too many methods. We should split them into separate files.
readme
fileTask: Obtaining iPad from the 3D Scanning Group.
Relevant questions:
-Which is the policy to rent the devices?
-How many devices can we get?
Task: show the normal maps derived by the cloud points in a new test or an existing one.
The dataset has a few flaws.
i + 1
in getSequentialFrame()
where at the start i = 0
. So we read the frame with number 1 in the first read call.It is a quick fix. Can you @Mirjang do those changes ASAP?
Task: Implementing the optimization part of the ICP algorithm to estimate the camera pose of each frame.
The implementation will be based on the exercise 3 [1] and 4 [2] of the course.
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP.
[2] Exercise 4: Feature Matching – Bundle Adjustment.
Task: Compute the camera intrinsics for the color map and depth map of the sensor.
References:
[0] Exercise 1: amera Intrinsics - Back-projection - Meshes.
[1] How to compute intrinsic camera matrix for a camera: https://codeyarns.com/2015/09/08/how-to-compute-intrinsic-camera-matrix-for-a-camera/
Task: Implement a test (WindowTests) to check the point cloud previous to the volumetric fusion.
Similart to the Exercise 1 [0], we can generate a simple mesh from the point cloud. Then we can check the result in MeshLab [1]
@juanraul8 support.
References:
[0] Exercise 1: Camera Intrinsics, Back-projection, Meshes.
[1] MeshLab: http://www.meshlab.net/
Task: Implementing the outliers pruning in order to estimate the camera pose of each frame.
The implementation will be based on the exercise 3 [1] and 4 [2] of the course.
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP.
[2] Exercise 4: Feature Matching – Bundle Adjustment.
References:
[0] Marching Cubes GPU implementation for Unity: https://github.com/pavelkouril/unity-marching-cubes-gpu/blob/master/Assets/MarchingCubes/Shaders/MarchingCubes.compute
Task: Appy bilateral filter to the raw depth map as it is proposed in the Kinect Fusion paper.
The depth values should be also scale to transform depth values to real scale in meters.
For example, the factor 5000 is iused in the datasets [1]. A value of 5000 in the depth map corresponds to 1 meter.
Reference:
[0] KinectFusion: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ismar2011.pdf
[1] RGB-D SLAM Dataset and Benchmark : https://vision.in.tum.de/data/datasets/rgbd-dataset
In unity a user can select an option to record/scan the scene. (Lets say maximum a few minutes of recording)
Then the scene has to be processed offline. In the UI the user will see some kind of progress bar with no possibility do use any buttons. Once the processing is finished and the mesh is loaded to our scene the UI is available again. The user can use the camera and our ICP pose estimation to look at the constructed mesh from different angles.
Linked: #86
Task: Implementing Marching Cubes algorithm to extract surface from a SDF, i.e. obtaining a semi-dense mesh from the scene.
@Zielon support.
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 2: Implicit Surfaces - Marching Cubes.
1- First slide explains the current level of progress
2- Second slide includes a road map for next two weeks
PS: For current status include everything before Fusion.
Second slide: Mesh integration to Unity, sensor integration with correct pose.
Task: the mesh generated using volumetric fusion have important holes.
Possible solution:
Report
Deadline is 31st January 23:55
Two page report with the summary and result figures/tables.
Same template as project proposals
Task: Implementing the correspondence matching between frames in order to estimate the camera pose of each frame.
The implementation will be based on the exercise 3 [1] and 4 [2] of the course.
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP.
[2] Exercise 4: Feature Matching – Bundle Adjustment.
Task: Implement a basic AR animation on unity using the color map and the camera pose given by the tracker.
The animation should consist on placing virtual objects on the points selected by the user (e.g. mouse interaction).
Unity modules:
@Mirjang support.
Task: Install Sensor software and libraries.
You can find instruction of the readme.
I am planning to integrate sensor to our project. So I need you install the dependencies so you can compile the project with sensor.
Task: Implement a basic tracker using ICP Algorithm.
The implementation will be based on the exercise 3 of the course [1].
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP
@juanraul8 support.
Task: Define and implement the tracker skeleton, which will be implemented in C++.
The implementation will be based on the exercise 3 of the course [1].
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP
Task: Implement a surface integration to compute the SDF representation (voxels representation) of the world.
@Mirjang support.
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 2: Implicit Surfaces - Marching Cubes.
Task: Integrate cuda ICP solution [0] to speed up further camera tracking .
References:
[0] ICPCUDA: https://github.com/mp3guy/ICPCUDA.
Task: Implement efficient Marching Cubes solution in order to be able to extract surfaces in realtime.
I would recommend to check the current solutions (libraries, frameworks, etc). Then, we can use it directly.
It would nice though to check the code and try to understand it (at least generally).
NB: Tasks that were already implemented in the exercises can be replaced with efficient external code (said by professor).
The implementation should be a class so we can easily change between naive implementation and gpu implementation.
Library References:
[0] PCL: http://www.pointclouds.org/
[1] GPU-Marching-Cubes: https://github.com/smistad/GPU-Marching-Cubes
[2] GPU-accelerated data expansion for the Marching Cubes algorithm :https://www.nvidia.com/content/gtc-2010/pdfs/gtc10_session2020_hp5mc_dyken_ziegler.pdf
Ideally the Dataset
reader should store already read files and return values from the storage for each request.
The function to fetch a trajectory for a given frame is needed.
Task: implement option to record video of the basics physics scene in Unity.
Task: Implement a complex AR animation on unity using the color map, the camera pose and the scene mesh given by the tracker.
The animation should consist on basic rigid body collision between virtual objects (e.g. spheres) and physical objects (e.g. table).
Unity modules:
Task: design an algorithm to generate mesh from a point cloud using Advancing Front approach.
Maybe we can use the PCL library [0].
References:
[0] http://pointclouds.org/
[1] Exercise 1: Camera Intrinsics - Back-projection - Meshes.
Task: Deployment of our first version project to iOS platforms using Unity on Mac OS.
The procedure is described on the website [0].
References:
[0] https://docs.unity3d.com/Manual/UnityCloudBuildiOS.html
Task: Implement the sensor functionality using OpenNI 2 in a similar way to our XtionStreamReader class.
References:
Bug: mesh generated by reconstruction is mirrored on the x/z plane.
Task: Fix the bug related to the camera movement.
Up and down movement of the camera is working on the opposite direction.
The resolution 320 x 240 for color map and depth map works fine. However, the depth maps for the resolution 640 x 480 does not work properly. It seems depth map processing is significantly slower.
There is no problem when only depth maps are used.
Task: use depth map estimation of the model (sdf function) instead of previous depth maps.
The depth map estimation can be obtained by raycasting the sdf function with the previous pose.
Task: Implement RGB-D Stream from device (iPad camera) to server (unity) using the library Uplink.
References:
[0] https://structure.io/structure-sensor/getstarted/first-use
[1] https://github.com/occipital/uplink
Task: real time test on professor´s computer.
On Tuesday, we should install and run our solution in professor's laptop.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.