zielon / 3dscanning Goto Github PK
View Code? Open in Web Editor NEWKinect Fusion for Markerless Augumented Reality on CUDA
Kinect Fusion for Markerless Augumented Reality on CUDA
Task: choosing the software technologies and tools and deciding how to integrate them depending of the task.
Task: Deployment of our first version project to iOS platforms using Unity on Mac OS.
The procedure is described on the website [0].
References:
[0] https://docs.unity3d.com/Manual/UnityCloudBuildiOS.html
Task: Implement efficient Marching Cubes solution in order to be able to extract surfaces in realtime.
I would recommend to check the current solutions (libraries, frameworks, etc). Then, we can use it directly.
It would nice though to check the code and try to understand it (at least generally).
NB: Tasks that were already implemented in the exercises can be replaced with efficient external code (said by professor).
The implementation should be a class so we can easily change between naive implementation and gpu implementation.
Library References:
[0] PCL: http://www.pointclouds.org/
[1] GPU-Marching-Cubes: https://github.com/smistad/GPU-Marching-Cubes
[2] GPU-accelerated data expansion for the Marching Cubes algorithm :https://www.nvidia.com/content/gtc-2010/pdfs/gtc10_session2020_hp5mc_dyken_ziegler.pdf
Task: implement option to record video of the basics physics scene in Unity.
Task: Obtaining iPad from the 3D Scanning Group.
Relevant questions:
-Which is the policy to rent the devices?
-How many devices can we get?
Bug: mesh generated by reconstruction is mirrored on the x/z plane.
Task: Implement a surface integration to compute the SDF representation (voxels representation) of the world.
@Mirjang support.
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 2: Implicit Surfaces - Marching Cubes.
Task: Implement RGB-D Stream from device (iPad camera) to server (unity) using the library Uplink.
References:
[0] https://structure.io/structure-sensor/getstarted/first-use
[1] https://github.com/occipital/uplink
Task: Implementing the outliers pruning in order to estimate the camera pose of each frame.
The implementation will be based on the exercise 3 [1] and 4 [2] of the course.
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP.
[2] Exercise 4: Feature Matching – Bundle Adjustment.
Task: Install Sensor software and libraries.
You can find instruction of the readme.
I am planning to integrate sensor to our project. So I need you install the dependencies so you can compile the project with sensor.
The dataset has a few flaws.
i + 1
in getSequentialFrame()
where at the start i = 0
. So we read the frame with number 1 in the first read call.It is a quick fix. Can you @Mirjang do those changes ASAP?
Task: real time test on professor´s computer.
On Tuesday, we should install and run our solution in professor's laptop.
The testing class has too many methods. We should split them into separate files.
readme
fileAdapt the code from Exercise 3 PointCloud.h
for generating a normal map from a depth map.
Check out the constructor of the class.
Task: Fix the bug related to the camera movement.
Up and down movement of the camera is working on the opposite direction.
Task: the mesh generated using volumetric fusion have important holes.
Possible solution:
1- First slide explains the current level of progress
2- Second slide includes a road map for next two weeks
PS: For current status include everything before Fusion.
Second slide: Mesh integration to Unity, sensor integration with correct pose.
Task: Implement a complex AR animation on unity using the color map, the camera pose and the scene mesh given by the tracker.
The animation should consist on basic rigid body collision between virtual objects (e.g. spheres) and physical objects (e.g. table).
Unity modules:
Task: use depth map estimation of the model (sdf function) instead of previous depth maps.
The depth map estimation can be obtained by raycasting the sdf function with the previous pose.
Task: Read current RGB camera from (DLL tracker) and set as texture for the sprite attached to the camera.
The screen should show the image created by the Tracker.
Task: show the normal maps derived by the cloud points in a new test or an existing one.
Task: Integrate cuda ICP solution [0] to speed up further camera tracking .
References:
[0] ICPCUDA: https://github.com/mp3guy/ICPCUDA.
In unity a user can select an option to record/scan the scene. (Lets say maximum a few minutes of recording)
Then the scene has to be processed offline. In the UI the user will see some kind of progress bar with no possibility do use any buttons. Once the processing is finished and the mesh is loaded to our scene the UI is available again. The user can use the camera and our ICP pose estimation to look at the constructed mesh from different angles.
Linked: #86
@juanraul8 support.
The resolution 320 x 240 for color map and depth map works fine. However, the depth maps for the resolution 640 x 480 does not work properly. It seems depth map processing is significantly slower.
There is no problem when only depth maps are used.
Task: Define and implement the tracker skeleton, which will be implemented in C++.
The implementation will be based on the exercise 3 of the course [1].
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP
References:
[0] Marching Cubes GPU implementation for Unity: https://github.com/pavelkouril/unity-marching-cubes-gpu/blob/master/Assets/MarchingCubes/Shaders/MarchingCubes.compute
@juanraul8 just backup.
Task: Install sensor functionality on @Mirjang computer to first version demo.
We should create the necessary functionality to store our test in order to load it and execute it in realtime time unity (now our reconstruction is not realtime).
Experiment Dataset requires:
@juanraul8 support.
Task: Implementing Marching Cubes algorithm to extract surface from a SDF, i.e. obtaining a semi-dense mesh from the scene.
@Zielon support.
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 2: Implicit Surfaces - Marching Cubes.
Task: Implement a basic tracker using ICP Algorithm.
The implementation will be based on the exercise 3 of the course [1].
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP
Task: Implement skeleton of C++ to Unity interoperability by calling functions from dll.
References
[0] https://docs.unity3d.com/Manual/UsingDLL.html
Task: Implementing the correspondence matching between frames in order to estimate the camera pose of each frame.
The implementation will be based on the exercise 3 [1] and 4 [2] of the course.
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP.
[2] Exercise 4: Feature Matching – Bundle Adjustment.
Task: Implement the sensor functionality using OpenNI 2 in a similar way to our XtionStreamReader class.
References:
Report
Deadline is 31st January 23:55
Two page report with the summary and result figures/tables.
Same template as project proposals
Task: Compute the camera intrinsics for the color map and depth map of the sensor.
References:
[0] Exercise 1: amera Intrinsics - Back-projection - Meshes.
[1] How to compute intrinsic camera matrix for a camera: https://codeyarns.com/2015/09/08/how-to-compute-intrinsic-camera-matrix-for-a-camera/
Task: Implementing the optimization part of the ICP algorithm to estimate the camera pose of each frame.
The implementation will be based on the exercise 3 [1] and 4 [2] of the course.
References:
[0] Lecture 5: Rigid Surface Tracking & Reconstruction (3D Scanning/Justus Thies Slides).
[1] Exercise 3: Registration – Procrustes & ICP.
[2] Exercise 4: Feature Matching – Bundle Adjustment.
Task: Implement efficient ICP solution in order to be able to estimate pose in realtime.
I would recommend to check the current solutions (libraries, frameworks, etc). Then, we can use it directly.
It would nice though to check the code and try to understand it (at least generally).
NB: Tasks that were already implemented in the exercises can be replaced with efficient external code (said by professor).
We should try to integrate the ICPCUDA library to our Tracker solution. The implementation should be a class so we can easily change between naive implementation and cuda implementation.
Library References:
[0] PCL: http://www.pointclouds.org/
[1] ICPCUDA: https://github.com/mp3guy/ICPCUDA.
Task: design an algorithm to generate mesh from a point cloud using Advancing Front approach.
Maybe we can use the PCL library [0].
References:
[0] http://pointclouds.org/
[1] Exercise 1: Camera Intrinsics - Back-projection - Meshes.
Task: Compute 3-levels multi scale representation of the point cloud (vertex and normal map) in order to improve ICP point-to-plane with projective data association.
@juanraul8 support.
References:
[0] https://github.com/chrdiller/KinectFusionLib/blob/master/src/pose_estimation.cpp
Task: Appy bilateral filter to the raw depth map as it is proposed in the Kinect Fusion paper.
The depth values should be also scale to transform depth values to real scale in meters.
For example, the factor 5000 is iused in the datasets [1]. A value of 5000 in the depth map corresponds to 1 meter.
Reference:
[0] KinectFusion: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ismar2011.pdf
[1] RGB-D SLAM Dataset and Benchmark : https://vision.in.tum.de/data/datasets/rgbd-dataset
Task: Implement a basic AR animation on unity using the color map and the camera pose given by the tracker.
The animation should consist on placing virtual objects on the points selected by the user (e.g. mouse interaction).
Unity modules:
@Mirjang support.
Ideally the Dataset
reader should store already read files and return values from the storage for each request.
The function to fetch a trajectory for a given frame is needed.
Task: Implement efficient Volumetric Fusion solution in order to be able to compute SDF in realtime.
I would recommend to check the current solutions (libraries, frameworks, etc) in order to adapt our current solultion to improve quality and performance.
References:
[0] CUDA TSDF: https://github.com/Scoobadood/TSDF
[1] Kinfu: http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php
[2] Open3D: http://www.open3d.org/docs/tutorial/Advanced/rgbd_integration.html#tsdf-volume-integration
[3] Elastic Reconstruction: https://github.com/qianyizh/ElasticReconstruction
Task: Implement a test (WindowTests) to check the point cloud previous to the volumetric fusion.
Similart to the Exercise 1 [0], we can generate a simple mesh from the point cloud. Then we can check the result in MeshLab [1]
@juanraul8 support.
References:
[0] Exercise 1: Camera Intrinsics, Back-projection, Meshes.
[1] MeshLab: http://www.meshlab.net/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.