Code Monkey home page Code Monkey logo

Comments (1)

weigert avatar weigert commented on July 18, 2024

That's exactly what I use it for in one of the examples in the main README (single image). It is not yet ready for multi-image, and also not as a simple CLI meshing tool. Sorry! There is still some development required:

A few considerations (single image):

  • In the triangulation, a single vertex might represent two (or more!) vertices in 3D space (i.e. occlusion is happening). To recognise this and spawn multiple vertices, a statistical argument has to be made based on plane goodness of fit for the surrounding triangles. This would lead to non-water tight meshes (which is true given occlusion).
  • RGB-D data can be non-dense and have holes. Therefore in some cases it makes sense to not triangulate areas without depth information, focusing on areas which do. I have almost completed adapting the triangulation algorithm to allow this, but it's very tricky. This would allow for non-rectangular boundaries, holes, etc. in the adaptive triangulation and not meshing where data is not available.

But this definitely currently works with single RGB-D images as you can see in the README.

Multiple is much more tricky:

  • A long video sequence can take any path through space and so we can't trivially build a global stitched image panorama
  • The images can be rectified using homographies given the current pose in the sequence. The stitched, rectified image could be used to form a large (non-rectangular boundary!) triangulation on a sphere, and use that as our mesh guess (it a position)
  • We then adaptively warp this mesh through the rectified image spheres as we move through the sequence. Note that the topological warping is not yet implemented either but is necessary for long image sequences, very tricky subject.
  • Finally, we have our camera positions, rectified image spheres at every position and warped meshes at every position. Using a registered point cloud, we can then fit the mesh to the data and use the images as the texture!

So overall the process would be:

  • For every pose, build rectified image spheres using homographies.
  • Triangulate the spheres and warp them adaptively to make sure that you get one, dynamic "mesh"
  • Pointcloud Registration on RGB-D data for a global 3D model. Downsample as far as you like.
  • Fit the mesh to the planes, apply statistics to duplicate the vertices in 3D

I currently don't have time to work on these aspect myself, but if you (or anyone) are interested in helping develop this concept feel free to let me know.

from t-pose.

Related Issues (1)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.