nv-tlabs / flexicubes Goto Github PK

License: Other

Python 100.00%

flexicubes's Introduction

Flexible Isosurface Extraction for Gradient-Based Mesh Optimization (FlexiCubes)
_{Official PyTorch implementation}

FlexiCubes is a high-quality isosurface representation specifically designed for gradient-based mesh optimization with respect to geometric, visual, or even physical objectives. For more details, please refer to our paper and project page.

Highlights

Getting Started

The core functions of FlexiCubes are now in Kaolin starting from v0.15.0. See installation instructions here and API documentations here

The original code of the paper is still visible in flexicube.py.

Example Usage

Gradient-Based Mesh Optimization

We provide examples demonstrating how to use FlexiCubes for reconstructing unknown meshes through gradient-based optimization. Specifically, starting from randomly initialized SDF, we optimize the shape towards the reference mesh by minimizing their geometric difference, measured by multiview mask and depth losses. This workflow is a simplified version of nvdiffrec with code largely borrowed from the nvdiffrec GitHub. We use the same pipeline to conduct the analysis in Section 3 and the main experiments described in Section 5 of our paper. We provide a detailed tutorial in examples/optimization.ipynb, along with an optimization script in examples/optimize.py which accepts command-line arguments.

To run the examples, it is suggested to install the Conda environment as detailed below:

conda create -n flexicubes python=3.9
conda activate flexicubes
conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.3 -c pytorch
pip install imageio trimesh tqdm matplotlib torch_scatter ninja
pip install git+https://github.com/NVlabs/nvdiffrast/
pip install kaolin==0.15.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-1.12.0_cu113.html

Then download the dataset collected by Myles et al. as follows. We include one shape in 'examples/data/inputmodels/block.obj' if you want to test without downloading the full dataset.

cd examples
python download_data.py

After downloading the data, run shape optimization with the following example command:

python optimize.py --ref_mesh data/inputmodels/block.obj --out_dir out/block

You can find visualization and output meshes in the out/block. Below, we show the initial and final shapes during optimization, with the reference shape on the right.

To further demonstrate the flexibility of our FlexiCubes representation, which can accommodates both reconstruction objectives and regularizers defined on the extracted mesh, you can add a developability regularizer (proposed by Stein et al.) to the previous reconstruction pipeline to encourage fabricability from panels:

python optimize.py --ref_mesh data/inputmodels/david.obj --out_dir out/david_dev --develop_reg True  --iter=1250

Extract mesh from known signed distance field

While not its designated use case, our function can extract a mesh from a known Signed Distance Field (SDF) without optimization. Please refer to the tutorial found in examples/extraction.ipynb for details.

Tips for using FlexiCubes

Regularization losses:

We commonly use three regularizers in our mesh optimization pipelines, referenced in lines L104-L106 in examples/optimize.py. The weights of these regularizers should be scaled according to the your application objectives. Initially, it is suggested to employ low weights because strong regularization can hinder convergence. You can incrementally increase the weights if you notice artifacts appearing in the optimized meshes. Specifically:

The loss function at L104 helps to remove floaters in areas of the shape that are not supervised by the application objective, such as internal faces when using image supervision only.
The L_dev loss at L105 can be increased if you observe artifacts in flat areas, as illustrated in the image below.
Generally, the L1 regularizer on flexible weights at L106 does not have a significant impact during the optimization of a single shape. However, we found it to be effective in stabilizing training in generative pipelines such as GET3D.

Resolution of voxel grid vs. tetrahedral grid:

If you are switching from our previous work, DMTet, it's important to note the difference in grid resolution when compared to FlexiCubes. In both implementations, the resolution is defined by the edge length: a grid resolution of n means the grid edge length is 1/n for both the voxel and tetrahedral grids. However, a tetrahedral grid with a resolution of n contains only (n/2+1)³ grid vertices, in contrast to the (n+1)³ vertices in a voxel grid. Consequently, if you are switching from DMTet to FlexiCubes while maintaining the same resolution, you will notice not only a denser output mesh but also a substantial increase in computational cost. To align the triangle count in the output meshes more closely, we recommend adopting a 4:5 resolution ratio between the voxel grid and the tetrahedral grid. For instance, in our paper, 64³ FlexiCubes generate approximately the same number of triangles as 80³ DMTet.

Applications

FlexiCubes is now integrated into NVIDIA applications as a drop-in replacement for DMTet. You can visit their GitHub pages to see how FlexiCubes is used in advanced photogrammetry and 3D generative pipelines.

Extracting Triangular 3D Models, Materials, and Lighting From Images (nvdiffrec)

GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images

License

This work is made available under the Nvidia Source Code License.

For business inquiries, please visit our website and submit the form: NVIDIA Research Licensing.

Citation

@article{shen2023flexicubes,
author = {Shen, Tianchang and Munkberg, Jacob and Hasselgren, Jon and Yin, Kangxue and Wang, Zian 
        and Chen, Wenzheng and Gojcic, Zan and Fidler, Sanja and Sharp, Nicholas and Gao, Jun},
title = {Flexible Isosurface Extraction for Gradient-Based Mesh Optimization},
year = {2023},
issue_date = {August 2023},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {42},
number = {4},
issn = {0730-0301},
url = {https://doi.org/10.1145/3592430},
doi = {10.1145/3592430},
journal = {ACM Trans. Graph.},
month = {jul},
articleno = {37},
numpages = {16}
}

flexicubes's People

Contributors

Stargazers

Watchers

flexicubes's Issues

How to export tet-mesh?

thanks for your awesome work!
I noticed that in your paper you claim that flexicubes can extend to tet-mesh .I am interested in this .Could you tell me how to export a tet-mesh with flexicubes?

In flexicubes.py , the class of FlexiCubes there is a option to export tetmesh :

def __call__(self, x_nx3, s_n, cube_fx8, res, beta_fx12=None, alpha_fx8=None,
                 gamma_f=None, training=False, output_tetmesh=False, grad_func=None):

but when i turn on it and run the optimize.py ,I found the rasterize method for tet-mesh have not implemented yet.

    v_pos_clip = xfm_points(mesh.vertices.unsqueeze(0), mvp)  # Rotate it to camera coordinates
    rast, db = dr.rasterize(
        dr.RasterizeGLContext(), v_pos_clip, mesh.faces.int(), iter_res)

this caused the error in nvdiffrast.

def rasterize(glctx, pos, tri, resolution, ranges=None, grad_db=True):

So how can i export and optimize a tet-mesh .

looking forward to some advise .

some issues during implementation.

when i want to run the program show the following error, anyone who has experienced in this , I try my best and check the method in render.py as well.

Traceback (most recent call last):
File "C:\Users\admin\FlexiCubes-main\examples\optimize.py", line 88, in
mv, mvp = render.get_random_camera_batch(FLAGS.batch, iter_res=FLAGS.train_res, device=device)
ValueError: too many values to unpack (expected 2)

fractures after refinement

Thank you for your great work.

I've been attempting to apply your work to my custom 3D model (which was actually created using NeRF). It seems that the model has been refined effectively, but it exhibits strange fractures on its exterior. None of the models included in your dataset had these fractures after refinement. What could be the issue?

It seems that the random initialization of the signed distance function (SDF) occasionally results in meshes with fractures that persist until the FlexiCube refinement process concludes. Interestingly, these fractures weren't evident in the dataset you provided. I'm curious to understand why this disparity exists.

Approximate runtime with NVDiffRec

Hi, roughly how long does it take to reconstruct an object using flexicubes and nvdiffrec?

Neural-network based input

Hi， it is is pleasure to enjoy this wonderful work.
I have a question when i test for MLP-based gradient func in the "mesh_with_grad_v, mesh_with_grad_f, _ = fc(x_nx3, sdf_n, cube_fx8, res, grad_func=self.network.gradient)"; Do you have any suggestion.

Thank you very much.

The following is the result:

NN-based sdf may cause failure of surface extraction.

Hi, I realy enjoy this excellent work.

I found the NN-based sdf may be unstable in the initial training stage.

Specifically, I replaced the geometric parameters (sdf,deform,cube_weight tensor) in nvdiffrec with a NN-based structure:
query-->(pvcnn/triplane/voxel grid)-->MLP--> sdf,deform,cube_weight

If I randomly initiate the network, the mesh extraction will quickly fail because sdf predictions are all >0 or <0.

I have tried:

reduce learning rate,
pre-train using a known sphere sdf
change different network init method.

These methods did alleviate this problem, and successfully trained the network under certain random initialization. But it was very unstable (failure usually occurs during the first 100 steps).

As I am relatively new to this area, I would like to ask you for some advice about this. Do I need to reduce the weight of regularisation, or use point cloud supervision at the initial stage?

Thank you so much.

No texture in example?

as title

Which meshes are used for evaluation in the paper?

Dear FlexiCubes authors,
first of all thanks this great work! I wanted to reproduce your results in Table 2 of the paper and downloaded the dataset by Myles et al. However, in the paper you state that you only use 79 of these shapes for evaluation.
Could you provide a list of the exact ones used?

And how exactly do you compute the scores listed in that table? I found a script by NDC which gives similar numbers:
https://github.com/czq142857/NDC/blob/main/eval_100000.py
Is that the correct one?

Many Thanks!

Random background using Flexicubes

I find that random background color will appear in the training process if using flexicubes instead of DMTet, the rendering engine is nvdiffrast, Is that normal?

Flexicubes suddenly collapses in some intermediate training steps

Hi, thanks for your excellent work! We are training a image-to-3D reconstruction model using flexicubes as the 3D representation. We query MLPs to obtain the SDFs, colors, deformations and weights, and use Flexicubes to extract the surface. We adopt image-based supervisions to train the network, such as RGB and mask errors. Besides, we also use the regularizers of Flexicubes refering to the implementation of Get3D (including the SDF regularization loss, L_dev and weights regularization loss). However, we found that flexicubes always suddenly collapses at some intermediate training steps and starts to generate floating surfaces. The following picture visualizes such artifacts:

We have tried adjusting the learning rate but still cannot solve the problem. Could you please give some suggestions on the reason of this problem and how to solve it?

possible to use flexicubes on macos?

Is it possible to use flexicubes on macos?

I'm not sure how to install kaolin on a mac laptop, so I tried importing the flexicuves.py code. I get a kernel crash and "zsh: illegal hardware instruction".

thank you!

How to extract FC from a custom fixed SDF

Hi I'd like to extract Flexicube mesh from a fixed SDF. The ideal shape (marching cube from SDF using trimesh) should look like Fig.1, but using the example/extract_mesh.ipynb, I got a wrong mesh. I'm wondering where could be wrong? For example, how to give grad_func as torch.autograd in FC? My SDF is alpha from NeRF, so inside regions=1, outside regions<1.

How we can prepare the input data format

Thanks for amazing work, I want to just know about how we can prepare the input models like we have obj format how we can make the ffied format for that pictures.

FlexiCubes from existing mesh

Hello! First of all, congrats on the fantastic project!

Is it possible to create FlexiCubes from an existing mesh that I want to optimize further?

Thank you!

issues during implementation

Hi dear @frankshen07 hope you will be in good health.
i tried my best to employ this amazing project but stuck in one step as you can see the following error
(flexicube) C:\Users\admin\FlexiCubes\examples>python optimize.py --ref_mesh data/inputmodels/block.obj --out_dir out/block
Traceback (most recent call last):
File "C:\Users\admin\FlexiCubes\examples\optimize.py", line 88, in
mv, mvp = render.get_random_camera_batch(FLAGS.batch, iter_res=FLAGS.train_res, device=device)
ValueError: too many values to unpack (expected 2)

i tried with different version of kaolin=0.13.0 but not a able to tacke this issue. I use window 10 with RTXGEFORCE.
another question is that can we use this for single view reconstruction purpose. Is it possible or not.

The merge from Kaolin makes some bugs

I download the latest code and run as README.md .

Then this error occured:
Traceback (most recent call last):
  File "/home/dcy/code/FlexiCubes/FlexiCubes/examples/optimize.py", line 88, in <module>
    mv, mvp = render.get_random_camera_batch(FLAGS.batch, iter_res=FLAGS.train_res, device=device)
ValueError: too many values to unpack (expected 2)

here i think mv,mvp should change to camera (according to the document of Kaolin) But there are some other bugs . As I am not familiar with the entire code , i cant fix them all .So i make this Issue

How is the octree data structure progressively formulated in your case, could you please provide some example code regarding this adaptive optimization process?

Regards