An extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc) using cutting edge algorithms (3DGS, NeRF, Differentiable Rendering, SDS/VSD Optimization, etc.)
ps: I'll show some generated result soon
[IMPORTANT!!!]
Currently this package is only been tested in following setups:
- Windows 10/11
- Miniconda/Conda Python 3.11.7
- I tried install this package with ComfyUI embed python env first, but I can't find a way to build CUDA related libraries, e.g. diff-gaussian-rasterization, nvdiffrast, simple-knn.
- Torch version: 2.1.2+cu121/V.2.1.2+cu118
-
For use case please check example workflows
- Note: you need to put Example Inputs Files & Folders under ComfyUI Root Directory\ComfyUI\input folder before you can run the example workflow
-
Triplane Gaussian Transformers:
- Enable single image to 3D Gaussian, then you can use Deep Marching Tetrahedrons node to convert it to mesh
picacho_gs_render.mp4
-
Stack Orbit Camera Poses node to automatically generate all range of camera pose combinations
-
You can use it to conditioning the StableZero123 (You need to Download the checkpoint first), with full range of camera poses in one prompt pass
-
You can use it to generate the orbit camera poses and directly input to other 3D process node (e.g. GaussianSplatting and BakeTextureToMesh)
-
Example usage:
-
-
Load 3D file (.obj, .ply, .glb)
- Also have node that support save and load 3DGS .ply file
-
3D Gaussian Splatting, with:
- Improved Differential Gaussian Rasterization
- Better Compactness-based Densification method from Gsgen,
- Support initialize gaussians from given 3D mesh (Optional)
- Support mini-batch optimazation
- Multi-View images as inputs
- Export to .ply support
-
Bake Multi-View images into UVTexture of given 3D mesh using Nvdiffrast, supports:
- Export to .obj, .ply, .glb
-
Deep Marching Tetrahedrons
- Allow convert 3DGS .ply file to 3D mesh
Note: I didn't spent time to turn the hyperprameters yet, the result will be improved in the future!
- Allow convert 3DGS .ply file to 3D mesh
-
Add DMTet algorithm to allow convertion from points cloud(Gaussian/.ply) to mesh (.obj, .ply, .glb)
-
Add interactive 3D UI inside ComfuUI to visulaize training and generated results for 3D representations
-
Improve DMTet result & add support to training it with images(RGB, Alpha, Normal Map)
-
Add a general SDS/VSD Optimization algorithm to allow training 3D representations with diffusion model, The real fun begins here ;)
-
Add a few best Nerf algorithms (No idea yet, instant-ngp maybe?)
Assume you have already downloaded ComfyUI
First download Miniconda (One of the best way to manage a clean and separated python envirments)
- Alternatively you can check this tutorial: Installing ComfyUI with Miniconda On Windows and Mac
# Go to your Your ComfyUI root directory, for my example:
cd C:\Users\reall\Softwares\ComfyUI_windows_portable
conda create -p ./python_miniconda_env/ComfyUI python=3.11
# conda will tell what command to use to activate the env
conda activate C:\Users\reall\Softwares\ComfyUI_windows_portable\python_miniconda_env\ComfyUI
# This package also works with cu118
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r ./ComfyUI/requirements.txt
# Then go to ComfyUI-3D-Pack directory under the .\ComfyUI\custom_nodes for my example is:
cd C:\Users\reall\Softwares\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-3D-Pack
# Finally you can double click following .bat script or run it in CLI:
install.bat
-
Install Plan B:
Just in case
install.bat
may not working in your PC, you could also run the following commands under this package's root directory:# First make sure the Conda env: python_miniconda_env\ComfyUI is activated, then go to Go to ComfyUI Root Directory\ComfyUI\custom_nodes\ComfyUI-3D-Pack and: pip install -r requirements.txt git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization pip install ./diff-gaussian-rasterization pip install ./simple-knn git clone --recursive https://github.com/NVlabs/nvdiffrast/` pip install ./nvdiffrast # Install pointnet2_ops cd tgs/models/snowflake/pointnet2_ops_lib && python setup.py install && cd ../../../../ # Install pytorch_scatter pip install git+https://github.com/rusty1s/pytorch_scatter.git # Install pytorch3d pip install git+https://github.com/facebookresearch/pytorch3d.git@stable
Copy the files inside folder __New_ComfyUI_Bats to your ComfyUI root directory, and double click run_nvidia_gpu_miniconda.bat to start ComfyUI!
- Alternatively you can just activate the Conda env: python_miniconda_env\ComfyUI, and go to your ComfyUI root directory then run command
python ./ComfyUI/main.py
- The world & camera coordinate system is the same as OpenGL:
World Camera
+y up target
| | /
| | /
|______+x |/______right
/ /
/ /
/ /
+z forward
elevation: in (-90, 90), from +y to -y is (-90, 90)
azimuth: in (-180, 180), from +z to +x is (0, 90)
- If you encounter OpenGL errors (e.g.,
[F glutil.cpp:338] eglInitialize() failed
), then setforce_cuda_rasterize
to true on corresponding node