Make ComfyUI generates 3D assets as good & convenient as it generates image/video!
This is an extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc.) using cutting edge algorithms (3DGS, NeRF, etc.) and models (InstantMesh, CRM, TripoSR, etc.)
-
For use case please check Example Workflows. [Last update: 02/06/2024]
- Note: you need to put Example Inputs Files & Folders under ComfyUI Root Directory\ComfyUI\input folder before you can run the example workflow
- tripoSR-layered-diffusion workflow by @Consumption
-
Era3D Diffusion Model: pengHTYX/Era3D
- Single image to 6 multi-view images & normal maps with resulution: 512X512
- Note: you need at least 16GB vram to run this model
-
InstantMesh Reconstruction Model: TencentARC/InstantMesh
- Sparse multi-view images with white background to 3D Mesh with RGB texture
- Works with arbitrary MVDiffusion models (Probably works best with Zero123++, but also works with CRM MVDiffusion model)
2024-05-23.17-35-22.mp4
2024-05-23.17-32-31.mp4
-
Zero123++: SUDO-AI-3D/zero123plus
- Single image to 6 view images with resulution: 320X320
-
CRM: thu-ml/CRM
-
Three stages pipeline:
- Single image to 6 view images (Front, Back, Left, Right, Top & Down)
- Single image & 6 view images to 6 same views CCMs (Canonical Coordinate Maps)
- 6 view images & CCMs to 3D mesh
-
Note: For low vram pc, if you can't fit all three models for each stages into your GPU memory, then you can divide those three stages into different comfy workflow and run them separately
2024-03-12.22-05-10.mp4
-
-
TripoSR: VAST-AI-Research/TripoSR | ComfyUI-Flowty-TripoSR
-
Generate NeRF representation and using marching cube to turn it into 3D mesh
2024-03-05.22-38-36.mp4
-
-
Wonder3D: xxlong0/Wonder3D
- Generate spatial consistent 6 views images & normal maps from a single image
- Generate spatial consistent 6 views images & normal maps from a single image
-
Large Multiview Gaussian Model: 3DTopia/LGM
-
Enable single image to 3D Gaussian in less than 30 seconds on a RTX3080 GPU, later you can also convert 3D Gaussian to mesh
2024-02-08.23-36-31.mp4
-
-
Triplane Gaussian Transformers: VAST-AI-Research/TriplaneGaussian
-
Enable single image to 3D Gaussian in less than 10 seconds on a RTX3080 GPU, later you can also convert 3D Gaussian to mesh
2024-02-08.23-57-37.mp4
-
-
Preview 3DGS and 3D Mesh: 3D Visualization inside ComfyUI:
-
Using gsplat.js and three.js for 3DGS & 3D Mesh visualization respectively
-
Custumizable background base on JS library: mdbassit/Coloris
2024-02-04.19-20-17.mp4
-
-
Stack Orbit Camera Poses: Automatically generate all range of camera pose combinations
-
You can use it to conditioning the StableZero123 (You need to Download the checkpoint first), with full range of camera poses in one prompt pass
-
You can use it to generate the orbit camera poses and directly input to other 3D process node (e.g. GaussianSplatting and BakeTextureToMesh)
-
Example usage:
-
Coordinate system:
- Azimuth: In top view, from angle 0 rotate 360 degree with step -90 you get (0, -90, -180/180, 90, 0), in this case camera rotates clock-wise, vice versa.
- Elevation: 0 when camera points horizontally forward, pointing down to the ground is negitive angle, vice versa.
-
-
FlexiCubes: nv-tlabs/FlexiCubes
-
Multi-View depth & mask (optional normal maps) as inputs
-
Export to 3D Mesh
-
Usage guide:
- voxel_grids_resolution: determine mesh resolution/quality
- depth_min_distance depth_max_distance : distance from object to camera, object parts in the render that is closer(futher) to camera than depth_min_distance(depth_max_distance) will be rendered with pure white(black) RGB value 1, 1, 1(0, 0, 0)
- mask_loss_weight: Control the silhouette of reconstrocted 3D mesh
- depth_loss_weight: Control the shape of reconstrocted 3D mesh, this loss will also affect the mesh deform detail on the surface, so results depends on quality of the depth map
- normal_loss_weight: Optional. Use to refine the mesh deform detail on the surface
- sdf_regularizer_weight: Helps to remove floaters in areas of the shape that are not supervised by the application objective, such as internal faces when using image supervision only
- remove_floaters_weight: This can be increased if you observe artifacts in flat areas
- cube_stabilizer_weight: This does not have a significant impact during the optimization of a single shape, however it helps to stabilizing training in somecases
2024-04-12-16-21-24.mp4
-
-
Instant NGP: nerfacc
- Multi-View images as inputs
- Export to 3D Mesh using marching cubes
-
3D Gaussian Splatting
- Improved Differential Gaussian Rasterization
- Better Compactness-based Densification method from Gsgen,
- Support initialize gaussians from given 3D mesh (Optional)
- Support mini-batch optimazation
- Multi-View images as inputs
- Export to standard 3DGS .ply format supported
-
Gaussian Splatting Orbit Renderer
- Render 3DGS to images sequences or video, given a 3DGS file and camera poses generated by Stack Orbit Camera Poses node
-
Mesh Orbit Renderer
- Render 3D mesh to images sequences or video, given a mesh file and camera poses generated by Stack Orbit Camera Poses node
-
Fitting_Mesh_With_Multiview_Images
- Bake Multi-View images into UVTexture of given 3D mesh using Nvdiffrast, supports:
- Export to .obj, .ply, .glb
-
NeuS
- Fit a coarse mesh from sparse multi-view images & normal maps, as little as 4 to 6 views, pretty good at reconstruct the shape from reference images but texture lacking details.
-
Deep Marching Tetrahedrons
- Allow convert 3DGS .ply file to 3D mesh
Note: I didn't spent time to turn the hyperprameters yet, the result will be improved in the future!
- Allow convert 3DGS .ply file to 3D mesh
-
Save & Load 3D file
- .obj, .ply, .glb for 3D Mesh
- .ply for 3DGS
-
Switch Axis for 3DGS & 3D Mesh
- Since different algorithms likely use different coordinate system, so the ability to re-mapping the axis of coordinate is crucial for passing generated result between differnt nodes.
-
Customizable system config file
- Custom clients IP address
-
Add DMTet algorithm to allow conversion from points cloud(Gaussian/.ply) to mesh (.obj, .ply, .glb)
-
Add interactive 3D UI inside ComfuUI to visulaize training and generated results for 3D representations
-
Add a new node to generate renderer image sequence given a 3D gaussians and orbit camera poses (So we can later feed it to the differentiable renderer to bake it onto a given mesh)
-
Integrate LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation
-
Add camera pose estimation from raw multi-views images
-
Add & Improve a few best MVS algorithms (e.g instant-ngp, NeuS2, GaussianPro, etc.)
-
Improve 3DGS/Nerf to Mesh conversion algorithms:
- Support to training DMTet with images(RGB, Alpha, Normal Map)
- Find better methods to converts 3DGS or Points Cloud to Mesh (Normal maps reconstruction maybe?)
-
Add a general SDS/ISM Optimization algorithm to allow training 3D representations with diffusion model
- Need to do some in-depth research on Interval Score Matching (ISM), since math behind it makes perfect sense and also there are so many ways we could improve upon the result obtained from LucidDreamer
- On Hold since runtime cost to generate an is too big (3+hours for an average RTX GPU like 3080)
[IMPORTANT!!!]
Currently this package is only been tested in following setups:
- Windows 10/11 (Tested on my laptop)
- Ubuntu 23.10 (Tested by @watsieboi)
- ComfyUI python_embed/Miniconda/Conda Python 3.11.x
- Torch version >= 2.1.2+cu121
Assume you have already downloaded ComfyUI & Configed your CUDA environment.
Currently support: (python3.10/3.11/3.12 cuda12.1)
First install Visual Studio Build Tools 2022/2019 with Workloads: Desktop development with C++ (There are a few JIT torch cpp extension that builds in runtime)
- Alternatively, according to @doctorpangloss, you can setup the c++/cuda build environments in windows by using chocolatey
Go to the Comfy3D root directory: ComfyUI Root Directory\ComfyUI\custom_nodes\ComfyUI-3D-Pack and run:
# Run .bat with python version corresponding to the version of your ComfyUI python environment
# install_windows_portable_win_py310_cu121.bat
install_windows_portable_win_py311_cu121.bat
# install_windows_portable_win_py312_cu121.batNote: In some edge cases Miniconda fails but Anaconda could fix the issue
First download Miniconda (One of the best way to manage a clean and separated python envirments)
Then running following commands to setup the Miniconda environment for ComfyUI:
# Go to your Your ComfyUI root directory, for my example:
cd C:\Users\reall\Softwares\ComfyUI_windows_portable
conda create -p ./python_miniconda_env/ComfyUI python=3.11
# conda will tell what command to use to activate the env
conda activate C:\Users\reall\Softwares\ComfyUI_windows_portable\python_miniconda_env\ComfyUI
# update pip
python -m pip install --upgrade pip
# You can using following command to installing CUDA only in the miniconda environment you just created if you don't want to donwload and install it manually & globally:
# conda install -c "nvidia/label/cuda-12.1.0" cuda-toolkit
# Install the main packahes
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r ./ComfyUI/requirements.txt
# Then go to ComfyUI-3D-Pack directory under the ComfyUI Root Directory\ComfyUI\custom_nodes for my example is:
cd C:\Users\reall\Softwares\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-3D-Pack- Alternatively you can check this tutorial: Installing ComfyUI with Miniconda On Windows and Mac
Go to the Comfy3D root directory: ComfyUI Root Directory\ComfyUI\custom_nodes\ComfyUI-3D-Pack and run:
install_miniconda.batJust in case install_miniconda.bat may not working in your OS, you could also run the following commands under the same directory: (Works with Linux & macOS)
pip install -r requirements.txt
pip install -r requirements_post.txtPlus:
- For those who want to run it inside Google Colab, you can check the install instruction from @lovisdotio
- You can find some of the pre-build wheels for Linux here: remsky/ComfyUI3D-Assorted-Wheels
Gpu support during Docker build time is required to install all requirenents.
On Linux host you could setup nvidia-container-runtime. On Windows
it is quite different and not checked at moment.
-
Install nvidia-container-runtime:
sudo apt-get install nvidia-container-runtime
-
Edit/create the /etc/docker/daemon.json with content:
{ "runtimes": { "nvidia": { "path": "/usr/bin/nvidia-container-runtime", "runtimeArgs": [] } }, "default-runtime": "nvidia" } -
Restart docker daemon:
sudo systemctl restart docker
Finally build and run docker container with:
docker build -t comfy3d . && docker run --rm -it -p 8188:8188 --gpus all comfy3dCopy the files inside folder __New_ComfyUI_Bats to your ComfyUI root directory, and double click run_nvidia_gpu_miniconda.bat to start ComfyUI!
- Alternatively you can just activate the Conda env:
python_miniconda_env\ComfyUI, and go to your ComfyUI root directory then run commandpython ./ComfyUI/main.py
- OpenGL world & camera coordinate system:
World Camera
+y up target
| | /
| | /
|______+x |/______right
/ /
/ /
/ /
+z forward
elevation: in (-90, 90), from +y to -y is (-90, 90)
azimuth: in (-180, 180), from +z to +x is (0, 90)
- Wonder3D world & camera coordinate system:
- Three.js coordinate system: (z-axis is pointing towards you and is coming out of the screen)
- If you encounter OpenGL errors (e.g.,
[F glutil.cpp:338] eglInitialize() failed), then setforce_cuda_rasterizeto true on corresponding node - If after the installation, your ComfyUI get stucked at starting or running, you could following the instruction in following link to solve the problem: Code Hangs Indefinitely When Evaluating Neuron Models on GPU
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent