skhu101 / gauhuman Goto Github PK
View Code? Open in Web Editor NEWCode for our CVPR'2024 paper "GauHuman: Articulated Gaussian Splatting from Monocular Human Videos"
License: Other
Code for our CVPR'2024 paper "GauHuman: Articulated Gaussian Splatting from Monocular Human Videos"
License: Other
Hi,What an amazing job!
I have some questions. Do you have any plans to release the code for making the dataset? And more,for ZJU_mocap DATASETS?
And more, I observe the value of SEQUENCES in the bash : train_zju_mocap_refine is mismatched. Do I need to make some modifications?
Hi @skhu101 ,
Thx for the great advance.
May I ask, the extrinsic param R
(from annots.npy
) seems not to be used, but smpl_param.Rh
(from smpl_params/*.npy
) plays the role. May u provide more info on this?
GauHuman/scene/dataset_readers.py
Line 687 in 781539f
Thx & Rgds,
hello! I've successfully reproduced your project, but I'm not sure about one detail. Do you convert the SMPL vertices in the target space to the canonical space and then perform Gaussian optimization?
Hi,I want to project the world_src_pts onto 2D image plane,and I use the projection() function from your previous work SHERF,and the projected 2d points seems like wrong,could you help me solve this?Following is my code:
src_uv = projection(world_src_pts.reshape(bs, -1, 3), camera_R, camera_T, camera_K) # [bs, N, 6890, 3]
src_uv = src_uv.view(-1, *src_uv.shape[2:])
Here camera_K is the camera intrinsic'),camera_R is the ['R'] in smpl_param,camera_T is the ['Th'] in smpl_param.
Hello there,
Could you please tell me what the problem is?
I really follow your script, but When I run train_zju.sh, I got something wrong:
/data/xhd/envs/gaussian/lib/python3.7/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None
for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=VGG16_Weights.IMAGENET1K_V1
. You can also use weights=VGG16_Weights.DEFAULT
to get the most up-to-date weights.
warnings.warn(msg)
Loading model from: /data/xhd/envs/gaussian/lib/python3.7/site-packages/lpips/weights/v0.1/vgg.pth
Optimizing
Output folder: ./output/zju_mocap_refine/386_100_pose_correction_lbs_offset_split_clone_merge_prune [05/04 15:41:47]
Tensorboard not available: not logging progress [05/04 15:41:47]
Found annots.json file, assuming ZJU_MoCap_refine data set! [05/04 15:41:47]
Reading Training Transforms [05/04 15:41:47]
Reading Test Transforms [05/04 15:41:54]
Loading Training Cameras [05/04 15:42:20]
Loading Test Cameras [05/04 15:42:24]
Number of points at initialisation : 6890 [05/04 15:42:27]
Training progress: 0%| | 0/1200 [00:00<?, ?it/s]Traceback (most recent call last):
File "train.py", line 301, in
training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from)
File "train.py", line 104, in training
render_pkg = render(viewpoint_cam, gaussians, pipe, background)
File "/data/xhd/3D/GauHuman/gaussian_renderer/init.py", line 119, in render
cov3D_precomp = cov3D_precomp)
ValueError: not enough values to unpack (expected 4, got 2)
Training progress: 0%|
Dear authors! Thanks for your excellent work.
I have a question about the mask loss. I am confused about the code. Is the bkgd_mask and alpha mask used to compute the mask loss? Where is the ground truth mask? And the bound_mask=1 means the person region or others?
By the way, when inference, do we neek the mask as input?
Why the code can be run directly in vscode, but a RuntimeError occurs when debugging:
发生异常: RuntimeError
Error building extension 'knn': [1/3] /usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -std=c++17 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cu -o knn.cuda.o
FAILED: knn.cuda.o
/usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -std=c++17 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cu -o knn.cuda.o
/bin/sh: 1: /usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc: not found
[2/3] c++ -MMD -MF knn.o.d -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O2 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp -o knn.o
FAILED: knn.o
c++ -MMD -MF knn.o.d -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O2 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp -o knn.o
In file included from /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp:3:
/home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: cuda_runtime_api.h: No such file or directory
5 | #include <cuda_runtime_api.h>
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
File "/home/qinzhen/GauHuman/scene/gaussian_model.py", line 23, in
from knn_cuda import KNN
File "/home/qinzhen/GauHuman/scene/dataset_readers.py", line 28, in
from scene.gaussian_model import BasicPointCloud
File "/home/qinzhen/GauHuman/scene/init.py", line 16, in
from scene.dataset_readers import sceneLoadTypeCallbacks
File "/home/qinzhen/GauHuman/gaussian_renderer/init.py", line 15, in
from scene.gaussian_model import GaussianModel
File "/home/qinzhen/GauHuman/train.py", line 16, in
from gaussian_renderer import render, network_gui
RuntimeError: Error building extension 'knn': [1/3] /usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -std=c++17 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cu -o knn.cuda.o
FAILED: knn.cuda.o
/usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -std=c++17 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cu -o knn.cuda.o
/bin/sh: 1: /usr/local/cuda-11.8:/usr/local/cuda-11.8/bin/nvcc: not found
[2/3] c++ -MMD -MF knn.o.d -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O2 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp -o knn.o
FAILED: knn.o
c++ -MMD -MF knn.o.d -DTORCH_EXTENSION_NAME=knn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/TH -isystem /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda-11.8:/usr/local/cuda-11.8/include -isystem /home/qinzhen/anaconda3/envs/gauhuman/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O2 -c /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp -o knn.o
In file included from /home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/knn_cuda/csrc/cuda/knn.cpp:3:
/home/qinzhen/anaconda3/envs/gauhuman/lib/python3.8/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: cuda_runtime_api.h: No such file or directory
5 | #include <cuda_runtime_api.h>
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Excellent work!
But if I want to train gauhuman on the custom dataset, how can I get or train the SMPL model of the custom dataset?
# get tar_pts, smpl space source pose
A, R, Th, joints = get_transform_params_torch(self.SMPL_NEUTRAL, params, rot_mats=rot_mats)
self.s_A = A
A = torch.matmul(bweights, self.s_A.reshape(bs, joints_num, -1))
A = torch.reshape(A, (bs, -1, 4, 4))
can_pts = torch.matmul(A[..., :3, :3], query_pts[..., None]).squeeze(-1)
smpl_src_pts = can_pts + A[..., :3, 3]
transforms = torch.matmul(A[..., :3, :3], transforms)
if return_transl:
translation = torch.matmul(A[..., :3, :3], translation[..., None]).squeeze(-1) + A[..., :3, 3]
# transform points from the smpl space to the world space
R_inv = torch.inverse(R)
world_src_pts = torch.matmul(smpl_src_pts, R_inv) + Th
transforms = torch.matmul(R, transforms) #< --- THIS LINE
if return_transl:
translation = torch.matmul(translation, R_inv).squeeze(-1) + Th
@skhu101 can you tell me why is transforms post multiplied with R even though we are pre multiplying smpl_src_pts by R_inv ?
Hello,
Thanks for your excellent work. I encountered some issues when using my customized dataset, specifically related to the SMPL estimations. I compared SMPL vertices visualizations as shown in the figure above. They are for monocap/lan_images620_1300/images/00/0000.jpg
and from various sources:
When processed through GauHuman, only 1 and 2 yield correct results, while 3 and 4 consistently produce incorrect outcomes (predicted images are all black). One observed issue is the generation of erroneous 2D bound masks from 3 and 4. My speculation is that this is rooted in the differences in SMPL estimations. As shown in the figure, 1 and 2 are closely aligned, as are 3 and 4. However, the differences between these two groups are significant.
Given these observations, I’m seeking your insights –
I am also looking forward to any additional insights that could assist in generating accurate results from monocular SMPL estimations. Thank you for your time and assistance.
Hello! Your work is remarkable! I would like to ask if there are any differences in the camera parameters between the refined ZJUMocap dataset and the original ZJUMocap dataset. If I want to directly use the ZJUMocap dataset in the original Gaussian splatting generation work, which dataset should I use?
Hi, I run the training code successfully, but after 30 iterations' training it crashes. The error occurs in the loss.backward(), and it raise an error "RuntimeError: merge_sort: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered". I have checked the losses, and it seems normal. Can you help me? Thanks a lot.
Thanks a lot for sharing this fantastic work!
Question: I see that you have added the diff-gaussian-rasterization
as a submodule (https://github.com/skhu101/GauHuman/tree/main/submodules/diff-gaussian-rasterization) with the repo and not as a git submodule. I wonder if GauHuman uses a modified Gsplat rasterizer developed on top of the original repo https://github.com/graphdeco-inria/diff-gaussian-rasterization.
As mentioned in the title.
“Traceback (most recent call last):
File "C:\Users\Razer\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1666, in _run_ninja_build
subprocess.run(
File "C:\Users\Razer\anaconda3\envs\gauhuman\lib\subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
......
File "C:\Users\Razer\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1681, in _run_ninja_build
message += f": {error.output.decode()}" # type: ignore[union-attr]
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 1179: invalid continuation byte
Anyone else having the same issue?
I wanted to test the render speed (FPS) of the model, but the output of the code (~300) was inconsistent with the results in the paper (100+). I made the following changes to the dataset_reader.py
(558-562), but the result is still incorrect. How can I solve it?
pose_start = 0
# pose_interval = 30
pose_interval = 1
pose_num = 17
Hi, seems there should be zjumocap related data.py files in data folder?
Traceback (most recent call last):
File "train.py", line 16, in
from gaussian_renderer import render, network_gui
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\gaussian_renderer_init_.py", line 15, in
from scene.gaussian_model import GaussianModel
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene_init_.py", line 16, in
from scene.dataset_readers import sceneLoadTypeCallbacks
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene\dataset_readers.py", line 28, in
from scene.gaussian_model import BasicPointCloud
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene\gaussian_model.py", line 23, in
from knn_cuda import KNN
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\knn_cuda_init_.py", line 38, in
knn = load_cpp_ext("knn")
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\knn_cuda_init.py", line 26, in load_cpp_ext
ext = load(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1310, in load
return _jit_compile(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1738, in _jit_compile
return _import_module_from_library(name, build_directory, is_python_module)
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 2138, in _import_module_from_library
module = importlib.util.module_from_spec(spec)
ImportError: DLL load failed while importing knn: 找不到指定的模块。
Hi,I wonder how to use the interactive viewers from original 3DGS in gauhuman?
I was inspired by your work in the GauHuman paper where you retrained InstantAvatar on the ZJU-Mocap and MonoCap datasets. I am interested in replicating the results presented.
Could you please share the code used for this specific retraining? It would greatly assist in my research.
Can you tell me the command line in train_zju_mocap_refine.sh has -eval and eval_zju_mocap_refine.sh does not have the -eval grin?
I see the code without -eval, test_cam_infos = [] would be equal to null. Wouldn't there be no new perspective rendering.
After set the environment,I run this script "bash run_zju_mocap_refine.sh",and encounter:
Traceback (most recent call last):
File "train.py", line 16, in
from gaussian_renderer import render, network_gui
File "/home/yejr/Digital_Avater/GauHuman-main/gaussian_renderer/init.py", line 15, in
from scene.gaussian_model import GaussianModel
File "/home/yejr/Digital_Avater/GauHuman-main/scene/init.py", line 16, in
from scene.dataset_readers import sceneLoadTypeCallbacks
File "/home/yejr/Digital_Avater/GauHuman-main/scene/dataset_readers.py", line 33, in
from data.dna_rendering.dna_rendering_sample_code.SMCReader import SMCReader
ModuleNotFoundError: No module named 'data.dna_rendering'
How to deal with it?
Hi, Thanks for sharing the codes. Is there any way to use SMPLX (instead of SMPL) for human driving?
Hi,Id like to calculate signed distances for world_src_pts in gaussian_model.py,so I can know whether this point is inside body or not.Could you tell me how to achieve this roughly?
Hello!!
Have you test your framework on dataset People_SnapShot?
How do I make my own video dataset?
Hello, thanks for your excellent work!
I met a problem when "pip install -r requirement.txt". I was stuck on installing pickle5. I also tried conda install -c conda-forge pickle5 but it doesn't work. I wnat to know if anyone else meet the same problem.
The error information is as follows:
Building wheels for collected packages: pickle5
Building wheel for pickle5 (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [38 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-38
creating build\lib.win-amd64-cpython-38\pickle5
copying pickle5\pickle.py -> build\lib.win-amd64-cpython-38\pickle5
copying pickle5\pickletools.py -> build\lib.win-amd64-cpython-38\pickle5
copying pickle5_init_.py -> build\lib.win-amd64-cpython-38\pickle5
creating build\lib.win-amd64-cpython-38\pickle5\test
copying pickle5\test\pickletester.py -> build\lib.win-amd64-cpython-38\pickle5\test
copying pickle5\test\test_pickle.py -> build\lib.win-amd64-cpython-38\pickle5\test
copying pickle5\test\test_picklebuffer.py -> build\lib.win-amd64-cpython-38\pickle5\test
copying pickle5\test_init_.py -> build\lib.win-amd64-cpython-38\pickle5\test
running build_ext
building 'pickle5._pickle' extension
creating build\temp.win-amd64-cpython-38
creating build\temp.win-amd64-cpython-38\Release
creating build\temp.win-amd64-cpython-38\Release\pickle5
"D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -ID:\anaconda3\envs\gauhuman\include -ID:\anaconda3\envs\gauhuman\Include "-ID:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\include" "-ID:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\ATLMFC\include" "-ID:\Visual studio\2022\Community\VC\Auxiliary\VS\include" "-ID:\Windows Kits\10\include\10.0.22621.0\ucrt" "-ID:\Windows Kits\10\include\10.0.22621.0\um" "-ID:\Windows Kits\10\include\10.0.22621.0\shared" "-ID:\Windows Kits\10\include\10.0.22621.0\winrt" "-ID:\Windows Kits\10\include\10.0.22621.0\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /Tcpickle5/_pickle.c /Fobuild\temp.win-amd64-cpython-38\Release\pickle5/_pickle.obj
_pickle.c
"D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -ID:\anaconda3\envs\gauhuman\include -ID:\anaconda3\envs\gauhuman\Include "-ID:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\include" "-ID:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\ATLMFC\include" "-ID:\Visual studio\2022\Community\VC\Auxiliary\VS\include" "-ID:\Windows Kits\10\include\10.0.22621.0\ucrt" "-ID:\Windows Kits\10\include\10.0.22621.0\um" "-ID:\Windows Kits\10\include\10.0.22621.0\shared" "-ID:\Windows Kits\10\include\10.0.22621.0\winrt" "-ID:\Windows Kits\10\include\10.0.22621.0\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /Tcpickle5/picklebufobject.c /Fobuild\temp.win-amd64-cpython-38\Release\pickle5/picklebufobject.obj
picklebufobject.c
pickle5/picklebufobject.c(19): warning C4273: “PyPickleBuffer_FromObject”: dll 链接不一致
D:\anaconda3\envs\gauhuman\include\picklebufobject.h(18): note: 参见“PyPickleBuffer_FromObject”的前一个定义
pickle5/picklebufobject.c(38): warning C4273: “PyPickleBuffer_GetBuffer”: dll 链接不一致
D:\anaconda3\envs\gauhuman\include\picklebufobject.h(22): note: 参见“PyPickleBuffer_GetBuffer”的前一个定义
pickle5/picklebufobject.c(57): warning C4273: “PyPickleBuffer_Release”: dll 链接不一致
D:\anaconda3\envs\gauhuman\include\picklebufobject.h(24): note: 参见“PyPickleBuffer_Release”的前一个定义
pickle5/picklebufobject.c(208): warning C4273: “PyPickleBuffer_Type”: dll 链接不一致
D:\anaconda3\envs\gauhuman\include\picklebufobject.h(13): note: 参见“PyPickleBuffer_Type”的前一个定义
"D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\bin\HostX86\x64\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:D:\anaconda3\envs\gauhuman\libs /LIBPATH:D:\anaconda3\envs\gauhuman /LIBPATH:D:\anaconda3\envs\gauhuman\PCbuild\amd64 "/LIBPATH:D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\ATLMFC\lib\x64" "/LIBPATH:D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:D:\Windows Kits\10\lib\10.0.22621.0\ucrt\x64" "/LIBPATH:D:\Windows Kits\10\lib\10.0.22621.0\um\x64" /EXPORT:PyInit__pickle build\temp.win-amd64-cpython-38\Release\pickle5/_pickle.obj build\temp.win-amd64-cpython-38\Release\pickle5/picklebufobject.obj /OUT:build\lib.win-amd64-cpython-38\pickle5_pickle.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-cpython-38\Release\pickle5_pickle.cp38-win_amd64.lib
正在创建库 build\temp.win-amd64-cpython-38\Release\pickle5_pickle.cp38-win_amd64.lib 和对象 build\temp.win-amd64-cpython-38\Release\pickle5_pickle.cp38-win_amd64.exp
正在生成代码
已完成代码的生成
picklebufobject.obj : error LNK2005: PyPickleBuffer_GetBuffer 已经在 python38.lib(python38.dll) 中定义
build\lib.win-amd64-cpython-38\pickle5_pickle.cp38-win_amd64.pyd : fatal error LNK1169: 找到一个或多个多重定义的符号
error: command 'D:\Visual studio\2022\Community\VC\Tools\MSVC\14.38.33130\bin\HostX86\x64\link.exe' failed with exit code 1169
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pickle5
Running setup.py clean for pickle5
Failed to build pickle5
ERROR: Could not build wheels for pickle5, which is required to install pyproject.toml-based projects
Hi, thanks for the great work and sharing code! I tested GauHuman, it was mind blowing.
From your paper, i see that equation transfroms Gaussians from camera coordinate to ray coordinate transformation in Section 3.1.
But i have not heard about ray-coordinate, i cannot fully understand what is the ray coordinate, why we need that process. (May be the ray coordinate is 2D image space?)
Would you explain about the ray coordinate and where is the code lines about transformation process?
Great work.
Have you ever tested KL split/clone performance in MVS scenarios, like static scenes with multi-camera?
When I try it, I see that the number of 3dgs has increased, not decreased, and the running time has not changed significantly. Is there something wrong?
Firstly, I want to thank you for your nice work. However, I am puzzled by three questions when I try to run your code.
1. No such file in ... annots.npy
That means the ZJU-MoCap
Dataset I downloaded here following the guide of instant-nvr does not contain annnots.npy
in some folders. Then I just rename annots_new.npy
into annots.npy
to solve the error. I am not sure if this solution is right because I met other two problems after that. This is similar to #3. @skhu101 Can you tell me if it is OK?
2. ValueError: not enough values to unpack (expected 4, got 2)
This happened to my_377
after I renamed annots_new.npy
. The complete output of the terminal is as follows. How can I solve this?
/home/ubuntu/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off]
/home/ubuntu/.local/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/home/ubuntu/.local/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
warnings.warn(msg)
Loading model from: /home/ubuntu/.local/lib/python3.8/site-packages/lpips/weights/v0.1/vgg.pth
Optimizing
Output folder: ./output/zju_mocap_refine/my_377_100_pose_correction_lbs_offset_split_clone_merge_prune [12/07 12:47:50]
Found annots.json file, assuming ZJU_MoCap_refine data set! [12/07 12:47:50]
Reading Training Transforms [12/07 12:47:50]
Reading Test Transforms [12/07 12:47:58]
Loading Training Cameras [12/07 12:48:28]
Loading Test Cameras [12/07 12:48:31]
Number of points at initialisation : 6890 [12/07 12:48:33]
Training progress: 0%| | 0/1200 [00:00<?, ?it/s]Traceback (most recent call last):
File "train.py", line 301, in <module>
training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from)
File "train.py", line 104, in training
render_pkg = render(viewpoint_cam, gaussians, pipe, background)
File "/home/ubuntu/lyj/Project/GauHuman/gaussian_renderer/__init__.py", line 111, in render
rendered_image, radii, depth, alpha = rasterizer(
ValueError: not enough values to unpack (expected 4, got 2)
Training progress: 0%|
3. IndexError: index 0 is out of bounds for axis 0 with size 0
It is very interesting that some folders in ZJU_MoCap have this problem and others have problem 2. For example, my_386
has this problem and the complete information is as follows:
/home/ubuntu/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off]
/home/ubuntu/.local/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/home/ubuntu/.local/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
warnings.warn(msg)
Loading model from: /home/ubuntu/.local/lib/python3.8/site-packages/lpips/weights/v0.1/vgg.pth
Optimizing
Output folder: ./output/zju_mocap_refine/my_386_100_pose_correction_lbs_offset_split_clone_merge_prune [12/07 12:48:43]
Found annots.json file, assuming ZJU_MoCap_refine data set! [12/07 12:48:43]
Reading Training Transforms [12/07 12:48:43]
Traceback (most recent call last):
File "train.py", line 301, in <module>
training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from)
File "train.py", line 45, in training
scene = Scene(dataset, gaussians)
File "/home/ubuntu/lyj/Project/GauHuman/scene/__init__.py", line 52, in __init__
scene_info = sceneLoadTypeCallbacks["ZJU_MoCap_refine"](args.source_path, args.white_background, args.exp_name, args.eval)
File "/home/ubuntu/lyj/Project/GauHuman/scene/dataset_readers.py", line 704, in readZJUMoCapRefineInfo
train_cam_infos = readCamerasZJUMoCapRefine(path, train_view, white_background, split='train')
File "/home/ubuntu/lyj/Project/GauHuman/scene/dataset_readers.py", line 611, in readCamerasZJUMoCapRefine
image_path = os.path.join(path, ims[pose_index][view_index].replace('\\', '/'))
IndexError: index 0 is out of bounds for axis 0 with size 0
These problems may be common for your paper readers, so I am looking forward to your kind response!
Hi, thanks for your excellent work!
I would like to know if this method can generate a new pose. Or how about the generalization ability under new postures?
Hi,
I download the 'my_377' of ZJU-MoCap dataset from instant-nvr. But the folder only contains annots_new.npy
instead of annots.npy
in your code.
If I change the file name in the code and load annots_new.npy
, annots only contains 'cams' while 'ims' is empty.
Thanks a lot for your great work!
I visualize the initialization of gaussians (big pose xyz). I save these point clouds and view them in meshlab.
The result is as this, all points are spread out. I think it should be dense point clouds like the canonical space gaussian in Fig 2 of the paper.
Hi,I wander how to change the size of output image in your network,should I modify the gaussian rasterizer in CUDA?
Hi!
I encountered an issue while running the training code using the provided train_zju_mocap_refine.sh script on a single Nvidia RTX 4090 GPU. It appears that the rasterizer is demanding 72 GB GPU memory.
I'm reaching out to inquire if any specific parameters within the script or configuration can be adjusted to reduce the GPU memory requirements to a more manageable level.
Hi,
I met a problem when "bash train_zju_mocap_refine.sh". The combination is gcc12.3+cuda11.8+ninja1.10.2.4+pytorch2.0.0.
The error information is as follows:
Traceback (most recent call last):
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1893, in _run_ninja_build
subprocess.run(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "train.py", line 16, in
from gaussian_renderer import render, network_gui
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\gaussian_renderer_init_.py", line 15, in
from scene.gaussian_model import GaussianModel
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene_init_.py", line 16, in
from scene.dataset_readers import sceneLoadTypeCallbacks
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene\dataset_readers.py", line 28, in
from scene.gaussian_model import BasicPointCloud
File "C:\Users\Mr.Lin\Desktop\PROJECT\gauhuman\scene\gaussian_model.py", line 23, in
from knn_cuda import KNN
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\knn_cuda_init_.py", line 38, in
knn = load_cpp_ext("knn")
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\knn_cuda_init.py", line 26, in load_cpp_ext
ext = load(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1284, in load
return _jit_compile(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1509, in _jit_compile
_write_ninja_file_and_build_library(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1624, in _write_ninja_file_and_build_library
_run_ninja_build(
File "C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\utils\cpp_extension.py", line 1909, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'knn': [1/1] "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64/link.exe" knn.o knn.cuda.o /nologo /DLL c10.lib c10_cuda.lib torch_cpu.lib torch_cuda.lib -INCLUDE:?warp_size@cuda@at@@yahxz torch.lib /LIBPATH:C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\lib torch_python.lib /LIBPATH:C:\Users\Mr.Lin\anaconda3\envs\gauhuman\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64" cudart.lib /out:knn.pyd
FAILED: knn.pyd
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64/link.exe" knn.o knn.cuda.o /nologo /DLL c10.lib c10_cuda.lib torch_cpu.lib torch_cuda.lib -INCLUDE:?warp_size@cuda@at@@yahxz torch.lib /LIBPATH:C:\Users\Mr.Lin\anaconda3\envs\gauhuman\lib\site-packages\torch\lib torch_python.lib /LIBPATH:C:\Users\Mr.Lin\anaconda3\envs\gauhuman\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64" cudart.lib /out:knn.pyd
Creating library knn.lib and object knn.exp
knn.o : error LNK2019: unresolved external symbol "public: long * __cdecl at::TensorBase::data_ptr(void)const " (??$data_ptr@J@TensorBase@at@@QEBAPEAJXZ) referenced in function "public: long * __cdecl at::Tensor::data(void)const " (??$data@J@Tensor@at@@QEBAPEAJXZ)
knn.pyd : fatal error LNK1120: 1 unresolved externals
ninja: build stopped: subcommand failed.
Hi,I want to render all the resulting images with a fixed camera,
Here is my code, i just change the parameters about the camera
# lights is the fixed camera
if lights is not None:
# Set up rasterization configuration
tanfovx = math.tan(lights[0].FoVx * 0.5)
tanfovy = math.tan(lights[0].FoVy * 0.5)
raster_settings = GaussianRasterizationSettings(
image_height=int(lights[0].height),# change
image_width=int(lights[0].width),# change
tanfovx=tanfovx, # change
tanfovy=tanfovy,
bg=bg_color,
scale_modifier=scaling_modifier,
viewmatrix=lights[0].viewmatrix,# change
projmatrix=lights[0].projmatrix,# change
sh_degree=pc.active_sh_degree,
campos=lights[0].position,# change
prefiltered=False,
debug=pipe.debug
)
else:
# Set up rasterization configuration
tanfovx = math.tan(viewpoint_camera.FoVx * 0.5)
tanfovy = math.tan(viewpoint_camera.FoVy * 0.5)
raster_settings = GaussianRasterizationSettings(
image_height=int(viewpoint_camera.image_height),
image_width=int(viewpoint_camera.image_width),
tanfovx=tanfovx,
tanfovy=tanfovy,
bg=bg_color,
scale_modifier=scaling_modifier,
viewmatrix=viewpoint_camera.world_view_transform,
projmatrix=viewpoint_camera.full_proj_transform,
sh_degree=pc.active_sh_degree,
campos=viewpoint_camera.camera_center,
prefiltered=False,
debug=pipe.debug
)
rasterizer = GaussianRasterizer(raster_settings=raster_settings)
and i got some missing in results
could you tell me how to deal with it?
@skhu101 can you share for how many iterations has the final model for zju_mocap_refine data set been trained that is being used in the paper ? Is it 1200 iterations as mentioned in the train_zju_mocap_refine.sh file ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.