vast-ai-research / triposr Goto Github PK

I found that the result of run.py is different from the result displayed in huggingface.
Are there any parameters that are different?

Following are these 2 different results.

Blender only support std .obj format with unwrapped uv in a texture.png file. Is it possible to use trimesh to achieve this?

Traceback (most recent call last):
File "D:\3d\TripoSR\run.py", line 154, in
meshes = model.extract_mesh(scene_codes)
File "D:\3d\TripoSR\tsr\system.py", line 185, in extract_mesh
v_pos, t_pos_idx = self.isosurface_helper(-(density - threshold))
File "D:\3d\TripoSR\env\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\3d\TripoSR\env\lib\site-packages\torch\nn\modules\module.py", line 1520, in call_impl
return forward_call(*args, **kwargs)
File "D:\3d\TripoSR\tsr\models\isosurface.py", line 45, in forward
v_pos, t_pos_idx = self.mc_func(level.detach(), 0.0)
File "D:\3d\TripoSR\env\lib\site-packages\torchmcubes_init.py", line 12, in marching_cubes
return mc.mcubes_cuda(vol, thresh)
AttributeError: module 'torchmcubes_module' has no attribute 'mcubes_cuda'. Did you mean: 'mcubes_cpu'?

tried to re-install and still getting error,using windows

Can it run under CPU? Is there a way to make the generated model more refined

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU 0 has a total capacity of 23.67 GiB of which 2.72 GiB is free. Including non-PyTorch memory, this process has 20.58 GiB memory in use. Of the allocated memory 20.20 GiB is allocated by PyTorch, and 68.42 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

https://link.hackersthegame.com/simlink.php?p=66110940&t=18491787&c=17415&q=881&s=385

great!!! Great!!! GREAT!!!

please don't close this, keep a few days. :-)

Engaged in 3D reconstruction and generation for a long time, although not an expert, but definitely experienced. I have read thousands of papers and run hundreds of codes. This is the first and currently the only one that I genuinely want to give a thumbs up to.

长期从事3D重构和生成，虽然不是高手，但绝对很有经验了。读了上千篇论文，跑了数百个代码。这是第一也是目前唯一一篇，我发自内心想点赞的。

这篇是唯一，在靠近我理想中那种：“终极视觉算法”的，其中比较主要的一段功能，而且有点质量了。 后面慢慢分享。

(代码已经非常干净了；超过99%的开源；但还是不够干净，我自己修改了一份更清爽的，包含训练。)

I'm trying to set --model-save-format to .glb, but the exported model have no color.

(venv) D:\TripoSR>python gradio_app.py
D:\TripoSR\venv\lib\site-packages\transformers\utils\generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
D:\TripoSR\venv\lib\site-packages\transformers\utils\generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
D:\TripoSR\tsr\models\transformer\attention.py:629: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)
hidden_states = F.scaled_dot_product_attention(
Traceback (most recent call last):
File "D:\TripoSR\venv\lib\site-packages\gradio\queueing.py", line 495, in call_prediction
output = await route_utils.call_process_api(
File "D:\TripoSR\venv\lib\site-packages\gradio\route_utils.py", line 235, in call_process_api
output = await app.get_blocks().process_api(
File "D:\TripoSR\venv\lib\site-packages\gradio\blocks.py", line 1627, in process_api
result = await self.call_function(
File "D:\TripoSR\venv\lib\site-packages\gradio\blocks.py", line 1173, in call_function
prediction = await anyio.to_thread.run_sync(
File "D:\TripoSR\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "D:\TripoSR\venv\lib\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "D:\TripoSR\venv\lib\site-packages\anyio_backends_asyncio.py", line 851, in run
result = context.run(func, *args)
File "D:\TripoSR\venv\lib\site-packages\gradio\utils.py", line 690, in wrapper
response = f(*args, **kwargs)
File "D:\TripoSR\gradio_app.py", line 60, in generate
mesh = model.extract_mesh(scene_codes)[0]
File "D:\TripoSR\tsr\system.py", line 185, in extract_mesh
v_pos, t_pos_idx = self.isosurface_helper(-(density - threshold))
File "D:\TripoSR\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\TripoSR\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in call_impl
return forward_call(*args, **kwargs)
File "D:\TripoSR\tsr\models\isosurface.py", line 45, in forward
v_pos, t_pos_idx = self.mc_func(level.detach(), 0.0)
File "D:\TripoSR\venv\lib\site-packages\torchmcubes_init.py", line 12, in marching_cubes
return mc.mcubes_cuda(vol, thresh)
AttributeError: module 'torchmcubes_module' has no attribute 'mcubes_cuda'. Did you mean: 'mcubes_cpu'?

I have 4gb Vram and it gives me out of memory error. How to switch to cpu only?

INFO - Initializing model finished in 16985.25ms.
INFO - Processing images ...
INFO - Processing images finished in 1468.80ms.
INFO - Running image 1/1 ...

Hey,

I was wondering whether it might be possible to improve the reconstruction performance by leveraging multiple images from different view angles, maybe in a similar fashion as it is done in One-2-3-45.

If you think it might be possible, I would be thankful for any pointer to the right direction.

Btw great work!

Is it possible to increase the Marching Cubes Resolution above 320 by changing the variable "maximum = 320" in the file "gradio_app.py"? Or would it cause an error?

when i export the 3d model as obj and import into blender the 3d model dont have texture, how we can get also the textures?

2024-03-05 17:28:53,505 - INFO - Initializing model finished in 16985.25ms.
2024-03-05 17:28:53,505 - INFO - Processing images ...
2024-03-05 17:28:54,974 - INFO - Processing images finished in 1468.80ms.
2024-03-05 17:28:54,974 - INFO - Running image 1/1 ...
2024-03-05 17:28:54,974 - INFO - Running model ...
Segmentation fault (core dumped)

I set marching cubes resolution to highest(320), there is significant jagged edges.

The model outputs density fields, I think using distance field will get better mesh quality?

After installing TripoSR

Per the previous instructions #40

I could not install gradio to test the GUI

Building wheels for collected packages: ffmpy
  Building wheel for ffmpy (setup.py) ... done
  Created wheel for ffmpy: filename=ffmpy-0.3.2-py3-none-any.whl size=5606 sha256=69e1845f85b6407bcb398b06d44c762c677705c4c4bbb2d473baad23d1d4e878
  Stored in directory: c:\users\user\appdata\local\pip\cache\wheels\8a\63\a8\fad16a1c5c990569a478f40782ce543788ddac224288008635
Successfully built ffmpy
Installing collected packages: pytz, pydub, ffmpy, websockets, tzdata, toolz, tomlkit, sniffio, six, shellingham, semantic-version, ruff, python-multipart, pyparsing, pygments, pydantic-core, orjson, mdurl, kiwisolver, importlib-resources, h11, fonttools, cycler, contourpy, click, annotated-types, aiofiles, uvicorn, typer, python-dateutil, pydantic, markdown-it-py, huggingface-hub, httpcore, anyio, starlette, rich, pandas, matplotlib, httpx, gradio-client, fastapi, altair, gradio
  Attempting uninstall: huggingface-hub
    Found existing installation: huggingface-hub 0.17.3
    Uninstalling huggingface-hub-0.17.3:
      Successfully uninstalled huggingface-hub-0.17.3
**ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tokenizers 0.14.1 requires huggingface-hub<0.18,>=0.16.4, but you have huggingface-hub 0.21.4 which is incompatible.**
Successfully installed aiofiles-23.2.1 altair-5.2.0 annotated-types-0.6.0 anyio-4.3.0 click-8.1.7 contourpy-1.2.0 cycler-0.12.1 fastapi-0.110.0 ffmpy-0.3.2 fonttools-4.49.0 gradio-4.20.1 gradio-client-0.11.0 h11-0.14.0 httpcore-1.0.4 httpx-0.27.0 huggingface-hub-0.21.4 importlib-resources-6.1.3 kiwisolver-1.4.5 markdown-it-py-3.0.0 matplotlib-3.8.3 mdurl-0.1.2 orjson-3.9.15 pandas-2.2.1 pydantic-2.6.3 pydantic-core-2.16.3 pydub-0.25.1 pygments-2.17.2 pyparsing-3.1.2 python-dateutil-2.9.0.post0 python-multipart-0.0.9 pytz-2024.1 rich-13.7.1 ruff-0.3.1 semantic-version-2.10.0 shellingham-1.5.4 six-1.16.0 sniffio-1.3.1 starlette-0.36.3 tomlkit-0.12.0 toolz-0.12.1 typer-0.9.0 tzdata-2024.1 uvicorn-0.27.1 websockets-11.0.3

How to change the path here to a relative path and where to change it

先给作者大牛们点赞，非常牛，是真的非常牛。

虽然这已经是开源中最好效果的了，但还是说说我的测试反馈和个人理解:

1. 先验不够。 相比于多（甚至很多）视图的暴力拟合重构类算法，从终极视觉系统最终需要来看，还是单张条件生成（不排斥支持多张）的做法更是未来。 但是：
   1.1 现在无论是TripoSR，还是LRM，这种大概1.5G的主模型（不含ViT图像特征预处理模型），能够“记忆”的先验远远不够。直白的说：1.5G的东西记不下这个世界，网友们上来千奇百怪的输入照片。
   1.2 除了模型容量，（我不确定训练部分），如果是在3D objects数据集上训练，就算objavase等里面的对象类别也不够。
   1.3 从算法选择上来说，没有那种动辄几十步上百甚至上千步的Diffusion逐步求精，导致直接gen没有fit的技术路线，gen出来的好则好，如果不好也就没有办法了。所以，个人大胆认为：这种算法路线还不是最终路线：
   a)要么是又更多steps的逐步gen；
   b)要么至少来个refine的后续环节；
   c)更必要的，是在最后像素空间，对输入image的直接利用fit。比如至少fix shape & fit texture/color。
   1.4 如果从3D loss上训练，迟早数据是个问题，解决办法：1）用传统的非AI手段搞数据，UE/Unity3D搞数据，用GaussianSplatting多视图搞数据，等等等等，但我总觉得成本太高。 2) 回到2D loss上去。在可微渲染之后，从海量图片，视频上去学习。
   1.5 在训练的时候，感觉将camera pose作为可选项（有或者没有都可以），DUSt3R之类的一些有效的做法可以利用。

2. 需要更强大的priori策略。（约束多点好：不要拒绝文字作为约束条件，也不要拒绝多图片输入）
   现在单图，shape上面的问题还是很大。 比如输入人的正面而人的侧面背面基本shape都很不对。
   关于priori，我个人的所知道的路径：
   2.1 就利用现在的网络，加深加宽，训练来记忆。
   2.2 类似memory network直接偏记忆的网络去retrival。
   2.3 类似LLM中vector db, rag等等各种增加信息的。
   2.4 我个人走的是Differentiable Hash的路，去在相对很小的主网络之外，去维护千千万万的对象（可以不需要重训练的增加类别），只要见的多，记忆的多，紧邻搜索的距离学号了，数据就一定会让模型输出质量更好。（本质上说，都靠记忆。）

Hi, I have a question about camera parameters.
In the paper, I saw the sentence "we have opted to allow the model to “guess” the camera parameters." and I couldn't find out how to guess the camera parameters.
I searched TripoSR/tsr/system.py>class TSR > def render and TripoSR/tsr/utils.py > def get_spherical_cameras. In these parts, it seems like camera parameters are just given.
Where can I find the "guessing" part?

2024-03-11 10:32:52,933 - INFO - Initializing model ...
Traceback (most recent call last):
File "/mnt/thaoanh/img23d/TripoSR/run.py", line 109, in
model = TSR.from_pretrained(
File "/mnt/thaoanh/img23d/TripoSR/tsr/system.py", line 69, in from_pretrained
model = cls(cfg)
File "/mnt/thaoanh/img23d/TripoSR/tsr/utils.py", line 57, in init
self.cfg = parse_structured(self.Config, cfg)
File "/mnt/thaoanh/img23d/TripoSR/tsr/utils.py", line 20, in parse_structured
scfg = OmegaConf.merge(OmegaConf.structured(fields), cfg)
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 119, in structured
return OmegaConf.create(obj, parent, flags)
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 172, in create
return OmegaConf._create_impl(
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 861, in _create_impl
format_and_raise(node=None, key=None, value=None, msg=str(e), cause=e)
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/_utils.py", line 741, in format_and_raise
_raise(ex, cause)
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/_utils.py", line 719, in _raise
raise ex.with_traceback(sys.exc_info()[2]) # set end OC_CAUSE=1 for full backtrace
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 832, in _create_impl
return DictConfig(
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 113, in init
format_and_raise(node=None, key=key, value=None, cause=ex, msg=str(ex))
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/_utils.py", line 741, in format_and_raise
_raise(ex, cause)
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/_utils.py", line 719, in _raise
raise ex.with_traceback(sys.exc_info()[2]) # set end OC_CAUSE=1 for full backtrace
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 97, in init
self._set_value(content, flags=flags)
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 638, in _set_value
raise e
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 635, in _set_value
self._set_value_impl(value, flags)
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 665, in _set_value_impl
data = get_structured_config_data(value, allow_objects=ao)
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/_utils.py", line 445, in get_structured_config_data
return get_dataclass_data(obj, allow_objects=allow_objects)
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/_utils.py", line 373, in get_dataclass_data
format_and_raise(
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/_utils.py", line 821, in format_and_raise
_raise(ex, cause)
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/_utils.py", line 719, in _raise
raise ex.with_traceback(sys.exc_info()[2]) # set end OC_CAUSE=1 for full backtrace
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/_utils.py", line 365, in get_dataclass_data
d[name] = _maybe_wrap(
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 1041, in _maybe_wrap
return _node_wrap(
File "/home/thaoanh/miniconda3/envs/gen/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 1023, in node_wrap
raise ValidationError(f"Unexpected object type: {type_str(type)}")
omegaconf.errors.ValidationError: Unexpected object type: dict
full_key: image_tokenizer
object_type=Config

Building wheels for collected packages: torchmcubes
Building wheel for torchmcubes (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-312
creating build\lib.win-amd64-cpython-312\torchmcubes
copying torchmcubes_init_.py -> build\lib.win-amd64-cpython-312\torchmcubes
running build_ext
C:\Users\TF_USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\utils\cpp_extension.py:381: UserWarning: Error checking compiler version for cl: [WinError 2] 지정된 파일을 찾을 수 없습니다
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
building 'torchmcubes_module' extension
CUDA environment was not successfully loaded!
Build only CPU module!
running bdist_wheel
running build
running build_py
copying torchmcubes_init_.py -> build\lib.win-amd64-cpython-312\torchmcubes
running build_ext
building 'torchmcubes_module' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for torchmcubes
Running setup.py clean for torchmcubes
Failed to build torchmcubes
ERROR: Could not build wheels for torchmcubes, which is required to install pyproject.toml-based projects

pls. solve the problem

I've developed a custom node for comfyui, one of the most popular stable diffusion UIs out there.
Many of the people who tried it complained about torchmcubes, and I started to look for a better implementation. looks like scikit-image is a pretty good alternative (See here), but it seems the textures don't sit properly now. If you can help me complete the switch, your work will become much more usable to everyone.

Collecting git+https://github.com/tatsy/torchmcubes.git
Cloning https://github.com/tatsy/torchmcubes.git to /tmp/pip-req-build-kj1fm03v
Running command git clone --quiet https://github.com/tatsy/torchmcubes.git /tmp/pip-req-build-kj1fm03v
Resolved https://github.com/tatsy/torchmcubes.git to commit 3aef8afa5f21b113afc4f4ea148baee850cbd472
Preparing metadata (setup.py) ... done
Building wheels for collected packages: torchmcubes
Building wheel for torchmcubes (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [59 lines of output]
running bdist_wheel
/home/anaconda3/envs/py39/lib/python3.9/site-packages/torch/utils/cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-39
creating build/lib.linux-x86_64-cpython-39/torchmcubes
copying torchmcubes/init.py -> build/lib.linux-x86_64-cpython-39/torchmcubes
running build_ext
/home/anaconda3/envs/py39/lib/python3.9/site-packages/torch/utils/cpp_extension.py:366: UserWarning:

                                 !! WARNING !!
  
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  Your compiler (g++ 4.8.5) may be ABI-incompatible with PyTorch!
  Please use a compiler that is ABI-compatible with GCC 5.0 and above.
  See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
  
  See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
  for instructions on how to install GCC 5 or higher.
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  
                                !! WARNING !!
  
    warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
  /home/anaconda3/envs/py39/lib/python3.9/site-packages/torch/utils/cpp_extension.py:388: UserWarning: The detected CUDA version (11.4) has a minor version mismatch with the version that was used to compile PyTorch (11.7). Most likely this shouldn't be a problem.
    warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
  CUDA environment was not successfully loaded!
  Build only CPU module!
  running bdist_wheel
  /home/anaconda3/envs/py39/lib/python3.9/site-packages/torch/utils/cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
    warnings.warn(msg.format('we could not find ninja.'))
  running build
  running build_py
  copying torchmcubes/__init__.py -> build/lib.linux-x86_64-cpython-39/torchmcubes
  running build_ext
  /home/anaconda3/envs/py39/lib/python3.9/site-packages/torch/utils/cpp_extension.py:366: UserWarning:
  
                                 !! WARNING !!
  
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  Your compiler (g++ 4.8.5) may be ABI-incompatible with PyTorch!
  Please use a compiler that is ABI-compatible with GCC 5.0 and above.
  See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
  
  See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
  for instructions on how to install GCC 5 or higher.
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  
                                !! WARNING !!
  
    warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
  building 'torchmcubes_module' extension
  creating build/temp.linux-x86_64-cpython-39
  creating build/temp.linux-x86_64-cpython-39/cxx
  gcc -pthread -B /home/anaconda3/envs/py39/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/anaconda3/envs/py39/include -I/home/anaconda3/envs/py39/include -fPIC -O2 -isystem /home/anaconda3/envs/py39/include -fPIC -I/home/anaconda3/envs/py39/lib/python3.9/site-packages/torch/include -I/home/anaconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home/anaconda3/envs/py39/lib/python3.9/site-packages/torch/include/TH -I/home/anaconda3/envs/py39/lib/python3.9/site-packages/torch/include/THC -I/home/anaconda3/envs/py39/include/python3.9 -c cxx/grid_interp_cpu.cpp -o build/temp.linux-x86_64-cpython-39/cxx/grid_interp_cpu.o -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=torchmcubes_module -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
  gcc: 错误：unrecognized command line option ‘-std=c++17’
  error: command '/usr/bin/gcc' failed with exit code 1
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for torchmcubes
Running setup.py clean for torchmcubes
Failed to build torchmcubes
ERROR: Could not build wheels for torchmcubes, which is required to install pyproject.toml-based projects

Very good results and unnatural time, wouldn't it be great if you could open up the training code to us and let us try to get more amazing results with more channel counts or better datasets.

Please how can one integrate it with nodeJS or is an API available for it?

1. representation 表达
   TripoSR的大牛用的triplane作为输入极简的表示，然后扩展到10层的NeRF-MLP中。
   WU-CVGL/MVControl-threestudio#5 核心：无论gspoints/GS还是mesh/FlexiCubes那种多视图强拟合的，基于梯度甚至直接增减点的优势，在偏forward gen的流程中，优势利用不上了。
   TGS是hybrid triplane gauss. https://zouzx.github.io/TriplaneGaussian/

2. camera pose 相机
   训练部分，如果是在3D对象上训练的，这个本来很难处理的问题，在这个算法定义的任务里就不存在。但camera-pose其实很重要。

我把我读到的主要的做法，包括表示和相机，写在这里了。（可能一下不是很全，后续可能更新）
https://github.com/yuedajiong/super-ai/blob/main/superai-20240310-3d-reconstruction-two-important-representation-and-camerapose.txt

另外：关于本文这个网络的图，标注了一下。感觉要在2D上监督，也就是要给出相机位置渲染出图片后做loss。 网络结构上NeRF的10层感觉好深，因为每次forward后，在triplane之后的MLP都要给出很多camera-pose去render，这里网络越深越慢。 第一是不知道这里多浅足够表示不同对象和场景；第二是几乎没有看到有文章研究什么网络最合适NeRF的MLP替代。
有一些疑问：1)learnable pos embed到底作用是什么，因为是一个与输入无关的东西，训练完后不就固定了吗？OpenLRM中也是一样的逻辑，名字pos_embed； 2)把输入图像的信息浓缩到triplane然后在放到一个不浅的MLP中，这样是不是导致就最终很难还原度很高的去利用输入图像。

(venv) D:\TripoSR>pip install -r requirements.txt
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/tatsy/torchmcubes.git (from -r requirements.txt (line 4))
Cloning https://github.com/tatsy/torchmcubes.git to c:\users\ws3-01\appdata\local\temp\pip-req-build-e6vgr7zd
Running command git clone --filter=blob:none --quiet https://github.com/tatsy/torchmcubes.git 'C:\Users\ws3-01\AppData\Local\Temp\pip-req-build-e6vgr7zd'
Resolved https://github.com/tatsy/torchmcubes.git to commit 3aef8afa5f21b113afc4f4ea148baee850cbd472
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [17 lines of output]
Traceback (most recent call last):
File "D:\TripoSR\venv\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 353, in
main()
File "D:\TripoSR\venv\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "D:\TripoSR\venv\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "C:\Users\ws3-01\AppData\Local\Temp\pip-build-env-jbmzoijq\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "C:\Users\ws3-01\AppData\Local\Temp\pip-build-env-jbmzoijq\overlay\Lib\site-packages\setuptools\build_meta.py", line 295, in _get_build_requires
self.run_setup()
File "C:\Users\ws3-01\AppData\Local\Temp\pip-build-env-jbmzoijq\overlay\Lib\site-packages\setuptools\build_meta.py", line 487, in run_setup
super().run_setup(setup_script=setup_script)
File "C:\Users\ws3-01\AppData\Local\Temp\pip-build-env-jbmzoijq\overlay\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
exec(code, locals())
File "", line 2, in
ModuleNotFoundError: No module named 'torch'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Running TripoSR locally using the command line results in a model that is rotated 90 degrees. Using the gradio app locally does not have this issue.
Is there an option that I'm missing to fix this?

Command I used:
python run.py "{img}" --output-dir output/ --model-save-format glb

https://link.hackersthegame.com/simlink.php?p=60471474&t=01051218&c=17415&q=476&s=343

为什么我这边执行之后output是普通的平面图像不是3D图像？
python3 run.py examples/chair.png --output-dir output/
2024-03-07 17:33:30,971 - INFO - Initializing model ...
/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
2024-03-07 17:33:36,197 - INFO - Initializing model finished in 5225.30ms.
2024-03-07 17:33:36,197 - INFO - Processing images ...
2024-03-07 17:33:37,006 - INFO - Processing images finished in 809.15ms.
2024-03-07 17:33:37,006 - INFO - Running image 1/1 ...
2024-03-07 17:33:37,006 - INFO - Running model ...
2024-03-07 17:33:53,939 - INFO - Running model finished in 16933.09ms.
2024-03-07 17:33:53,939 - INFO - Exporting mesh ...
torchmcubes was not compiled with CUDA support, use CPU version instead.
2024-03-07 17:33:57,141 - INFO - Exporting mesh finished in 3201.11ms.

Does the software support this kind of household map wall to automatically become 3D effect?
For example, the height of the wall is 2.6 meters.
Thanks!

Can you release the curated cc-by dataset?

First up - big kudos. Really great work, by far the best I've seen.

I just wanted to point out that the generation of the mesh from the field is being done incorrectly; there's probably an 'off-by-0.5' error somewhere in the generation of the mesh via marching cubes (I haven't had a chance to look myself yet - but will when I have some free time). I suspect the underlying model is just fine.

You can see the result of the sample generation here:

If you look at the cushion, you will see a terracing effect - this is a sign that the sampling method used in the cubes is duplicating reads of the edges of a 'cell', instead of reading into the neighbourhood correctly (oft caused by chunking); it can also be caused by rounding of coordinates.

This same issue also seems to be affecting the generation of the normals.

作者说TripoSR的算法网络大结构和LRM类似，我也比较了关键模块是可以对应的，所以我借用LRM的算法图来标注TripoSR的实际数据，如下图：

当我们希望通过这样的网络结构，来“学习/记忆/泛化”海量的对象的时候，是困难的。
个人可能不正确的理解。
1）在NeRF中：Render部分是没有parameters的；当海量的对象被训练，NeRF-Decoder/MLP要能应对所有的对象，最终将被训练的非常“平凡”，意思是很难“学习记忆保留”下某些对象很“特定”的特征。我们也可以假定学到了一个“完美”的NeRF-Decoder，只要前面的网络，能够将希望得到的完美的立体，都表示在后面提到的信息流最窄的那480K的表示中。

2）但我们要产生：形态准确，细节非常丰富的模型，其实主要靠那个 backbone/Transformer1D。看这个网络的输入输出，是在相对比较抽象的3M左右的tri-plane tokens上变换；变换完后，被Upsample给压缩（丢失信息）到480K左右的满足后面的表示上。虽然在不同空间，但我们还是可以大概比较480k和mesh空间的数量级的数据差(6M+)： 一个高质量的静态人，不含精细的头发，需要100万/1M顶点xyz(其中要40万给人头特别脸），用顶点色来表示纹理，每个点也有rgb三个值，所以不考虑数值精度，得6M个数值点。（480K抽象的表示去指导产生6M+个点位的准确数值）（按照我个人的经验，这里需要说明的是：这种1M顶点的人，顶点位置是相对优化比较好的；反之，如果顶点位置优化的不好，可能需要甚至多一个数量级的点，才表示的差不多准确。对应到E2E的这种学习中，其实很难说学到的latent表示是“有利于”产生“压缩比高”的mesh。）
我也很好奇，TripoSR把triplane那里的维度，从LRM的80降低到40，而grid/64切分并没有更细，效果为啥反而更好。

3）在一种抽象/压缩过的480K空间上，要表示一个“跨越千千万万对象”复杂的物体，既没有没有修改NeRF部分来有short-cut利用原始输入或原始输入编码后的信息，也没有额外的refine步骤（比如估计出相机位置然后直接利用原始image给fit texture)，所以必然丢失大量像素空间的的信息。也就是说，给定一个比如人类含人脸这种，大概率会丢失人脸像素空间的一些特征，产生出来的人，就不可能非常“像”。

4）所以，总之，我个人的感觉，就是在红色五角星的细瓶颈的地方，准确的形态/丰富的纹理/比较严格的和输入图片一致，比较难。

I have setup as per github instruction but now i want to fine-tune with my own dataset so please share me the infrance for train model and create new checkpoint - model.ckpt.

https://github.com/zhaoyun0071/TripoSR-windows-GUI

Permanent hosting of TrioSR for free, should perform better than Hugging Face spaces.

https://imageto3d.org

As discussed in tatsy/torchmcubes#5

Here is the complete solution to install TripoSR

Install python https://www.python.org/ftp/python/3.12.2/python-3.12.2-amd64.exe
Install rust package manager https://win.rustup.rs/x86_64
Install Visual Studio Community https://visualstudio.microsoft.com/visual-cpp-build-tools/
Include "Desktop C++" workload and the buildtools (should be selected by default)
download triposr https://github.com/VAST-AI-Research/TripoSR/archive/refs/heads/main.zip
uncompress with tar -xf TripoSR-main.zip

You'll want to restart your computer or at least your terminal here

Now run, in a console, in the folder TripoSR-main

pip install --upgrade setuptools
pip install wheel
pip install ninja
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

Finally test triposr with

python run.py examples/chair.png --output-dir output/

This should get you a chair like this

For viewing OBJ files, you can use
https://www.meshlab.net/
or
https://sourceforge.net/projects/glc-player/

You can then try installing gradio, but that did not work for me at the moment due to a pip problem

Here are the results from pip list:
(venv) (base) c0nsume@c0nsume-sys:~/dev/TripoSR$ pip list
Package Version

filelock 3.13.1
fsspec 2024.2.0
Jinja2 3.1.3
MarkupSafe 2.1.5
mpmath 1.3.0
networkx 3.2.1
numpy 1.26.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.19.3
nvidia-nvjitlink-cu12 12.3.101
nvidia-nvtx-cu12 12.1.105
pillow 10.2.0
pip 24.0
setuptools 65.5.0
sympy 1.12
torch 2.2.1+cu121
torchaudio 2.2.1+cu121
torchvision 0.17.1+cu121
triton 2.2.0
typing_extensions 4.10.0

As you can see torch is successfully installed. Upon further inspection, the error is directly associated with this line in requirements.txt:
git+https://github.com/tatsy/torchmcubes.git

The install procedure for torchmcubes seems to be unable to find the local venv install of torch. If you install torch globally and follow the installation instructions for the TripoSR repo, everything should work fine. Just a heads up for anyone struggling

(.venv) ➜  TripoSR git:(main) python run.py examples/chair.png --output-dir output/
2024-03-05 03:56:32,235 - INFO - Initializing model ...
/media/user/home/Tools/06_MachineLearning/24_3D/TripoSR/.venv/lib/python3.9/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/media/user/home/Tools/06_MachineLearning/24_3D/TripoSR/.venv/lib/python3.9/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
2024-03-05 03:56:35,758 - INFO - Initializing model finished in 3523.10ms.
2024-03-05 03:56:35,758 - INFO - Processing images ...
2024-03-05 03:56:36,412 - INFO - Processing images finished in 654.22ms.
2024-03-05 03:56:36,412 - INFO - Running image 1/1 ...
2024-03-05 03:56:36,413 - INFO - Running model ...
2024-03-05 03:56:37,038 - INFO - Running model finished in 625.03ms.
2024-03-05 03:56:37,038 - INFO - Exporting mesh ...
Traceback (most recent call last):
  File "/media/user/home/Tools/06_MachineLearning/24_3D/TripoSR/run.py", line 154, in <module>
    meshes = model.extract_mesh(scene_codes)
  File "/media/user/home/Tools/06_MachineLearning/24_3D/TripoSR/tsr/system.py", line 185, in extract_mesh
    v_pos, t_pos_idx = self.isosurface_helper(-(density - threshold))
  File "/media/user/home/Tools/06_MachineLearning/24_3D/TripoSR/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/user/home/Tools/06_MachineLearning/24_3D/TripoSR/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/user/home/Tools/06_MachineLearning/24_3D/TripoSR/tsr/models/isosurface.py", line 45, in forward
    v_pos, t_pos_idx = self.mc_func(level.detach(), 0.0)
  File "/media/user/home/Tools/06_MachineLearning/24_3D/TripoSR/.venv/lib/python3.9/site-packages/torchmcubes/__init__.py", line 12, in marching_cubes
    return mc.mcubes_cuda(vol, thresh)
AttributeError: module 'torchmcubes_module' has no attribute 'mcubes_cuda'

This is from an virtualenv install for Python 3.9 on Ubuntu 20.04

2024-03-05 13:55:09,053 - INFO - Initializing model finished in 3986.39ms.
2024-03-05 13:55:09,053 - INFO - Processing images ...
EP Error D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1131 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\site-packages\onnxruntime\capi\onnxruntime_providers_tensorrt.dll"
when using ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
Falling back to ['CUDAExecutionProvider', 'CPUExecutionProvider'] and retrying.
2024-03-05 13:55:13,743 - INFO - Processing images finished in 4689.23ms.
2024-03-05 13:55:13,743 - INFO - Running image 1/1 ...
2024-03-05 13:55:13,743 - INFO - Running model ...
2024-03-05 13:55:14,634 - INFO - Running model finished in 891.16ms.
2024-03-05 13:55:14,634 - INFO - Exporting mesh ...
Traceback (most recent call last):
File "C:\Users\User\Documents\GitHub\TripoSR\run.py", line 154, in
meshes = model.extract_mesh(scene_codes)
File "C:\Users\User\Documents\GitHub\TripoSR\tsr\system.py", line 185, in extract_mesh
v_pos, t_pos_idx = self.isosurface_helper(-(density - threshold))
File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1527, in call_impl
return forward_call(*args, **kwargs)
File "C:\Users\User\Documents\GitHub\TripoSR\tsr\models\isosurface.py", line 45, in forward
v_pos, t_pos_idx = self.mc_func(level.detach(), 0.0)
File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\site-packages\torchmcubes_init.py", line 12, in marching_cubes
return mc.mcubes_cuda(vol, thresh)
AttributeError: module 'torchmcubes_module' has no attribute 'mcubes_cuda'. Did you mean: 'mcubes_cpu'?

RuntimeError: numel needs to be smaller than int32_t max; otherwise, please use packed_accessor64
I want to get mesh with higher resolution, but when changing resolution to 512, I got this error. Does it support export higher resolution mesh?

meshes = model.extract_mesh(scene_codes, resolution=512)

Is there any way to improve the texture? It's very low res and often buggy. The mesh quality is fine though.

I played with foreground ratio but can't make it any better. I also changed the chunk size and I don't see any noticeable effect.