zielon / insta-pytorch Goto Github PK

View Code? Open in Web Editor NEW

123.0 5.0 8.0 717 KB

INSTA - Instant Volumetric Head Avatars [Demo]

Home Page: https://zielon.github.io/insta/

License: Other

Python 10.67% C++ 22.32% Cuda 7.06% C 59.65% Shell 0.30%

3dmm avatars cuda flame instant-ngp neural-network volumetric-rendering

insta-pytorch's Introduction

INSTA - Instant Volumetric Head Avatars (pytorch)

Wojciech Zielonka, Timo Bolkart, Justus Thies

Max Planck Institute for Intelligent Systems, Tübingen, Germany

Video Paper Project Website Dataset Face Tracker Email

This repository is based on torch-ngp and implements most of the C++ version. Please note that this is not 1:1 port of C++ version and some features are missing thus the speed of training, rendering, and the quality are not exactly the same like in the case of the paper version of INSTA. Therefore, for any comparisons please use the C++ version.

Installation

Please follow the installation from torch-ngp. This implementation requires to use cuda ray marching, which is enabled by --cuda-ray. Therefore, C++ extension must be built. Moreover, for a fast nearest neighbor search we are using BVH repository.

First create the environment.

conda create -n insta-pytorch python=3.9
conda activate insta-pytorch
pip install -r requirements.txt

After that use the install.sh script to compile all the required libraries and prepare the workbench.

Usage

The dataset structure is compatible with the C++ version. In order to generate a new sequence input follow instructions. The hardware requirement are the same as in the case of C++ version.

The released avatars are compatible with training, however, the checkpoints were generated for the C++ version.

# Offscreen rendering
python main_insta.py data/nerf/wojtek --workspace workspace/wojtek -O --tcnn

# GUI
python main_insta.py data/nerf/wojtek --workspace workspace/wojtek -O --tcnn --gui

Using GUI in the Menu/Options you can control the selected mesh.

Citation

If you use this project in your research please cite INSTA:

@proceedings{INSTA:CVPR2023,
  author = {Zielonka, Wojciech and Bolkart, Timo and Thies, Justus},
  title = {Instant Volumetric Head Avatars},
  journal = {Conference on Computer Vision and Pattern Recognition},
  year = {2023}
}

insta-pytorch's People

Contributors

Stargazers

Watchers

Forkers

chenchy eknim akinc-flwls johndpope wasahaiah mrtornado24 lighttwist psal0

insta-pytorch's Issues

no network_ff file in ./insta folder

there is no network_ff file in ./insta folder, but there is option for ff in main_insta.py line 80

if opt.ff:
opt.fp16 = True
assert opt.bg_radius <= 0, "background model is not implemented for --ff"
from insta.network_ff import NeRFNetwork

Definition of depth

Thanks for the nice code!
I have a question about the definition of depth.

In the raymarching.cu, it seems that the rendered depth (calculated from deltas[1]) is the distance along the rays_d (point distance).
- rays_d are normalized vector as in utils.py
The concept of depth used in depthmap (e.g. stereo or mvs) is usually the z-axis length (plane distance).
In utils.py, L1Loss is directly applied to outputs['depth'] and data['depth'], does it mean that the data['depth'] is actually point distance instead of the usually used plane distance?

is there a way to export the mesh (.obj) and mesh texture (wrapped) for the avatar

want to export output for use in 3d application so .obj, .fbx + mesh needed

will you provide the torch version source code of freq encoder and grid encoder?

INSTA-pytorch/freqencoder/src/freqencoder.cu
INSTA-pytorch/gridencoder/src/gridencoder.cu

could you provide torch version source code for above files?

bvh setup error

line 66 in bvh/setup.py : -DNUM_THREADS=256, it causes the error:
block_merge_sort.cuh(169): error: expected a "," or ">"

block_merge_sort.cuh use NUM_THREADS as an input param. And I commented out this line in setup.py, it works.

Ubuntu 22.04.3 LTS
python 3.10
torch 2.1.2
cuda 12.1

ModuleNotFoundError: No module named 'bvh_distance_queries_cuda'

Hi Zielon, thank you for your amazing work!

I wanted to ask if there is a missing module in the repository. When I run the INSTA-pytorch code, I receive an error message stating "No module named 'bvh_distance_queries_cuda'". I have searched both your repository and the BVH repository, but I cannot find any module with that name. Could you please advise me on how to resolve this issue?

Looking forward to your reply!

What is the cond parameter in model.density?

I am trying to generate the mesh after training. There is a cond parameter in the model.density function that is not passed in line 599 in utils.py.

sigma = self.model.density(pts.to(self.device), cond=1)['sigma']

CUDA memory misused

Hi Zielon,
When I used your codes to train INSTA on my own video, an error occurred. It is important to note that to reduce VRAM requirements I disabled -O and just used --fp16 --cuda_ray according to #2.

Below is the error report.

  0% 0/1500 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/wanghy/INSTA-pytorch/main_insta.py", line 160, in <module>
    trainer.train(train_loader, valid_loader, max_epoch)
  File "/home/wanghy/INSTA-pytorch/insta/utils.py", line 625, in train
    self.train_one_epoch(train_loader)
  File "/home/wanghy/INSTA-pytorch/insta/utils.py", line 838, in train_one_epoch
    self.model.update_extra_state(loader)
  File "/home/wanghy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/wanghy/INSTA-pytorch/insta/renderer.py", line 572, in update_extra_state
    cas_xyzs, faces = self.warp_position(triangles, cas_xyzs)
  File "/home/wanghy/INSTA-pytorch/insta/renderer.py", line 293, in warp_position
    distances, closest_points, closest_faces, closest_bcs = m(triangles, xyzs.unsqueeze(dim=0))
  File "/home/wanghy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/wanghy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/wanghy/INSTA-pytorch/bvh/bvh_distance_queries/bvh_search_tree.py", line 109, in forward
    output = BVHFunction.apply(
  File "/home/wanghy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/wanghy/INSTA-pytorch/bvh/bvh_distance_queries/bvh_search_tree.py", line 42, in forward
    outputs = bvh_distance_queries_cuda.distance_queries(
RuntimeError: triangles must be a CUDA tensor

PS: My machine is NVIDIA G-RTX 3090 with 24GB VRAM and over 200G RAM, so I am also confused why it cannot meet the memory requirement when using -O.

TypeError: density() missing 1 required positional argument: 'cond'

Traceback (most recent call last):
File "/INSTA-pytorch/main_insta.py", line 132, in
trainer.save_mesh(resolution=256, threshold=10)
File "/INSTA-pytorch/insta/utils.py", line 602, in save_mesh
vertices, triangles = extract_geometry(self.model.aabb_infer[:3], self.model.aabb_infer[3:], resolution=resolution, threshold=threshold, query_func=query_func)
File "/INSTA-pytorch/insta/utils.py", line 182, in extract_geometry
u = extract_fields(bound_min, bound_max, resolution, query_func)
File "/INSTA-pytorch/insta/utils.py", line 175, in extract_fields
val = query_func(pts).reshape(len(xs), len(ys), len(zs)).detach().cpu().numpy() # [S, 1] --> [x, y, z]
File "/INSTA-pytorch/insta/utils.py", line 599, in query_func
sigma = self.model.density(pts.to(self.device))['sigma']
TypeError: density() missing 1 required positional argument: 'cond'

rotation_6d_to_matrix instead of batch_rodrigues

I was wondering the reasoning behind this change. As far as I know, batch_rodrigues method was being used in FLAME to compute rotation matrix. I also see that full_pose has 30 elements instead of 15.

I have a project which uses a FLAME version that uses batch_rodrigues, I am unable to use it due to this change. Is there a way to coınvert these changes?

RuntimeError: CUDA error: device-side assert triggered

Hello, thank you for the great work!

I encounter a cuda error when running the command line python main_insta.py data/test/ --workspace workspace/test/ -O --tcnn

Namespace(H=256, O=True, W=256, bg_radius=-1, bound=2, ckpt='latest', clip_text='', color_space='srgb', cuda_ray=True, density_thresh=10, dt_gamma=0.00390625, error_map=False, ff=False, fovy=50, fp16=True, gui=False, iters=100000, lr=0.0025, max_ray_batch=4096, max_spp=64, max_steps=1024, min_near=0.01, num_rays=4096, num_steps=1024, offset=[0, 0, 0], patch_size=16, path='data/test/', preload=True, radius=5, rand_pose=-1, scale=1, seed=49, tcnn=True, test=False, update_extra_interval=16, upsample_steps=0, val_freq=5, workspace='workspace/test/')
=========== NETWORK TCNN ===========
Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
NeRFNetwork(
(encoder): Encoding(n_input_dims=3, n_output_dims=128, seed=1337, dtype=torch.float16, hyperparams={'base_resolution': 16, 'hash': 'CoherentPrime', 'interpolation': 'Linear', 'log2_hashmap_size': 18, 'n_features_per_level': 8, 'n_levels': 16, 'otype': 'Grid', 'per_level_scale': 1.4472692012786865, 'type': 'Hash'})
(sigma_net): Network(n_input_dims=144, n_output_dims=16, seed=1337, dtype=torch.float16, hyperparams={'encoding': {'offset': 0.0, 'otype': 'Identity', 'scale': 1.0}, 'network': {'activation': 'ReLU', 'n_hidden_layers': 2, 'n_neurons': 64, 'otype': 'CutlassMLP', 'output_activation': 'None'}, 'otype': 'NetworkWithInputEncoding'})
(encoder_dir): Encoding(n_input_dims=3, n_output_dims=16, seed=1337, dtype=torch.float16, hyperparams={'degree': 4, 'otype': 'SphericalHarmonics'})
(color_net): Network(n_input_dims=31, n_output_dims=3, seed=1337, dtype=torch.float16, hyperparams={'encoding': {'offset': 0.0, 'otype': 'Identity', 'scale': 1.0}, 'network': {'activation': 'ReLU', 'n_hidden_layers': 3, 'n_neurons': 64, 'otype': 'CutlassMLP', 'output_activation': 'None'}, 'otype': 'NetworkWithInputEncoding'})
)
100%|████████████████████████████████████████████████████████████████████████████████████| 1463/1463 [01:06<00:00, 22.15it/s]
Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off]
Loading model from: /root/.local/conda/envs/insta/lib/python3.8/site-packages/lpips/weights/v0.1/alex.pth
Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off]
Loading model from: /root/.local/conda/envs/insta/lib/python3.8/site-packages/lpips/weights/v0.1/vgg.pth
[INFO] Trainer: ngp | 2023-04-23_18-25-34 | cuda | fp16 | trails/trail_woman_1_4/
[INFO] #parameters: 26590464
[INFO] Loading latest checkpoint ...
[WARN] No checkpoint found, model randomly initialized.
100%|████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 28.17it/s]
[mark untrained grid] 3998599 from 4194304
==> Start Training Epoch 1, lr=0.002500 ...
0% 0/1463 [00:00<?, ?it/s]/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [0,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [1,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [2,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [3,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [4,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [5,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [6,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [7,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [8,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [9,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [197,0,0], thread: [29,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [197,0,0], thread: [30,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [197,0,0], thread: [31,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [8,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [9,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [10,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [11,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [12,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [13,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [14,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [15,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [16,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [17,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [18,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [19,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [20,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [21,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [22,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [23,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [24,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [25,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [115,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [116,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [117,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [118,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [119,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [120,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [121,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [122,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [123,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [124,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [125,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [126,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [127,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [54,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [55,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [56,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [57,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [58,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [59,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [60,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [61,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [62,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [63,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [302,0,0], thread: [125,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [302,0,0], thread: [126,0,0] Assertion srcIndex < srcSelectDimSize failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [302,0,0], thread: [127,0,0] Assertion srcIndex < srcSelectDimSize failed.
Traceback (most recent call last):
File "main_insta.py", line 160, in
trainer.train(train_loader, valid_loader, max_epoch)
File "/root/INSTA/insta/utils.py", line 626, in train
self.train_one_epoch(train_loader)
File "/root/INSTA/insta/utils.py", line 839, in train_one_epoch
self.model.update_extra_state(loader)
File "/root/.local/conda/envs/insta/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/root/INSTA/insta/renderer.py", line 572, in update_extra_state
cas_xyzs, faces = self.warp_position(triangles, cas_xyzs)
File "/root/INSTA/insta/renderer.py", line 295, in warp_position
canonical_triangles = torch.index_select(self.canonical_triangles, 1, closest_faces[0])
RuntimeError: CUDA error: device-side assert triggered
0% 0/1463 [00:01<?, ?it/s]

FP32 training provides blurry and incorrect resulting images

I've been trying to train the models on the wojtek data, on FP32, since I want to gather data for a research project and fully fused MLPs obscure weight data. However, when I train on FP32, the resulting images after evaluation and testing are blurry and are not the same as the images produced by the TCNN, FP16 and fully fused model. Is this behaviour expected, or is something wrong with the code?

Inference Speed

I trained a model with provided Wojtek data, inference speed is around 1.4 seconds per frame.

I also trained a model using different data which I generated following the steps mentioned. However, it takes 7 seconds to infer 1 frame. I used same amount of num_rays. What might be causing this slowness?

Assertion `failed on every thread

Thanks for your pytorch version! While after the training start, it says there NaN or Inf found in input tensor, and Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. The output is listed below:

python main_insta.py data/wojtek_1 --workspace workspace/wojtek_1 -O --tcnn                                                                  (insta-pytorch)
Namespace(path='data/wojtek_1', O=True, test=False, workspace='workspace/wojtek_1', seed=49, iters=100000, val_freq=10, lr=0.0025, ckpt='latest', num_rays=4096, cuda_ray=True, max_steps=1024, num_steps=1024, upsample_steps=0, update_extra_interval=16, max_ray_batch=4096, patch_size=1, fp16=True, ff=False, tcnn=True, color_space='srgb', preload=True, bound=2, scale=1, offset=[0, 0, 0], dt_gamma=0.00390625, min_near=0.01, density_thresh=10, bg_radius=-1, gui=False, W=512, H=512, radius=5, fovy=50, max_spp=64, error_map=False, clip_text='', rand_pose=-1)
=========== NETWORK TCNN ===========
NeRFNetwork(
  (encoder): Encoding(n_input_dims=3, n_output_dims=128, seed=1337, dtype=torch.float16, hyperparams={'base_resolution': 16, 'hash': 'CoherentPrime', 'interpolation': 'Linear', 'log2_hashmap_size': 18, 'n_features_per_level': 8, 'n_levels': 16, 'otype': 'Grid', 'per_level_scale': 1.4472692012786865, 'type': 'Hash'})
  (sigma_net): Network(n_input_dims=144, n_output_dims=16, seed=1337, dtype=torch.float16, hyperparams={'encoding': {'offset': 0.0, 'otype': 'Identity', 'scale': 1.0}, 'network': {'activation': 'ReLU', 'n_hidden_layers': 2, 'n_neurons': 64, 'otype': 'FullyFusedMLP', 'output_activation': 'None'}, 'otype': 'NetworkWithInputEncoding'})
  (encoder_dir): Encoding(n_input_dims=3, n_output_dims=16, seed=1337, dtype=torch.float16, hyperparams={'degree': 4, 'otype': 'SphericalHarmonics'})
  (color_net): Network(n_input_dims=31, n_output_dims=3, seed=1337, dtype=torch.float16, hyperparams={'encoding': {'offset': 0.0, 'otype': 'Identity', 'scale': 1.0}, 'network': {'activation': 'ReLU', 'n_hidden_layers': 3, 'n_neurons': 64, 'otype': 'FullyFusedMLP', 'output_activation': 'None'}, 'otype': 'NetworkWithInputEncoding'})
)
data/wojtek_1
data/wojtek_1/canonical.obj
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1500/1500 [00:38<00:00, 39.42it/s]
Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off]
/home/wangzy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/wangzy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Loading model from: /home/wangzy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/lpips/weights/v0.1/alex.pth
[INFO] Trainer: ngp | 2023-04-07_15-40-45 | cuda | fp16 | workspace/wojtek_1
[INFO] #parameters: 26590464
[INFO] Loading latest checkpoint ...
[WARN] No checkpoint found, model randomly initialized.
data/wojtek_1
data/wojtek_1/canonical.obj
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 43.93it/s]
[mark untrained grid] 4194084 from 4194304
==> Start Training Epoch 1, lr=0.002500 ...
  0% 0/1500 [00:00<?, ?it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.002500: :   0% 1/1500 [00:00<06:20,  3.94it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.002500: :   0% 1/1500 [00:00<06:20,  3.94it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.002500: :   0% 2/1500 [00:00<06:20,  3.94it/s]NaN or Inf found in input tensor.
loss=353813397595729098627955600845176832.0000 (nan), lr=0.002500: :   0% 5/1500 [00:00<01:33, 15.93it/s]NaN or Inf found in input tensor.
loss=25.2939 (nan), lr=0.002500: :   0% 6/1500 [00:00<01:33, 15.93it/s]NaN or Inf found in input tensor.
loss=0.1733 (nan), lr=0.002500: :   1% 10/1500 [00:00<01:07, 22.14it/s]                       NaN or Inf found in input tensor.
loss=459.3705 (nan), lr=0.002500: :   1% 13/1500 [00:00<00:58, 25.51it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.002500: :   1% 13/1500 [00:00<00:58, 25.51it/s]     NaN or Inf found in input tensor.
loss=1.0937 (nan), lr=0.002500: :   1% 15/1500 [00:00<00:58, 25.51it/s]../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [100,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [101,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [102,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [103,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [104,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [105,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [106,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [107,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [108,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [109,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [111,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [112,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [113,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [114,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [115,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [116,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [117,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [118,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [119,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [120,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [122,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [123,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [124,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [125,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [126,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [127,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [64,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [65,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [66,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [67,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [68,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [69,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [70,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [71,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [72,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [73,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [74,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [75,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [76,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [77,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [78,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [79,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [80,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [81,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [82,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [83,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [84,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [85,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [86,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [87,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [88,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [89,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [90,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [91,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [92,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [93,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [94,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [95,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [33,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [34,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [36,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [37,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [38,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [39,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [40,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [41,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [42,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [43,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [44,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [45,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [46,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [47,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [48,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [49,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [50,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [51,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [52,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [53,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [54,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [55,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [56,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [57,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [58,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [59,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [60,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [61,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [62,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [63,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
terminate called after throwing an instance of 'thrust::system::system_error'
  what():  CUDA free failed: cudaErrorAssert: device-side assert triggered
fish: Job 1, 'python main_insta.py data/wojte…' terminated by signal SIGABRT (Abort)

If there's any other information I need to provide, please let me know!

How do I use snapshot.msgpack trained from INSTA (C++)?

Hello,

I trained a NeRF using the original INSTA repository, and have my trained weights. How do I convert those weights to something I can use in INSTA-pytorch? (ie. how do I convert snapshot.msgpack to .pth?)

Thank you!

mesh_masks/lips.obj is missed in the project

Hi，

Thanks for your excellent work., I'm interested in the amazing results. However, i found sth was missed in the code when i run the project. In insta/masks.py, '/home/wzielonka/CLionProjects/real-time-avatar/mesh_masks/lips.obj' was not included, can you fix it, thank you very much!

Memory requirements

I have 32 GB of RAM and Tesla T4 with 16 GB VRAM.

I see that 24GB of VRAM is required. Disabling -O and just using --fp16 --cuda_ray seem to reduce VRAM requirements.

However, I am running out of regular memory during data preparation at the start of third epoch.

What is the memory requirements with this setting?