Without masks, unable to extract mesh and low quality NeRF with DTU dataset (scan122 was tested) about instant-nsr-pl HOT 9 OPEN

Actually, I had to change to VanillaMLP instead of FullyFusedMLP in the colmap config, as tiny-cuda-nn errored at my V100.

from instant-nsr-pl.

Hi! The choice of the MLP network does not make much difference. The key here should be to use a background model. You could wait for my implementation, which is expected to come out in a few days.

from instant-nsr-pl.

I see. Great! Looking forward to it. Thanks for the reply.

from instant-nsr-pl.

@kelvincai522 I've just pushed a new branch that supports NeuS training with a background model. I'm still testing this feature on different scenes. Welcome to try it out!

from instant-nsr-pl.

@bennyguo Thank you for the new branch! I tested it with my own dataset and found the following in my Fedora with a V100 GPU (16GB vram):

• train/inv_s in the log never change (increase/decrease) while the loss is decreasing but dropped down to nan though
• I had to change model.texture.mlp_network_config.otype from FullyFusedMLP to VanillaMLP. Otherwise, tcnn would have Internal Error (I had to do that in previous versions as well).
• I had to disable lambda_distortion(_bg) by changing it to 0. Otherwise, it would error with "RuntimeError: max(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument."
• I had to change occupancy_grid_bg resolution from 256 to 128. Otherwise, Cuda is OOM.
• Had a warning during validation: data/users/kcai/instant-nsr-pl/utils/mixins.py:90: RuntimeWarning: invalid value encountered in true_divide | 0/2 [00:00<?, ?it/s]
  img = (img - img.min()) / (img.max() - img.min())
• While the img converge okay (not as good as nerf unbounded), it couldn't extract a mesh with the following error: File "/data/users/kcai/instant-nsr-pl/models/geometry.py", line 107, in isosurface
  vmin, vmax = mesh_coarse['v_pos'].amin(dim=0), mesh_coarse['v_pos'].amax(dim=0)
  IndexError: amin(): Expected reduction dim 0 to have non-zero size.

from instant-nsr-pl.

Thanks for the detailed information about your experiment! Is it possible to provide the dataset you used so that I could try it on my side? It would be very valuable!

from instant-nsr-pl.

That would be great. Here is the dataset https://drive.google.com/file/d/1EPhaA7vg9P3neH-T7MiVUIKE9nwgU04T/view?usp=sharing

from instant-nsr-pl.

It seems like most of the errors were caused by divergence. By setting model.radius=1.5 and turning off all the regularizations, i.e. system.loss.lambda_distortion=0. system.loss.lambda_distortion_bg=0. system.loss.lambda_sparsity=0., I was able to fairly reconstruct the scene:

I'll come up with an automatic strategy to determine model.radius and try to fix the stuffed area below the table.

from instant-nsr-pl.

About the error using FullyFusedMLP, you may have to run tiny-cuda-nn samples to see if the installation works.

from instant-nsr-pl.