nerfstudio-project / nerfstudio Goto Github PK

Can we determine if a config isn't used?

The initialized b_matrix at this line wouldn't be saved together with the model if it is not a buffer or parameter. You might want to consider adding self.register_buffer to it.

We should outline the steps to setup development environment and how to run the code checks. Will be useful/necessary when others want to contribute.

• change up direction to be correct
• updating the damping factor and remove settings unrelated to existing TrackballControls

• add interface between nerf backend to read out possible outputs
• toggle between different options for rendering
• fix the accumulation 'apply_depth_colormap' things

Pushing to git will now fail if there are warnings in the doc compilation. Can we add a doc compilation step in our run_action.py script so that we can catch those warning before pushing. This likely involves calling make clean and make html in the docs folder.

This is a major refactoring, but we should change "Graph" to a name that is less confusing for first-time users attempting to understand our code. "Graph" has been too often confused with "computation graph".

Update the ipynb camera visualization with the following:

• Move visualization commands into camera class and out of notebook
• Add description and figure for coordinate system used in pyrad
• Add more descriptions to visualization
• Improve ray visualization
• Maybe visualize frustums

• Add TLDR table for the various encodings
• Add more descriptions and links for each encoding method

• Convert plots to plotly
• Add PDF sampler vis
• Include table or guidance for when to use each sampler

I added some func to handle the colormap stuff
https://github.com/plenoptix/pyrad/blob/069cf2c40fb3ab68c483501f18713992b3c00d8a/pyrad/graphs/instant_ngp.py#L148-L162

And set it as a base class thing to get everything to work:
https://github.com/plenoptix/pyrad/blob/069cf2c40fb3ab68c483501f18713992b3c00d8a/pyrad/graphs/base.py#L139-L145

But i dont think this is best way to handle, so need to figure out a more robust way of handling this across implementations

This is where the function is referenced in visualizer code:
https://github.com/plenoptix/pyrad/blob/069cf2c40fb3ab68c483501f18713992b3c00d8a/pyrad/viewer/server/viewer_utils.py#L79

The goals of this PR are the following:

1. Change raw data loaders to use classes instead of functions (e.g., functions like this should be classes. This will help with cleanliness and handling new data types.
2. "dataset_format" should be an attribute of the new classes (see above).
3. get_dataset_inputs() should not have if/else checks on dataset_format. Rather, it should know classes are implemented (see above) and choose appropriately.

Bonus

• a good way to cache dataset inputs to avoid having to read large COLMAP binaries, etc., which can take a while. Some code attempts to do this already, but it's too hacky to be used by users.

Allow user to pause training for smoother rendering.

• add start/stop button in viewer + add signalling logic across server
• figure out how to stop training but keep rendering (thinking the pause logic should keep looping in: _is_render_step)

• Create ipynb that visualizes spatial distortion method
• Include description when different spatial distortion methods should be used

Will involve linting

We want to refactor how computing losses and metrics work. Currently get_loss_dict returns a dictionary of losses. These losses are then combined in get_aggregated_loss_dict using coefficients defined in the config. This workflow is not the most transparent, ie. if I add a new loss, I then need to know that I must update the configs accordingly.

Proposal:
Change get_loss_dict(outputs, batch) to get_loss(outputs, batch, metrics=None, coefficients=None) -> float and get_metrics_dict(outputs, batch) -> dict.
Remove get_aggregated_loss_dict

Relevant code (would need to update for all models):
https://github.com/plenoptix/pyrad/blob/56661b5d9aa8adfec9cad60bce53036cb0ceca43/pyrad/graphs/vanilla_nerf.py#L143-L148

https://github.com/plenoptix/pyrad/blob/b0594935af747ba5487aee4816e1cbcdfc408967/pyrad/graphs/base.py#L162-L174

this is a strange location to put it...

Make consistent with rest of code.

Perhaps an issue the recent changes that allow you to switch outputs?

Traceback (most recent call last):
  File "scripts/run_train.py", line 221, in main
    launch(
  File "scripts/run_train.py", line 161, in launch
    main_func(local_rank=0, world_size=1, config=config)
  File "scripts/run_train.py", line 128, in _train
    trainer.train()
  File "/projects/pyrad/pyrad/engine/trainer.py", line 118, in train
    self.visualizer_state.update_scene(step, self.graph)
  File "/projects/pyrad/pyrad/viewer/server/viewer_utils.py", line 92, in update_scene
    self._render_image_in_viewer(graph)
  File "/projects/pyrad/pyrad/utils/profiler.py", line 34, in wrapper
    ret = func(*args, **kwargs)
  File "/projects/pyrad/pyrad/viewer/server/viewer_utils.py", line 226, in _render_image_in_viewer
    image_output = outputs[output_type].cpu().numpy() * 255
KeyError: 'rgb'

File currently contains unused code and does not match the style guidelines.

When running python scripts/run_train.py on vanilla NeRF the following error is raised. The gather indices are out of bounds.

  File "/projects/pyrad/pyrad/engine/trainer.py", line 205, in test_image
    outputs = self.graph.get_outputs_for_camera_ray_bundle(camera_ray_bundle)
  File "/home/tancik/miniconda3/envs/pyrad/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/projects/pyrad/pyrad/graphs/base.py", line 188, in get_outputs_for_camera_ray_bundle
    outputs = self.forward_after_ray_generator(ray_bundle)
  File "/projects/pyrad/pyrad/graphs/base.py", line 145, in forward_after_ray_generator
    outputs = self.get_outputs(intersected_ray_bundle)
  File "/projects/pyrad/pyrad/graphs/vanilla_nerf.py", line 121, in get_outputs
    ray_samples_pdf = self.sampler_pdf(ray_bundle, ray_samples_uniform, weights_coarse)
  File "/home/tancik/miniconda3/envs/pyrad/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/projects/pyrad/pyrad/graphs/modules/ray_sampler.py", line 47, in forward
    ray_samples = self.generate_ray_samples(*args, **kwargs)
  File "/home/tancik/miniconda3/envs/pyrad/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/projects/pyrad/pyrad/graphs/modules/ray_sampler.py", line 324, in generate_ray_samples
    cdf_g1 = torch.gather(cdf, -1, above)
RuntimeError: CUDA error: device-side assert triggered

This is likely caused by #116
@liruilong940607

see https://stackoverflow.com/questions/19052834/is-it-possible-to-exclude-files-from-git-language-statistics for details

I think we want these values to be the same in the config.

After training for a long time ~81,990 steps on instant ngp, get the following stack trace:

create run_eval_dataset.py

Rather than setting in the config, the user should be able to set the minimum and maximum render resolution from the viewer ui.

• add dat gui slider with min max values
• add logic to read value back to server
• update backend-side code to reflect max value

All of the components needed should already be implemented. Just need to create a graph/config and benchmark against the paper.

• cache the dataloader with pickle (for now). later, maybe add proper serialization
• scale and shift the friends dataset so it's centered about the origin
• add config to sample from only the mask or not

  ImageDataset classes will return batches of images and masks, etc.
  PixelSamplers will choose which pixels to use from the image datasets.
  

Related to PR #134 and Issue #117
cc: @brentyi

eps: 1e-4
iter 200:

iter 500

eps: Default
iter 200:

iter 500:

Currently configs need to specify train and test dataset configurations separately:

https://github.com/plenoptix/pyrad/blob/b0594935af747ba5487aee4816e1cbcdfc408967/configs/graph_default.yaml#L48-L75

Intended behaviour should be the train dataset config is the default for the test dataset.
Only if test dataset is specified, then it will overwrite for test.

Issue of the F.grid_sample: padding_mode==zeros means padding zero voxel values outside the grid. a query point that is slightly outside the grid would be interpolated by the voxel values on the boarder of the grid and the outside zero voxels. So it gives non-zero value at regions slightly outside the grid.

Toy code to show:

grid = torch.ones((1, 1, 128, 128, 128))
positions = torch.tensor([[0.5, 1.004, 0.5]])
values = F.grid_sample(
    grid,
    positions.view(1, -1, 1, 1, 3),
    align_corners=True,
    padding_mode="zeros",
)
print(values.flatten())  # >> 0.7460

Relevant code in our code base:

https://github.com/plenoptix/pyrad/blob/91ef54963c43beb34f13301f8496faf2f0de8a2e/pyrad/fields/occupancy_fields/occupancy_grid.py#L131