Thanks for creating and maintaining this great library!
I was wondering if you have stats for the time taken to train the different models on one of the standard datasets documented somewhere. If not, these would be useful for reference and comparisons.

"I am trying to train a model using plenoxel but some errors occured. It is shown that there is no attribute "sample_grid" in '_C'.
How should I adjust the program for training?

Traceback (most recent call last):
  File "run.py", line 230, in <module>
...
...
/miniconda3/envs/nerf_factory/lib/python3.8/posixpath.py", line 152, in dirname
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType

This is running on WSL2 in the nerf_factory conda env with cuda enabled. What could be causing this? Any help would be greatly appreciated.

Thanks for releasing this excellent NeRF factory! The code works very well and I just have some questions about the Mip-NeRF 360 pose process and hope you could help me understand it. Thank you very much!

1. I want to confirm that are you using OpenCV coordinate to train Mip-NeRF 360 data? I suppose the poses before processing is OpenGL coordinate as follows: https://github.com/kakaobrain/NeRF-Factory/blob/ce06663abe385c5cbe85fddbca8b9b5ace80dbee/src/data/data_util/nerf_360_v2.py#L365
   After processing (Q2 and Q3), https://github.com/kakaobrain/NeRF-Factory/blob/ce06663abe385c5cbe85fddbca8b9b5ace80dbee/src/data/data_util/nerf_360_v2.py#L381
   The coordinate is transformed to OpenCV coordinate. Is this correct? If so, since many NeRF datasets use OpenGL coordinate, why do you choose to use OpenCV coordinate instead of OpenGL coordinate?
2. I do not understand the function of transform_pose_llff. It seems to change some values of extrinsic matrix but I do not understand the reason to do so. Could you please help clarify this? https://github.com/kakaobrain/NeRF-Factory/blob/ce06663abe385c5cbe85fddbca8b9b5ace80dbee/src/data/data_util/nerf_360_v2.py#L337
3. I do not understand the function of similarity_from_cameras. I guess this function has some relation with the transform_pose_llff, but I cannot figure out its effect, as well as the the math to transform from OpenGL to OpenCV coordinate using Q2 and Q3 functions. Could you please help clarify this? https://github.com/kakaobrain/NeRF-Factory/blob/ce06663abe385c5cbe85fddbca8b9b5ace80dbee/src/data/data_util/nerf_360_v2.py#L279

Thank you very much for your help!
Best regards!

Hi,

Thanks for your awesome framework. But I found the "generate_ellipse_path" function borrowed from the https://github.com/google-research/multinerf is not compatiable with the coordinate defined in "nerf_360_v2.py". Thus, with this function we cannot render the expected camera path.

Hi, I really appreciate your work, well done!
I have a question about your implementation.

How much does your work reproduce the results of Ref-NeRF?
The paper reports 33.99 of average PSNR on Blender dataset.
On the other hand, as I calculate the average PSNR of 8 scenes you report in the webpage, I get 33.27, slight lower than the official result.
Did I miss something, or do you have ideas on why there is a difference?

Thanks!

Hi,

Thanks a lot for this repo.

To the best of my knowledge you never actually visualize the normals, so I added some lines basically saying the normals are the weighted (by rendering weights) sum of the per-ray-sample normals, then normalized again.

       normals_deriv = (rendered_results[1]["weights"][..., None] * rendered_results[1]["normals"]).sum(1)
        normals_deriv = ref_utils.l2_normalize(normals_deriv)
        normals_pred = (rendered_results[1]["weights"][..., None] * rendered_results[1]["normals_pred"]).sum(1)
        normals_pred = ref_utils.l2_normalize(normals_pred)

What I get makes sense for the derived normals, but not for the predicted ones. They seem to be collapsing, such that the shading part of the MLP just doesn't use them and only the view direction?

Thanks in advance for any tips

Thank you in advance!

Thank you for your excellent work about the NeRF factory! The code is functioning well, and I have some questions regarding the pose processing in Mip-NeRF 360. I was hoping you could help me better understand it. Thank you very much!
I would like to inquire about the purpose of this line of code,

Hi, thank you for this work!

I have a question of the dataloader of llff dataset and 360 dataset, which is very different!
In order to use nerf or mipnerf360, the method of prepare my own dataset is the same ,which is using COLMAP to prepross images . so why the load method is different?
By the way, in the function load_nerf_360_v2_data/nerf_360_v2.py， what's puerpose of calling "similarity_from_cameras"?

Thanks a lot!
qc

Everything is setup properly to the best of my knowledge, when I start the training I simply get a Killed message and it stops.

Is there a way to see a log or is this a known issue?

running on a 3090.

I'm noticing Memory maxing out and then the Killed happens, is there a memory requirement?
I've played with the batch size and such in the .gin file to see if it alleviates it but with no luck.
Currently i have 32GB of RAM and 24GB VRAM

When you run python3 run.py --ginc configs/nerf/blender.gin --scene chair as suggested in https://kakaobrain.github.io/NeRF-Factory/docs/installation/#training-code, this line is failing:

AttributeError: 'Trainer' object has no attribute 'num_devices'
  In call to configurable 'run' (<function run at 0x7f76abbdcb80>)

    fn(*args, **kwargs)
  File "/home/mishig/NeRF-Factory/src/data/interface.py", line 60, in setup
    self.num_devices = self.trainer.num_devices
AttributeError: 'Trainer' object has no attribute 'num_devices'
  In call to configurable 'run' (<function run at 0x7f7dca95ca60>)

Thanks in advance! 👍

Hi,

Thank you all for the great work and the release of the code. I want to try the RefNeRF in real datasets, starting by reproducing the results in the scenes from the paper.

I used the following command to execute the code.
python3 run.py --ginc configs/refnerf/refnerf_real.gin --scene gardenspheres

But the rendering result is not good.

I'm not sure how I can get the results of the real data of refnerf.

Thank you.

This code seems to be incompatible with the latest PyTorch and pytorch-lightning version.

The error occurs in 'BatchSampler'...

Do you have any plan to revise the code for the latest version of PyTorch, pytorch-lightning on the latest CUDA environment?

Many thanks for your awesome work ! I ran into several errors when training and testing the model for the tnt dataset.

1. When I use my custom tnt dataset with the NeRF++/Mipnerf360 models, the loss is always NAN

2. When I render the video by
   python3 run.py --ginc configs/dvgo/tnt.gin --scene_name playground --ginb run.run_train=False --ginb run.run_eval=False --ginb run.run_render=True
   The error is as below:

compute_bbox_by_cam_frustrm: xyz_min [-1.2015233 -1.2791067 -1.1872349]
compute_bbox_by_cam_frustrm: xyz_max [1.1897845 1.1122011 1.204073 ]
Traceback (most recent call last):
File "/home/evsjtu2/yetianxiang/NeRF-Factory/run.py", line 231, in
run(
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1605, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
raise proxy.with_traceback(exception.traceback) from None
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1582, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/home/evsjtu2/yetianxiang/NeRF-Factory/run.py", line 191, in run
trainer.predict(model, data_module, ckpt_path=ckpt_path)
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 874, in predict
return call._call_and_handle_interrupt(
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 921, in _predict_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1037, in _run
self._call_setup_hook() # allow user to setup lightning_module in accelerator environment
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1286, in _call_setup_hook
self._call_lightning_module_hook("setup", stage=fn)
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1342, in _call_lightning_module_hook
output = fn(*args, **kwargs)
File "/home/evsjtu2/yetianxiang/NeRF-Factory/src/model/dvgo/model.py", line 399, in setup
self.render_kwargs = self.get_render_kwargs()
File "/home/evsjtu2/yetianxiang/NeRF-Factory/src/model/dvgo/model.py", line 816, in get_render_kwargs
"stepsize": self._stepsize,
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LitDVGO' object has no attribute '_stepsize'
In call to configurable 'run' (<function run at 0x7fdde3e2aa60>)
compute_bbox_by_cam_frustrm: xyz_min [-1.2015233 -1.2791067 -1.1872349]
compute_bbox_by_cam_frustrm: xyz_max [1.1897845 1.1122011 1.204073 ]
/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py:438: UserWarning: Error handling mechanism for deadlock detection is uninitialized. Skipping check.
rank_zero_warn("Error handling mechanism for deadlock detection is uninitialized. Skipping check.")
Traceback (most recent call last):
File "run.py", line 231, in
run(
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1605, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
raise proxy.with_traceback(exception.traceback) from None
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1582, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "run.py", line 191, in run
trainer.predict(model, data_module, ckpt_path=ckpt_path)
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 874, in predict
return call._call_and_handle_interrupt(
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 90, in launch
return function(*args, **kwargs)
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 921, in _predict_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1037, in _run
self._call_setup_hook() # allow user to setup lightning_module in accelerator environment
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1286, in _call_setup_hook
self._call_lightning_module_hook("setup", stage=fn)
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1342, in _call_lightning_module_hook
output = fn(*args, **kwargs)
File "/home/evsjtu2/yetianxiang/NeRF-Factory/src/model/dvgo/model.py", line 399, in setup
self.render_kwargs = self.get_render_kwargs()
File "/home/evsjtu2/yetianxiang/NeRF-Factory/src/model/dvgo/model.py", line 816, in get_render_kwargs
"stepsize": self._stepsize,
File "/home/evsjtu2/miniconda3/envs/nerf_factory/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LitDVGO' object has no attribute '_stepsize'
In call to configurable 'run' (<function run at 0x7ff3b72a6940>)

Here is my environment setting:

_libgcc_mutex 0.1 main anaconda
_openmp_mutex 5.1 1_gnu anaconda
aiohttp 3.8.3 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
async-timeout 4.0.2 pypi_0 pypi
attrs 22.1.0 pypi_0 pypi
beautifulsoup4 4.11.1 pypi_0 pypi
blas 1.0 mkl
brotlipy 0.7.0 py38h0a891b7_1004 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2022.12.7 ha878542_0 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
cffi 1.14.6 py38ha65f79e_0 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
click 8.1.3 pypi_0 pypi
cloudpickle 2.2.0 pypi_0 pypi
configargparse 1.5.3 pypi_0 pypi
contourpy 1.0.6 pypi_0 pypi
cryptography 37.0.2 py38h2b5fc30_0 conda-forge
cudatoolkit 11.3.1 h9edb442_10 conda-forge
cycler 0.11.0 pypi_0 pypi
docker-pycreds 0.4.0 pypi_0 pypi
ffmpeg 4.3 hf484d3e_0 pytorch
fonttools 4.38.0 pypi_0 pypi
freetype 2.10.4 h0708190_1 conda-forge
frozenlist 1.3.3 pypi_0 pypi
fsspec 2022.11.0 pypi_0 pypi
functorch 0.1.1 pypi_0 pypi
gdown 4.6.0 pypi_0 pypi
gin-config 0.5.0 pypi_0 pypi
gitdb 4.0.10 pypi_0 pypi
gitpython 3.1.29 pypi_0 pypi
gmp 6.2.1 h58526e2_0 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
gym 0.26.2 pypi_0 pypi
gym-notices 0.0.8 pypi_0 pypi
idna 3.4 pyhd8ed1ab_0 conda-forge
imageio 2.23.0 pypi_0 pypi
imageio-ffmpeg 0.4.7 pypi_0 pypi
importlib-metadata 5.2.0 pypi_0 pypi
intel-openmp 2021.4.0 h06a4308_3561
jpeg 9e h166bdaf_1 conda-forge
kiwisolver 1.4.4 pypi_0 pypi
lame 3.100 h7f98852_1001 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.38 h1181459_1 anaconda
libffi 3.3 he6710b0_2 anaconda
libgcc-ng 11.2.0 h1234567_1 anaconda
libgomp 11.2.0 h1234567_1 anaconda
libiconv 1.17 h166bdaf_0 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libstdcxx-ng 11.2.0 h1234567_1 anaconda
libtiff 4.2.0 hf544144_3 conda-forge
libuv 1.43.0 h7f98852_0 conda-forge
libwebp-base 1.2.2 h7f98852_1 conda-forge
lightning-utilities 0.4.2 pypi_0 pypi
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
matplotlib 3.6.2 pypi_0 pypi
mkl 2021.4.0 h06a4308_640
mkl-service 2.4.0 py38h95df7f1_0 conda-forge
mkl_fft 1.3.1 py38h8666266_1 conda-forge
mkl_random 1.2.2 py38h1abd341_0 conda-forge
multidict 6.0.3 pypi_0 pypi
ncurses 6.3 h5eee18b_3 anaconda
nettle 3.6 he412f7d_0 conda-forge
networkx 2.8.8 pypi_0 pypi
ninja 1.11.1 pypi_0 pypi
numpy 1.23.4 py38h14f4228_0
numpy-base 1.23.4 py38h31eccc5_0
olefile 0.46 pyh9f0ad1d_1 conda-forge
opencv-python 4.6.0.66 pypi_0 pypi
openh264 2.1.1 h780b84a_0 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1q h7f8727e_0 anaconda
packaging 22.0 pypi_0 pypi
pandas 1.5.2 pypi_0 pypi
pathtools 0.1.2 pypi_0 pypi
pillow 9.3.0 pypi_0 pypi
pip 22.1.2 py38h06a4308_0 anaconda
piqa 1.2.2 pypi_0 pypi
promise 2.3 pypi_0 pypi
protobuf 3.20.3 pypi_0 pypi
psutil 5.9.4 pypi_0 pypi
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyopenssl 22.0.0 pyhd8ed1ab_1 conda-forge
pyparsing 3.0.9 pypi_0 pypi
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.8.13 h12debd9_0 anaconda
python-dateutil 2.8.2 pypi_0 pypi
python_abi 3.8 2_cp38 conda-forge
pytorch 1.11.0 py3.8_cuda11.3_cudnn8.2.0_0 pytorch
pytorch-lightning 1.8.5.post0 pypi_0 pypi
pytorch-mutex 1.0 cuda pytorch
pytz 2022.7 pypi_0 pypi
pywavelets 1.4.1 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
readline 8.1.2 h7f8727e_1 anaconda
requests 2.28.1 pyhd8ed1ab_1 conda-forge
scikit-image 0.19.3 pypi_0 pypi
scipy 1.9.3 pypi_0 pypi
seaborn 0.12.1 pypi_0 pypi
sentry-sdk 1.12.1 pypi_0 pypi
setproctitle 1.3.2 pypi_0 pypi
setuptools 61.2.0 py38h06a4308_0 anaconda
shortuuid 1.0.11 pypi_0 pypi
six 1.16.0 pyh6c4a22f_0 conda-forge
smmap 5.0.0 pypi_0 pypi
soupsieve 2.3.2.post1 pypi_0 pypi
sqlite 3.39.2 h5082296_0 anaconda
tifffile 2022.10.10 pypi_0 pypi
tk 8.6.12 h1ccaba5_0 anaconda
torch-efficient-distloss 0.1.3 pypi_0 pypi
torch-scatter 2.1.0 pypi_0 pypi
torchaudio 0.11.0 py38_cu113 pytorch
torchmetrics 0.11.0 pypi_0 pypi
torchvision 0.12.0 py38_cu113 pytorch
tqdm 4.64.1 pypi_0 pypi
typing_extensions 4.4.0 pyha770c72_0 conda-forge
urllib3 1.26.13 pyhd8ed1ab_0 conda-forge
wandb 0.13.7 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0 anaconda
xz 5.2.5 h7f8727e_1 anaconda
yarl 1.8.2 pypi_0 pypi
zipp 3.11.0 pypi_0 pypi
zlib 1.2.12 h7f8727e_2 anaconda
zstd 1.5.0 ha95c52a_0 conda-forge

Could you please give me some hints why the errors appear, thanks so much for your help in advance.

when I try python3 run.py --ginc configs/nerf/blender.gin --scene chair
after training,this happened,can you help me?

logs:
`Global seed set to 220901
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
[rank: 0] Global seed set to 220901
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
[rank: 1] Global seed set to 220901
[rank: 1] Global seed set to 220901
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2

distributed_backend=nccl
All distributed processes registered. Starting with 2 processes

You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set torch.set_float32_matmul_precision('medium' | 'high') which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:613: UserWarning: Checkpoint directory /home/disk1/lcy/pycharm/nerf-factory/logs/nerf_blender_chair_220901 exists and is not empty.
rank_zero_warn(f"Checkpoint directory {dirpath} exists and is not empty.")
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]

| Name | Type | Params

0 | model | NeRF | 1.2 M

1.2 M Trainable params
0 Non-trainable params
1.2 M Total params
4.767 Total estimated model params size (MB)
/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1609: PossibleUserWarning: The number of training batches (50) is smaller than the logging interval Trainer(log_every_n_steps=100). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
rank_zero_warn(
Epoch 0: 0%| | 0/3957 [00:00<?, ?it/s]/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py:362: LightningDeprecationWarning: The NVIDIA/apex AMP implementation has been deprecated upstream. Consequently, its integration inside PyTorch Lightning has been deprecated in v1.9.0 and will be removed in v2.0.0. The LitNeRF.optimizer_step() hook is overridden, including the using_native_amp argument. Removing this argument will avoid this message, you can expect it to return True.
rank_zero_deprecation(
Epoch 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3957/3957 [20:52<00:00, 3.16it/s, loss=0.208, v_num=0, train/psnr1=9.500, train/psnr0=9.490]Downloading: "https://github.com/richzhang/PerceptualSimilarity/raw/master/lpips/weights/v0.1/vgg.pth" to /home/lcy/.cache/torch/hub/checkpoints/vgg.pth█████████████████████████████████████████████████████| 3907/3907 [20:45<00:00, 3.14it/s]
Downloading: "https://github.com/richzhang/PerceptualSimilarity/raw/master/lpips/weights/v0.1/vgg.pth" to /home/lcy/.cache/torch/hub/checkpoints/vgg.pth
Traceback (most recent call last):
File "run.py", line 229, in
run(
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1605, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
raise proxy.with_traceback(exception.traceback) from None
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1582, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "run.py", line 173, in run
trainer.fit(model, data_module, ckpt_path=ckpt_path)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
call._call_and_handle_interrupt(
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 88, in launch
return function(*args, **kwargs)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
results = self._run_stage()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1191, in _run_stage
self._run_train()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1214, in _run_train
self.fit_loop.run()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
self.advance(*args, **kwargs)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
self.on_advance_end()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 250, in on_advance_end
self._run_validation()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 308, in _run_validation
self.val_loop.run()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 206, in run
output = self.on_run_end()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 180, in on_run_end
self._evaluation_epoch_end(self._outputs)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 288, in _evaluation_epoch_end
self.trainer._call_lightning_module_hook(hook_name, output_or_outputs)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1356, in _call_lightning_module_hook
output = fn(*args, **kwargs)
File "/home/disk1/lcy/pycharm/nerf-factory/src/model/nerf/model.py", line 297, in validation_epoch_end
lpips_mean = self.lpips_each(rgbs, targets).mean()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/disk1/lcy/pycharm/nerf-factory/src/model/interface.py", line 62, in lpips_each
lpips_model = LPIPS(network="vgg").to(device=self.device)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/piqa/lpips.py", line 131, in init
self.weights = nn.ParameterList(get_weights(network=network))
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/piqa/lpips.py", line 38, in get_weights
weights = torch.hub.load_state_dict_from_url(
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/torch/hub.py", line 727, in load_state_dict_from_url
download_url_to_file(url, cached_file, hash_prefix, progress=progress)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/torch/hub.py", line 593, in download_url_to_file
u = urlopen(req)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 1397, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 1357, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1131)>
In call to configurable 'run' (<function run at 0x7faccce38430>)
Traceback (most recent call last):
File "/home/disk1/lcy/pycharm/nerf-factory/run.py", line 229, in
run(
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1605, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
raise proxy.with_traceback(exception.traceback) from None
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1582, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/home/disk1/lcy/pycharm/nerf-factory/run.py", line 173, in run
trainer.fit(model, data_module, ckpt_path=ckpt_path)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
call._call_and_handle_interrupt(
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
results = self._run_stage()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1191, in _run_stage
self._run_train()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1214, in _run_train
self.fit_loop.run()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
self.advance(*args, **kwargs)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
self.on_advance_end()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 250, in on_advance_end
self._run_validation()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 308, in _run_validation
self.val_loop.run()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 206, in run
output = self.on_run_end()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 180, in on_run_end
self._evaluation_epoch_end(self._outputs)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 288, in _evaluation_epoch_end
self.trainer._call_lightning_module_hook(hook_name, output_or_outputs)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1356, in _call_lightning_module_hook
output = fn(*args, **kwargs)
File "/home/disk1/lcy/pycharm/nerf-factory/src/model/nerf/model.py", line 297, in validation_epoch_end
lpips_mean = self.lpips_each(rgbs, targets).mean()
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/disk1/lcy/pycharm/nerf-factory/src/model/interface.py", line 62, in lpips_each
lpips_model = LPIPS(network="vgg").to(device=self.device)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/piqa/lpips.py", line 131, in init
self.weights = nn.ParameterList(get_weights(network=network))
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/piqa/lpips.py", line 38, in get_weights
weights = torch.hub.load_state_dict_from_url(
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/torch/hub.py", line 727, in load_state_dict_from_url
download_url_to_file(url, cached_file, hash_prefix, progress=progress)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/torch/hub.py", line 593, in download_url_to_file
u = urlopen(req)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 1397, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/home/lcy/anaconda3/envs/nerf_factory/lib/python3.8/urllib/request.py", line 1357, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1131)>
In call to configurable 'run' (<function run at 0x7fce11f52550>)
Epoch 0: 100%|██████████| 3957/3957 [22:18<00:00, 2.96it/s, loss=0.208, v_num=0, train/psnr1=9.500, train/psnr0=9.490] `

In the code we get a .ckpt model as an output. Could you let me know/point to some resource on how I can convert this model into a 3D mesh? Also would there be different mesh conversion codes for different algorithms (say nerf and mip-nerf)?

How can we use metashape data type or nerfstudio data in this repo?

I'm trying to launch MipNerf traing but like in #12 I have huge CPU memory consumption while GPU idles. I guess it's because data loader and sampler working on CPU.
Is there a way to move this processing to GPU?

I've been trying to run the rendering code from the nerf_factory using the pretrained models available in the repository, specifically from this path: https://huggingface.co/nrtf/nerf_factory/tree/main/mipnerf360_nerf_360_v2_garden_220901.
I ran the following command:
python3 run.py --ginc configs/mipnerf360/360_v2.gin --scene garden --ginb run.run_train=False --ginb run.run_eval=False --ginb run.run_render=True
Unfortunately, the resulting rendered images and videos have very poor quality. I also tried the llff_flower model and encountered similar poor results.

I would like to know if there is an issue with the rendering code or if I'm missing something in the process. Any help or guidance would be greatly appreciated.

MipNerf360 implementation diverges from the formal implementation and creates two different Prop modules.
https://github.com/kakaobrain/nerf-factory/blob/main/src/model/mipnerf360/model.py#L229

In the article and formal implementation, a single Prop module is created and used (twice).
https://github.com/google-research/multinerf/blob/main/internal/models.py#L99
https://github.com/google-research/multinerf/blob/main/internal/models.py#L221

as indicated by the article, in the beginning of 5. Optimization section:
"We do two stages of evaluation and resampling of the proposal MLP"

Hello,

I tried to install the requirements.txt and run the model. But, I'm having problem with torch_scatter library.

Traceback (most recent call last):
File "D:/nerf-factory/run.py", line 26, in
from utils.select_option import select_callback, select_dataset, select_model
File "D:\nerf-factory\utils\select_option.py", line 19, in
from src.model.dvgo.model import LitDVGO
File "D:\nerf-factory\src\model\dvgo\model.py", line 23, in
import src.model.dvgo.dcvgo as dcvgo
File "D:\nerf-factory\src\model\dvgo\dcvgo.py", line 17, in
import torch_scatter
File "C:\Users\clair\Anaconda3\envs\nerf_factory\lib\site-packages\torch_scatter_init_.py", line 16, in
torch.ops.load_library(spec.origin)
File "C:\Users\clair\Anaconda3\envs\nerf_factory\lib\site-packages\torch_ops.py", line 255, in load_library
ctypes.CDLL(path)
File "C:\Users\clair\Anaconda3\envs\nerf_factory\lib\ctypes_init_.py", line 373, in init
self._handle = _dlopen(self._name, mode)
OSError: [WinError 127] 지정된 프로시저를 찾을 수 없습니다.

To solve this error, I've been told to downgrade torch version to under 1.5.1, but it seems the nerf-factory require torch version higher than 1.11.0.

Could you help me?

Hi,

Thanks again for your great work. I got an error when evaluating RefNeRF on shiny blender dataset:

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
  In call to configurable 'run' (<function run at 0x7fe61d01fdc0>)

The error comes from this line: https://github.com/kakaobrain/NeRF-Factory/blob/ac10296a39d8e5f1940b590ca18e3689e17eadf4/src/model/refnerf/model.py#L153

It seems that torch.autograd.grad only works when backward is called but in the evaluation step there's no backpropagation. Is there anything I missed?

Here is the command I used:
python3 run.py --ginc configs/refnerf/shiny_blender.gin --scene car --ginb run.run_train=False --ckpt_path logs/refnerf_shiny_blender_car_220901/best.ckpt

Thanks in advance!

Hi！When I test the psnr of ref-nerf in ball scene， the checkpoint is the download from your open-source website. the psnr result is 10.56， but the rendering images are pretty good, so could you tell me the reason to the problem? Looking farward to your reply! Thanks!

Here is my run code:
nohup python3 run.py --ginc /paddle/zengpengcheng01/code/NeRF-Factory/configs/refnerf/shiny_blender.gin --scene_name ball --ginb run.run_train=False --ckpt_path logs/best.ckpt > test.log 2>&1

When to support neus

The official website seems to be unable to open when I want to see the comparative effect of several methods. Is it convenient to re-post?

Hi, thank you for this work. I am recently starting using NeRF factory. It looks like transform_pose_llff() will inverse the z and y rotation matrix. Why do we need transform_pose_llff() for the llff dataset? I did not see this in load_llff.py from NeRF GitHub.

Hi, thanks for releasing this great work! I think there is an error in the implementation of LitPlenoxel. You define self.curr_step here and use it to determine lr decay, voxel upsampling, etc. However, it seems that you never increase this value? I add a print(gstep) after this line, and it always prints 0.

Hello author, thank you for your great work! I recently tried to make some improvements based on the mipnerf360 code you reproduced, and then I ran into some confusion and wanted.

I noticed that the mipnerf360 paper mentions training 250k iterations, but I noticed that you set 250k epochs in the code, this brings to a question that I would like to ask.

Pytorch-lightning defaults to completing an epoch of training when all the data has been loaded once, so often an epoch corresponds to hundreds or thousands of steps already completed. So is there something wrong with setting it to 250k epochs? Perhaps it would be more appropriate to set max_step to 250k in pytorch-lightning
https://github.com/kakaobrain/NeRF-Factory/blob/ac10296a39d8e5f1940b590ca18e3689e17eadf4/configs/mipnerf360/360_v2.gin#L11
https://github.com/kakaobrain/NeRF-Factory/blob/ac10296a39d8e5f1940b590ca18e3689e17eadf4/configs/mipnerf360/360_v2.gin#L18

I may not have understood correctly so that hope you can point out the problem, and I apologize for disturbing you. Looking forward to your reply!

Thank you for this repo,
I want to learn nerf model on blender dataset, but i meet this issue.
ㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡ

1. File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\multiprocessing\reduction.py", line 60, in dump
   ForkingPickler(file, protocol).dump(obj)
   OSError: [Errno 22] Invalid argument

2. File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\multiprocessing\spawn.py", line 126, in _main
   self = reduction.pickle.load(from_parent)
   _pickle.UnpicklingError: pickle data was truncated
   ㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡ

Epoch 0: 0%| | 0/257813 [00:00<?, ?it/s]Traceback (most recent call last):
File "run.py", line 231, in
run(
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\gin\config.py", line 1605, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\gin\utils.py", line 41, in augment_exception_message_and_reraise
raise proxy.with_traceback(exception.traceback) from None
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\gin\config.py", line 1582, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "run.py", line 175, in run
trainer.fit(model, data_module, ckpt_path=ckpt_path)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 608, in fit
call._call_and_handle_interrupt(
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\trainer\call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 650, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1112, in _run
results = self._run_stage()
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1191, in _run_stage
self._run_train()
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1214, in _run_train
self.fit_loop.run()
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\loops\loop.py", line 199, in run
self.advance(*args, **kwargs)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 267, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\loops\loop.py", line 194, in run
self.on_run_start(*args, **kwargs)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py", line 160, in on_run_start
_ = iter(data_fetcher) # creates the iterator inside the fetcher
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 179, in iter
self._apply_patch()
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 120, in _apply_patch
apply_to_collections(self.loaders, self.loader_iters, (Iterator, DataLoader), _apply_patch_fn)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 156, in loader_iters
return self.dataloader_iter.loader_iters
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 556, in loader_iters
self._loader_iters = self.create_loader_iters(self.loaders)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 596, in create_loader_iters
return apply_to_collection(loaders, Iterable, iter, wrong_dtype=(Sequence, Mapping))
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\lightning_utilities\core\apply_func.py", line 51, in apply_to_collection
return function(data, *args, **kwargs)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\torch\utils\data\dataloader.py", line 430, in iter
self._iterator = self._get_iterator()
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\torch\utils\data\dataloader.py", line 381, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\site-packages\torch\utils\data\dataloader.py", line 1034, in init
w.start()
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
OSError: [Errno 22] Invalid argument
In call to configurable 'run' (<function run at 0x00000160C9D12E50>)
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\JIHUNBAE\anaconda3\envs\nerf\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated

I follow requirement.txt but because of my cuda version, i change the pytorch(=1.13.1), torch-scatter version.
Can you check and give me some solutions ?
Thank you very much.

Hi, first thanks for your work. I got an error when I run run python run.py --ginc configs/nerf/blender.gin --scene chair --ginb run.run_train=False. Could you give me any suggestion. thanks in advance.

/home/xxx/anaconda3/envs/nerf_factory/bin/python /home/xxx/Documents/tools/nerf-factory/run.py --ginc configs/nerf/blender.gin --scene chair --ginb run.run_train=False 
Global seed set to 220901
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Restoring states from the checkpoint path at logs/nerf_blender_chair_220901/best.ckpt
Traceback (most recent call last):
  File "/home/xxx/Documents/tools/nerf-factory/run.py", line 231, in <module>
    run(
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/xxx/Documents/tools/nerf-factory/run.py", line 183, in run
    trainer.test(model, data_module, ckpt_path=ckpt_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 794, in test
    return call._call_and_handle_interrupt(
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 842, in _test_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1056, in _run
    self._restore_modules_and_callbacks(ckpt_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 998, in _restore_modules_and_callbacks
    self._checkpoint_connector.resume_start(checkpoint_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 90, in resume_start
    loaded_checkpoint = self.trainer.strategy.load_checkpoint(checkpoint_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 359, in load_checkpoint
    return self.checkpoint_io.load_checkpoint(checkpoint_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/lightning_fabric/plugins/io/torch_io.py", line 84, in load_checkpoint
    raise FileNotFoundError(f"Checkpoint at {path} not found. Aborting training.")
FileNotFoundError: Checkpoint at logs/nerf_blender_chair_220901/best.ckpt not found. Aborting training.
  In call to configurable 'run' (<function run at 0x7f9d3edd0310>)

Hi, first thanks for your work.
I am new to deep learning and have just started to learn about NERF. When I run the command

python run.py --ginc configs/nerf/blender.gin --scene chair --ginb run.run_train=False

I encounter some errors. Could you please guide me on how to resolve these errors? Also, is it possible that these errors are occurring because I haven't trained the model yet and the checkpoint is a place to save the trained model? thank you in advance.

/home/xxx/anaconda3/envs/nerf_factory/bin/python /home/xxx/Documents/tools/nerf-factory/run.py --ginc configs/nerf/blender.gin --scene chair --ginb run.run_train=False 
Global seed set to 220901
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Restoring states from the checkpoint path at logs/nerf_blender_chair_220901/best.ckpt
Traceback (most recent call last):
  File "/home/xxx/Documents/tools/nerf-factory/run.py", line 231, in <module>
    run(
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/xxx/Documents/tools/nerf-factory/run.py", line 183, in run
    trainer.test(model, data_module, ckpt_path=ckpt_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 794, in test
    return call._call_and_handle_interrupt(
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 842, in _test_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1056, in _run
    self._restore_modules_and_callbacks(ckpt_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 998, in _restore_modules_and_callbacks
    self._checkpoint_connector.resume_start(checkpoint_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 90, in resume_start
    loaded_checkpoint = self.trainer.strategy.load_checkpoint(checkpoint_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 359, in load_checkpoint
    return self.checkpoint_io.load_checkpoint(checkpoint_path)
  File "/home/xxx/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/lightning_fabric/plugins/io/torch_io.py", line 84, in load_checkpoint
    raise FileNotFoundError(f"Checkpoint at {path} not found. Aborting training.")
FileNotFoundError: Checkpoint at logs/nerf_blender_chair_220901/best.ckpt not found. Aborting training.
  In call to configurable 'run' (<function run at 0x7f9d3edd0310>)

Hi! Thanks for your wonderful work. I have a question,When I run mipnerf360 at the room scene in the 360_v2 dataset, no depth maps are generated. And I don't seem to see any code in the source code to output depth maps. I am not sure if I misunderstood. Looking forward to your answer.

Hi,

I am having issues running this project. I suspect it's an issue with the version of pytorch_lightning.

Here's the output:

> python3 -m run --ginc configs/nerf/blender.gin
Traceback (most recent call last):
  File "!/anaconda3/envs/nerf_factory/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "~/anaconda3/envs/nerf_factory/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "~/nerf-factory/run.py", line 23, in <module>
    from pytorch_lightning.plugins import DDPPlugin
ImportError: cannot import name 'DDPPlugin' from 'pytorch_lightning.plugins' (~/anaconda3/envs/nerf_factory/lib/python3.8/site-packages/pytorch_lightning/plugins/__init__.py)

System:

I'm running on a Mac studio with MacOS Ventura 13.3.1.

Here are the versions installed by pip -r requirements.txt:

> pip3 install -r requirements.txt
...
Successfully built torch-scatter torch-efficient-distloss pathtools
Installing collected packages: multidict, frozenlist, yarl, smmap, attrs, async-timeout, aiosignal, soupsieve, packaging, gitdb, fsspec, aiohttp, tqdm, torchmetrics, tifffile, setproctitle, sentry-sdk, scipy, PyYAML, PyWavelets, psutil, protobuf, pathtools, networkx, lightning-utilities, lazy-loader, imageio, GitPython, filelock, docker-pycreds, Click, beautifulsoup4, appdirs, wandb, torch-scatter, torch-efficient-distloss, scikit-image, pytorch-lightning, piqa, opencv-python, ninja, imageio-ffmpeg, gin-config, gdown, functorch, configargparse
Successfully installed Click-8.1.3 GitPython-3.1.31 PyWavelets-1.4.1 PyYAML-6.0 aiohttp-3.8.4 aiosignal-1.3.1 appdirs-1.4.4 async-timeout-4.0.2 attrs-22.2.0 beautifulsoup4-4.12.2 configargparse-1.5.3 docker-pycreds-0.4.0 filelock-3.11.0 frozenlist-1.3.3 fsspec-2023.4.0 functorch-0.1.1 gdown-4.7.1 gin-config-0.5.0 gitdb-4.0.10 imageio-2.27.0 imageio-ffmpeg-0.4.8 lazy-loader-0.2 lightning-utilities-0.8.0 multidict-6.0.4 networkx-3.1 ninja-1.11.1 opencv-python-4.7.0.72 packaging-23.1 pathtools-0.1.2 piqa-1.2.2 protobuf-4.22.3 psutil-5.9.4 pytorch-lightning-2.0.1.post0 scikit-image-0.20.0 scipy-1.9.1 sentry-sdk-1.19.1 setproctitle-1.3.2 smmap-5.0.0 soupsieve-2.4 tifffile-2023.4.12 torch-efficient-distloss-0.1.3 torch-scatter-2.1.1 torchmetrics-0.11.4 tqdm-4.65.0 wandb-0.14.2 yarl-1.8.2

Similar problems found in:

Lightning-AI/pytorch-lightning#17191

That hints to this migration guide:

https://lightning.ai/docs/pytorch/stable/upgrade/migration_guide.html

Possible temporary solution

Use pytorch_lightning==1.9.5 in requirements.txt

Hi,

Thanks for your awesome work. I evaluate MIpNeRF360 on the "room" scene of 360_v2 dataset, the PSNR result can match the orficial JAX implementation (even slightly better). But there are some 'floaters' artifacts in the results, which I think is related to the distortion loss. Could you please list some different implementation points comparied with the JAX one?

I only have a Linux machine with CUDA 12.4, currently can not reinstall with CUDA 11, is it possible running on that?

I am a newcomer who has just started using NeRF,and when running nerf from nerf-factory, I noticed that it is very slow.
I used
run --ginc configs/nerf/llff.gin --scene fern
But it occupies larger GPU memory and provides slower speed compared to nerf-pytorch.
It seems like
0%| | 300/200000 [00:27<5:10:40, 10.71it/s][TRAIN] Iter: 300 Loss: 0.0341896116733551 PSNR: 17.609861373901367 in nerf-pytorch
and
Epoch 0: 0%| | 295/250070 [05:21<75:30:57, 1.09s/it, loss=0.0281, v_num=0, train/psnr1=15.90, train/psnr0=15.90] in nerf-factory-NeRF
I noticed that no one has mentioned this issue before,so it might be specific to my setup. Can anyone help me with this?
My GPU is tesla V100 with 32GB of memory.

I'm trying to use mipnerf360, but the

.ckpt doesn't seem to be saving

Do you know if there is something I need to set?

Hi, this is an awesome repository. The performance and the visualization comparison on the webpage look great. However, I didn't find the visualization code and cannot render images after training models. Do you have any suggestions?

Hello,

Thank you so much for this implementation of ref nerf!

I have a question regarding on the gradient of the density to compute the normals. Why is raw_density.sum() used as outputs in torch.autograd as shown here.

Is it different than calculating the gradient as
torch.autograd.grad(raw_density, means, grad_outputs=torch.ones_like(raw_density), retain_graph=True). ?

Hi,

Thanks for the awesome framework for NeRF! I have a question about the data loader speed, what would be the expected time for loading shiny blender dataset for example?
I'm using a NVIDIA T4 device with a single GPU and cuda 11.7, and the estimated loading time for testing data loader is around 4.5 hours. Here is the command I used to load your provided checkpoints, the config file is unchanged:
python3 run.py --ginc configs/nerf/shiny_blender.gin --scene car --ginb run.run_train=False --ckpt_path logs/nerf_shiny_blender_car_220901/best.ckpt
I created the conda environment as suggested in README, then also tried to upgrade to latest cuda toolkit, torch and torch lightning version for cuda11.7, in both cases the testing data loader needs more than 4 hours to load.
I would really appreciate it if you can shed some light on this issue. Please let me know if you need more information. Thanks in advance!

While training MipNeRF360 on dataset nerf_360_v2 and it turned out loss nan
Config as followed:

360-v2 Specific Arguments

run.dataset_name = "nerf_360_v2"
run.datadir = "data/nerf_360_v2"
#run.dataset_name = "nerf_360

LitData.batch_sampler = "all_images"

MipNeRF Standard Specific Arguments

run.model_name = "mipnerf360"
run.max_steps = 1000000
run.log_every_n_steps = 100

LitData.load_radii = True
LitData.batch_size = 4096
LitData.chunk = 4096
LitData.use_pixel_centers = True
LitData.epoch_size = 250000

LitDataNeRF360V2.near = 0.1
LitDataNeRF360V2.far = 1e6

MipNeRF360.opaque_background = True

run.grad_max_norm = 0.001

hi,thanks for your remarkable work,and i am going to introduce this code as baseline.
I found this "We also provide convenient visualizers for NeRF researchers." on the page,however,i could not find the visualizers tools as showed in the top of introduction.
if it is convenient for you ,could you please support the visualizers tools,thanks a lot.