KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/vizua/vid2avatar/code/lib/datasets/dataset.py", line 186, in __getitem__
"img_size": images["img_size"]
KeyError: 'img_size'
# images = {"rgb": samples["rgb"].astype(np.float32)}
images = {
"rgb": samples["rgb"].astype(np.float32),
"img_size": img_size
}
$ python test.py
test.py:9: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
@hydra.main(config_path="confs", config_name="base")
/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/hydra/_internal/hydra.py:127: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
configure_logging=with_log_configuration,
Global seed set to 42
Working dir: /home/vizua/vid2avatar/outputs/Video/parkinglot
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
Restoring states from the checkpoint path at checkpoints/epoch=6299-loss=0.01887552998960018.ckpt
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Loaded model weights from checkpoint at checkpoints/epoch=6299-loss=0.01887552998960018.ckpt
Testing: 0it [00:00, ?it/s]/home/vizua/vid2avatar/code/v2a_model.py:219: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
1) // pixel_per_batch
mise.MISE(32, 4, 0):
points shape: 35937
points: []
/home/vizua/vid2avatar/code/lib/utils/meshing.py:49: FutureWarning: marching_cubes_lewiner is deprecated in favor of marching_cubes. marching_cubes_lewiner will be removed in version 0.19
level=level_set)
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [6,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [6,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
.................................................................................
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [3,0,0], thread: [93,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [3,0,0], thread: [94,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [3,0,0], thread: [95,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Error executing job with overrides: []
Traceback (most recent call last):
File "test.py", line 39, in <module>
main()
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/hydra/main.py", line 99, in decorated_main
config_name=config_name,
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/hydra/_internal/utils.py", line 401, in _run_hydra
overrides=overrides,
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/hydra/_internal/utils.py", line 458, in _run_app
lambda: hydra.run(
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/hydra/_internal/utils.py", line 461, in <lambda>
overrides=overrides,
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "test.py", line 36, in main
trainer.test(model, testset, ckpt_path=checkpoint)
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 907, in test
return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule)
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 683, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 950, in _test_impl
results = self._run(model, ckpt_path=self.tested_ckpt_path)
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1195, in _run
self._dispatch()
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1271, in _dispatch
self.training_type_plugin.start_evaluating(self)
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 206, in start_evaluating
self._results = trainer.run_stage()
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1282, in run_stage
return self._run_evaluate()
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1330, in _run_evaluate
eval_loop_results = self._evaluation_loop.run()
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/loops/base.py", line 145, in run
self.advance(*args, **kwargs)
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 110, in advance
dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/loops/base.py", line 145, in run
self.advance(*args, **kwargs)
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 122, in advance
output = self._evaluation_step(batch, batch_idx, dataloader_idx)
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 213, in _evaluation_step
output = self.trainer.accelerator.test_step(step_kwargs)
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 244, in test_step
return self.training_type_plugin.test_step(*step_kwargs.values())
File "/home/vizua/anaconda3/envs/v2a/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 222, in test_step
return self.model.test_step(*args, **kwargs)
File "/home/vizua/vid2avatar/code/v2a_model.py", line 250, in test_step
batch_inputs = {"uv": inputs["uv"][:, indices],
RuntimeError: CUDA error: device-side assert triggered
Testing: 0%| | 0/42 [00:06<?, ?it/s]