unisim's People
Contributors
Stargazers
Watchers
unisim's Issues
torch.cuda.OutOfMemoryError on RTX3090
Hi,
thanks for the great work!
I run unisim on 3090, during traing, it will occupy more than 24G GPU memory and will cause torch.cuda.OutOfMemoryError.
I noticed that GPU usage is constantly changing, around 24G. What could I do?
400 (2.00%) 311.871 ms 1 h, 41 m, 52 s 132.86 K
410 (2.05%) 312.729 ms 1 h, 42 m, 6 s 132.29 K
420 (2.10%) 303.868 ms 1 h, 39 m, 10 s 134.83 K
430 (2.15%) 361.016 ms 1 h, 57 m, 45 s 121.40 K
440 (2.20%) 362.957 ms 1 h, 58 m, 19 s 120.57 K
450 (2.25%) 308.685 ms 1 h, 40 m, 35 s 132.94 K
460 (2.30%) 308.850 ms 1 h, 40 m, 35 s 132.83 K
470 (2.35%) 304.781 ms 1 h, 39 m, 12 s 134.43 K
480 (2.40%) 315.080 ms 1 h, 42 m, 30 s 132.05 K
490 (2.45%) 319.595 ms 1 h, 43 m, 55 s 130.18 K
----------------------------------------------------------------------------------------------------
Viewer running locally at: http://localhost:7007 (listening on 0.0.0.0)
Printing profiling stats, from longest to shortest duration in seconds
Trainer.train_iteration: 0.3007
UniSimPipeline.get_train_loss_dict: 0.1705
ADPipeline.get_eval_loss_dict: 0.1642
Trainer.eval_iteration: 0.0003
torch.cuda.OutOfMemoryError:
CUDA out of memory. Tried to allocate 936.00 MiB (GPU 0; 23.49 GiB total capacity; 8.18 GiB already allocated; 460.69 MiB free; 11.54 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Problem of the rendered image
Hi, I'm trying to train a model using unisim plug-in, in the tensorboard, the rendered image is like this, which is very weird ... Do you know what causes this? Thanks.
Code Error Out Due to Shape Misalignment Between rgb and image
The code errored out at step 2490, at this line of code
Line 786 in 09b1df9
I checked rgb shape changed from [4096, 3, 3, 3] to [4, 96, 96, 3] at step 2490. Can anyone explain why there would be a shape change here, and how to handle this bug?
Traceback (most recent call last):
File "nerfstudio/scripts/train.py", line 278, in
entrypoint()
File "nerfstudio/scripts/train.py", line 269, in entrypoint
main(
File "nerfstudio/scripts/train.py", line 254, in main
launch(
File "nerfstudio/scripts/train.py", line 196, in launch
main_func(local_rank=0, world_size=world_size, config=config)
File "nerfstudio/scripts/train.py", line 107, in train_loop
trainer.train()
File "/neurad-studio/nerfstudio/engine/trainer.py", line 316, in train
loss, loss_dict, metrics_dict = self.train_iteration(step)
File "/neurad-studio/nerfstudio/utils/profiler.py", line 112, in inner
out = func(*args, **kwargs)
File "/neurad-studio/nerfstudio/engine/trainer.py", line 555, in train_iteration
_, loss_dict, metrics_dict = self.pipeline.get_train_loss_dict(step=step)
File "/neurad-studio/nerfstudio/utils/profiler.py", line 112, in inner
out = func(*args, **kwargs)
File "/neurad/lib/python3.8/site-packages/unisim/unisim_pipeline.py", line 75, in get_train_loss_dict
metrics_dict = self.model.get_metrics_dict(model_outputs, batch)
File "/neurad/lib/python3.8/site-packages/unisim/unisim.py", line 786, in get_metrics_dict
metrics_dict["psnr"] = self.psnr(rgb.detach(), image)
File "/neurad/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/neurad/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/neurad/lib/python3.8/site-packages/torchmetrics/metric.py", line 311, in forward
self._forward_cache = self._forward_reduce_state_update(*args, **kwargs)
File "/neurad/lib/python3.8/site-packages/torchmetrics/metric.py", line 380, in _forward_reduce_state_update
self.update(*args, **kwargs)
File "/neurad/lib/python3.8/site-packages/torchmetrics/metric.py", line 492, in wrapped_func
raise err
File "/neurad/lib/python3.8/site-packages/torchmetrics/metric.py", line 482, in wrapped_func
update(*args, **kwargs)
File "/neurad/lib/python3.8/site-packages/torchmetrics/image/psnr.py", line 132, in update
sum_squared_error, num_obs = _psnr_update(preds, target, dim=self.dim)
File "/neurad/lib/python3.8/site-packages/torchmetrics/functional/image/psnr.py", line 73, in _psnr_update
sum_squared_error = torch.sum(torch.pow(preds - target, 2))
RuntimeError: The size of tensor a (96) must match the size of tensor b (3) at non-singleton dimension 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.