Comments (3)
That's weird.
Disabling logging won't break anything, but if you want logging, try to update pytorch lightning, in case it's a bug in a specific version, or call .item() on the losses to be logged.
from surfemb.
Hi
It shouldn't use that much memory. I'm training on a single RTX 2080 with 8 GB of ram.
Also, I don't see why changing the number of scenes would affect GPU memory usage.
I'm not sure what's going on there.
Let me know, if you find out.
BR, Rasmus
from surfemb.
Hi, thanks for the quick response.
I have noticed that there is a memory leak on surface_embedding.py
I guess it is logging during training and thus it increases the memory usage in every 4-5 seconds. I have checked it with "nvidia-smi" command during training
So I have commented on these 3 lines and now it even works with batch-size=32 and multiple threads :)
self.log(f'{log_prefix}/loss', loss)
self.log(f'{log_prefix}/mask_loss', mask_loss)
self.log(f'{log_prefix}/nce_loss', nce_loss)
I hope this disabled logging wont break things
thanks again for the great work,
from surfemb.
Related Issues (20)
- 'ambient_occlusion' filter throws core dumped for tless models HOT 6
- error when resuming from checkpoint
- question about bbox and mask HOT 2
- Question about inference data HOT 9
- question about pose score. HOT 4
- assert len(obj_ids) > 0 AssertionError HOT 1
- Depth inference fails due to value error
- How do you create the `depth`, `mask` and `mask_visib` from the `rgb` image? HOT 1
- 您好,请问如何用mask rcnn训练icbin数据集呢,谢谢
- What does the `current_pose` array values mean? HOT 1
- Pose Refiner Diverges HOT 4
- Could you tell me how to train icbin dataset in mask rcnn ? HOT 1
- instance = self.instances[i].copy()-1 IndexError: list index out of range HOT 1
- Number of threads explodes when training HOT 1
- Can you upload a RGBD version of the code
- A question about ambiguities HOT 2
- Any Plan to upgrade? HOT 8
- dataset settings
- CosyPose BOP-trained detection results
- Why to normalize coordinates of objects when rendering? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from surfemb.