Automatically cache dataset about hub HOT 6 CLOSED

@glenn-jocher I don't know if I will manage to, but I will try my best to help!

from hub.

👋 Hello @Powercube7, thank you for raising an issue about Ultralytics HUB 🚀! Please visit https://ultralytics.com/hub to learn more, and see our ⭐️ HUB Guidelines to quickly get started uploading datasets and training YOLOv5 models.

If this is a 🐛 Bug Report, please provide screenshots and steps to recreate your problem to help us get started working on a fix.

If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.

We try to respond to all issues as promptly as possible. Thank you for your patience!

from hub.

@Powercube7 that's a great idea! We check available RAM and HDD space in YOLOv5 here:
https://github.com/ultralytics/yolov5/blob/813eba85b266fe46b0ac02a62fce8b25e3eeabac/utils/__init__.py#L24-L30

The main question is how to estimate a cached dataset size without actually caching the entire thing. Maybe we could attempt to cache 1% of each train and val splits and extrapolate the full requirements from there, and proceed to cache if they extrapolation estimates <80% RAM or <50% HDD usage for full caching.

If we apply this to the base YOLOv5 repo then HUB trainings will naturally inherit this.

from hub.

@Powercube7 if you'd like to start a PR on this that would be a huge help!

Please see our ✅ Contributing Guide to get started.

from hub.

@Powercube7 awesome, thanks!

from hub.

Removing TODO and closing issue as HUB trainings starting from web dashboard now automatically attempt RAM caching :)

from hub.