Comments (6)
π Hello @willuhmjs, thank you for raising an issue about Ultralytics HUB π! Please visit our HUB Docs to learn more:
- Quickstart. Start training and deploying YOLO models with HUB in seconds.
- Datasets: Preparing and Uploading. Learn how to prepare and upload your datasets to HUB in YOLO format.
- Projects: Creating and Managing. Group your models into projects for improved organization.
- Models: Training and Exporting. Train YOLOv5 and YOLOv8 models on your custom datasets and export them to various formats for deployment.
- Integrations. Explore different integration options for your trained models, such as TensorFlow, ONNX, OpenVINO, CoreML, and PaddlePaddle.
- Ultralytics HUB App. Learn about the Ultralytics App for iOS and Android, which allows you to run models directly on your mobile device.
- Inference API. Understand how to use the Inference API for running your trained models in the cloud to generate predictions.
If this is a π Bug Report, please provide screenshots and steps to reproduce your problem to help us get started working on a fix.
If this is a β Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.
We try to respond to all issues as promptly as possible. Thank you for your patience!
from hub.
@willuhmjs hello! π
Thank you for bringing this issue to our attention. It seems you're encountering a problem with data not uploading to the Ultralytics HUB when training with multiple GPUs. This is indeed unusual, and we appreciate your detailed report.
A few things to consider that might help resolve this issue:
-
Check Network Connectivity: Ensure that your High-Performance Computing (HPC) environment has stable internet access throughout the training process. Network instability could cause the upload process to fail.
-
Verify HUB Version: Make sure you're using the latest version of the Ultralytics HUB. Outdated versions might have compatibility issues or bugs that have been addressed in newer releases.
-
Review HPC Restrictions: Some HPC environments have strict firewall rules or network policies that might restrict data uploads. It's worth checking with your system administrator if there are any such limitations in place.
-
Single GPU Test: As a diagnostic step, try running a smaller training job with a single GPU to see if the issue persists. This can help isolate whether the problem is specifically related to multi-GPU setups.
-
Logs and Error Messages: If possible, review any logs or error messages generated during the training process. These can provide valuable clues about what might be going wrong.
For further troubleshooting and guidance, please refer to our documentation at https://docs.ultralytics.com/hub. If the issue persists, feel free to provide additional details such as logs or error messages, and we'll be happy to assist further.
Thanks for your contribution to the Ultralytics community! π
from hub.
I'm having the same problem, using 8 A100 for calculations
from hub.
I'm having the same problem, using 8 A100 for calculations
one gpu just work fine
from hub.
@darkkun0824 hello again! π
It's insightful to know that the issue is specifically occurring in a multi-GPU setup, especially with high-end GPUs like the A100s, but works fine with a single GPU. This narrows down the scope of the problem significantly.
Given this additional context, here are a few more tailored suggestions:
-
Distributed Training Configuration: Ensure that your distributed training setup is correctly configured. Multi-GPU training often requires specific settings to manage data synchronization and communication effectively across GPUs.
-
Resource Monitoring: Monitor system resources during training. High-end GPUs like the A100 can process data very quickly, potentially leading to bottlenecks elsewhere in the system that might affect data upload processes.
-
Batch Size Adjustment: Consider adjusting your batch size. While larger batch sizes are feasible with more powerful GPUs, they might introduce unexpected behavior in how data is processed and uploaded.
-
Update Drivers and Libraries: Ensure that your NVIDIA drivers and CUDA libraries are up to date. Compatibility issues between drivers, CUDA, and PyTorch can lead to unexpected behavior in multi-GPU setups.
-
Consult HPC Documentation: Given the specialized nature of HPC environments, it's also a good idea to consult any available documentation or support resources specific to your HPC setup. There might be known issues or configurations specific to multi-GPU training in these environments.
If these steps do not resolve the issue, we recommend gathering detailed logs from your training sessions and reaching out for further support through our documentation portal at https://docs.ultralytics.com/hub. Detailed logs can provide crucial insights for deeper troubleshooting.
Thank you for your patience and for contributing to the Ultralytics community! We're here to help. π
from hub.
π Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.
For additional resources and information, please see the links below:
- Docs: https://docs.ultralytics.com
- HUB: https://hub.ultralytics.com
- Community: https://community.ultralytics.com
Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!
Thank you for your contributions to YOLO π and Vision AI β
from hub.
Related Issues (20)
- Train and predict multiple classes detecting for single object YOLO8 HOT 3
- Inquiry About Live Video Feed Deployment on Ultralytics Hub HOT 10
- on upload : No response from the server. HOT 8
- INTEGRATIONGUNDETECTION HOT 2
- Is it possible to use the checkpoints that the platform makes if the model is overtrained? HOT 5
- A.S.D.O. Proyect HOT 8
- Project is disconnected HOT 11
- YOLOv10 ? HOT 1
- Ultralytics HUB API Issue HOT 2
- Ultralytics Pro member doesn't work dataset HOT 2
- instances.keypoints: index is out of bounds HOT 2
- Labeling tool in the process HOT 4
- This account is currently being deleted. HOT 4
- Model parallelization for inference on large images HOT 2
- 'yolov8n-pose.yaml' HOT 19
- det HOT 2
- No response from the server. HOT 14
- canΒ΄t remove Roboflow linked workspaces HOT 9
- How to download training logs to view in tensorflow? HOT 1
- Team member invitation is not sending the email invitation HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hub.