Epochs remaining gone negative and stuck on optimising weights about hub HOT 13 CLOSED

Hello @tlong123!
First of all, I am sorry for the inconvenience. This must be an issue on our end.
Can you share your model ID (it is in the model page URL) so we can investigate this and prevent this from happening in the future? Also, I will refund the credits used for training this model.

from hub.

yeah sure its : y9dPahdYO4ShpAfpD6pG

from hub.

Hi Burhan! only other things I can think of are that they were trained on the Ultralytics cloud, and both disconnected at some point before 100 epochs so I had to click resume training

from hub.

@tlong123 I have refunded the credits used for training the second model.
Our team is investigating the cause of the issue with your trainings and exploring ways to prevent such incidents in the future. I will keep you updated on our progress.

Please accept our apologies for the inconvenience caused.

from hub.

Hello @tlong123!
Great news! Our team has released a fix for the issue you reported. You should no longer experience this problem in new Cloud Training sessions.
Thanks for your patience!

from hub.

👋 Hello @tlong123, thank you for raising an issue about Ultralytics HUB 🚀! Please visit our HUB Docs to learn more:

• Quickstart. Start training and deploying YOLO models with HUB in seconds.
• Datasets: Preparing and Uploading. Learn how to prepare and upload your datasets to HUB in YOLO format.
• Projects: Creating and Managing. Group your models into projects for improved organization.
• Models: Training and Exporting. Train YOLOv5 and YOLOv8 models on your custom datasets and export them to various formats for deployment.
• Integrations. Explore different integration options for your trained models, such as TensorFlow, ONNX, OpenVINO, CoreML, and PaddlePaddle.
• Ultralytics HUB App. Learn about the Ultralytics App for iOS and Android, which allows you to run models directly on your mobile device.
  • iOS. Learn about YOLO CoreML models accelerated on Apple's Neural Engine on iPhones and iPads.
  • Android. Explore TFLite acceleration on mobile devices.
• Inference API. Understand how to use the Inference API for running your trained models in the cloud to generate predictions.

If this is a 🐛 Bug Report, please provide screenshots and steps to reproduce your problem to help us get started working on a fix.

If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.

We try to respond to all issues as promptly as possible. Thank you for your patience!

from hub.

Additionally, other than deleting the model I can't see any way to stop it from actively costing me money in its current state, and I don't want to delete it as I've paid money for it to be trained!

from hub.

@tlong123 hello! First off, thanks for reaching out and detailing the issue you're facing with the training process on Ultralytics HUB. 🌟 It sounds like you've stumbled upon a rare glitch, particularly with the training session not properly concluding and going into negative epochs – this indeed is not expected behavior.

Rest assured, we prioritize both the performance and the billing concerns of our users. Here's a couple of steps you can take:

1. If you haven't already, please attempt to manually stop the training session. While you've mentioned an issue with stopping the model without deleting it, there should be a stop or pause option available in the UI for your training session.
2. Regarding the unexpected billing, we absolutely understand the importance of fair billing practices. We recommend reaching out to our support through the official communication channels mentioned in our documentation. They will be able to look into your billing details and make necessary adjustments based on the glitch.

We're here to ensure a smooth and efficient training experience. Your detailed report is incredibly valuable for us to improve and rectify such issues. If anything else comes up or if you have further questions, don't hesitate to update the issue or reach out to our support team.

Thanks for your patience and understanding. 👍

from hub.

I've attached a screenshot of where its at now. It eventually disconnected by itself

from hub.

@tlong123 I have refunded the credits used for training this model.
Our team is investigating the cause of the issue with your training and exploring ways to prevent such incidents in the future. I will keep you updated on our progress.

from hub.

Thanks Sergiu! I think I should mention here that I've just had the same issue with another model I've tried to train - model ID is PmUMb8RaZufqpap3APXi

from hub.

@tlong123 I'm trying to help with troubleshooting the issue and wanted to ask for more information about you configurations. If you could share:

• Model (YOLOv8n | s | m | ....)
• Task (detect, pose, etc.)
• Training settings (imgsz, batch, etc.)

I can better help out with testing something more closely resembling your setup.

from hub.

Looks like most of the info is actually in the screenshots you shared, so I'll test with those. Anything else you can recall that's not shown in the screenshot would be helpful to know

from hub.