Custom model using timed training doesn't allow its use about hub HOT 9 OPEN

👋 Hello @JhonFrederick, thank you for raising an issue about Ultralytics HUB 🚀! Please visit our HUB Docs to learn more:

• Quickstart. Start training and deploying YOLO models with HUB in seconds.
• Datasets: Preparing and Uploading. Learn how to prepare and upload your datasets to HUB in YOLO format.
• Projects: Creating and Managing. Group your models into projects for improved organization.
• Models: Training and Exporting. Train YOLOv5 and YOLOv8 models on your custom datasets and export them to various formats for deployment.
• Integrations. Explore different integration options for your trained models, such as TensorFlow, ONNX, OpenVINO, CoreML, and PaddlePaddle.
• Ultralytics HUB App. Learn about the Ultralytics App for iOS and Android, which allows you to run models directly on your mobile device.
  • iOS. Learn about YOLO CoreML models accelerated on Apple's Neural Engine on iPhones and iPads.
  • Android. Explore TFLite acceleration on mobile devices.
• Inference API. Understand how to use the Inference API for running your trained models in the cloud to generate predictions.

If this is a 🐛 Bug Report, please provide screenshots and steps to reproduce your problem to help us get started working on a fix.

If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.

We try to respond to all issues as promptly as possible. Thank you for your patience!

from hub.

Hello @JhonFrederick!

First of all, please accept our apologies for the inconvenience caused.

Based on the screenshot you shared, you used Epochs training (not Timed training) but I would like to investigate this further. Can you please share your model ID (you can find it in the URL) here?

Also, looking at the right side of your screenshot, I can see negative epochs which makes me think that you might face an issue we are currently trying to solve (#622).

from hub.

I remember setting the timed training to a value of 1 day, but now I'm not sure. Mainly because I am currently running other test with Epoch Training and the way the information is displayed was not the same as the attempt shown in the screenshot. But since you mention the issue, it could be due to that.
Model ID: FuVEbxOoAcWJCFA7fa9m

Edit
When I ran my model (FuVEbxOoAcWJCFA7fa9m), a few minutes later I reviewed the billing data and the information corresponded to the time entered (1 day), the total value was already calculated. But with epoch training this is calculated over time. I don't know if it's relevant, but I noticed this now that I'm running other model (with Epoch Training).

from hub.

@JhonFrederick hello again, and thank you for providing the model ID and additional details. It clarifies your situation significantly.

Given the information and your experience with both timed and epoch training, it indeed sounds like the unusual behavior you encountered with the model FuVEbxOoAcWJCFA7fa9m might be related to the issue we're currently addressing.

I appreciate your patience and understanding as we work towards resolving this. In the meantime, it seems you've correctly identified different billing behaviors between timed and epoch training—timed training estimates your total cost upfront based on the duration, whereas epoch training's cost accumulates over time.

Your observations are indeed relevant and help us ensure the platform works as expected for everyone. We'll keep you updated on our progress with the mentioned issue. Please, stay tuned! 😊

from hub.

According to the above screen, my second model finished (with epoch training) with ID: tj2HLEVdErYxgunZzH9Z, but when I go to preview or deployment tab, I get the following message "Model not trained".

Attached a screenshot of the billing summary, which shows the different attempts to complete the training.

Please tell me in this case what I could be doing wrong so that it doesn't allow me to use the trained model?

I appreciate your help again in advance

from hub.

@JhonFrederick hello again!

Thanks for reaching out with these details. It looks like an issue on our end where the model's training status hasn't correctly updated in the UI, despite the training completion. This misalignment is likely causing the "Model not trained" message you're seeing.

For now, could you try refreshing the page or logging out and back into the platform to see if that helps sync the status? Sometimes, a simple refresh can resolve such discrepancies.

If the issue persists, rest assured, we're here to help! We'll investigate further using the model ID tj2HLEVdErYxgunZzH9Z you provided and ensure your model becomes accessible for preview and deployment.

Again, we truly appreciate your patience and feedback as we work to improve the platform. Stay tuned! 🌟

from hub.

@JhonFrederick
Something went wrong with the first model (FuVEbxOoAcWJCFA7fa9m) and we are not yet sure what. Our team is investigating this issue. Regarding the second model (tj2HLEVdErYxgunZzH9Z), it appears that although the model finished training, the final upload of weights failed, which is why the model is unusable.
We have refunded the account balance you used and kindly ask you to start the training process again from scratch. Once again, our apologies for the inconvenience caused.

from hub.

Hi,

I tried again with another test using epoch training, but again I had problems, I attached proof of this.
Model ID: xrGz5bRPDQvMniPK8eIR

Billing information

In this case the training was going well up to a certain point, after 75%, I had to retry the training a couple of times until it was completed, but without the possibility of using the model, until it finally ended in the state shown

from hub.

Hello @JhonFrederick!

I apologize once again for the inconvenience.

Based on our internal tests, we've observed that, in approximately 10% of cases, the final weights upload fails. This results in the model being stuck at 100%. If the training is resumed, the session fails since the training has already completed. Our team is currently working on updating the logic for uploading weights to the Ultralytics HUB to prevent this issue.

Meanwhile, we have refunded the account balance you used.

CC @hassaanfarooq01

from hub.