Different servers to load the trained PredEst model issue about openkiwi HOT 4 CLOSED

Hello @HanchengDeng,

The behaviour you're describing is unexpected. There's no extra operation to make the trained model work on a new machine.

What could be happening is an unintended consequence of the saving behaviour we implemented in OpenKiwi.
You see, when you're training a model and you run the validation scoring the model is instantly saved if it is one of the best runs. If that is the case, in order to avoid using too much disk space, we save the best_model.torch as a path to you're actual best model (which will have a different name). So if the best model is the exact last validation, this best_model will only be a link to the actual file where the model is saved. You can check the behaviour on trainers/callbacks.py save_latest.

Of course, this works when you're in the same machine where you have all the other checkpoints saved but won't work when you move to a new server.

Can you check if this is the case?

from openkiwi.

Thank you for your response. The issue had been solved already. I can make the trained model work on a new machine with no extra operation now. Ironically, it was the CPU limitation of my cloud server that led to the issue. I expanded the CPU size of the cloud server, and the trained model could work perfectly.

Thank you very much!

from openkiwi.

Hmmm, I can't really understand how a CPU limitation might cause the trained model not to be able to predict..

I'm glad your issue is solved, please let me know if you gain some insight into why this happened!

from openkiwi.

This issue has been solved.

Feel free to re-open if you still have problems

from openkiwi.