Add a notebook example for training using TPUs about keras-io HOT 10 CLOSED

@sayakpaul I am also interested in collaborating with you to write this guide. I was initially planning for a notebook example without custom-training loops, to keep it simple as an example.

I fully agree having an end to end detailed guide about using Keras is necessary.

from keras-io.

@kurianbenoy do you plan to cover how one should use TPUs in custom training loops as well? If so, I am willing to collaborate. @fchollet I think an end-to-end tutorial on how one should should use TPUs for custom training loops would be really helpful. WDYT?

from keras-io.

If you want to write a detailed guide about using TPUs, feel free to send a PR (as guides/training_on_tpu.py. Note that it will be a fairly difficult guide to write. It should cover everything -- fit, callbacks, model saving, custom training loops -- since it will be the official TPU guide :)

from keras-io.

@fchollet thanks for passing along. How about we did it in parts to maintain brevity and simplicity? I am also interested to see what @kurianbenoy has to say here.

from keras-io.

@kurianbenoy when would you like to start? Here's a notebook we can extensively refer to while writing the part about using TPUs in custom training loops. Let me know your thoughts.

@fchollet would you like the guide in two parts i.e. part I for the simpler one and part II for the more complex one or would you like to see one?

from keras-io.

@fchollet would you like the guide in two parts i.e. part I for the simpler one and part II for the more complex one or would you like to see one?

Ideally just one guide, but it could have 2 sections :)

from keras-io.

Is this what you are looking for? https://www.tensorflow.org/guide/tpu

from keras-io.

I'm aware of @yashk2810. But this guide proposed in the PR would cover two scenarios including the things to keep in mind while using TPUs. Two scenarios:

• The classic fit and compile one.
• Custom training loops.

from keras-io.

That guide covers both those scenarios right?

from keras-io.

I agree but the official TensorFlow guide leaves out on the little details that are needed in order to make TPU-based training work. For example:

• How should one set the batch size that is specifically suited for TPU-based training?
• How should one go about setting the learning rate?

It also lacks guidance on aggregating the loss after each training step which is absolutely crucial to understand while doing any form of distributed training in general.

from keras-io.