Have the plan to support fp8 inference? about keras HOT 9 CLOSED

@james77777778 any thoughts on this?

from keras.

If the model is trained with fp8, it is ready for inference. We can fix the scaling factor and drop the amax_history if we don't train the model in the future.

If the model is not trained with fp8 and we don't plan to train it in the future, we need a mechanism to calibrate it. Calibration is similar to fp8 training but we only need to compute the scaling factor offline with an additional calibration dataset.

I'm unsure whether we should add the calibration logic into Keras.

from keras.

Thanks for your reply. It seems keras need more discussion to decide whether to support fp8 calibration. Maybe you can update the latest progress if have any result in the future.

from keras.

And for fp8 inference after fp8 training, keras seems support not well. Can we add is_training argument in float8_call to decide whether to compute new scale? New amax history is also not need.

from keras.

And for fp8 inference after fp8 training, keras seems support not well. Can we add is_training argument in float8_call to decide whether to compute new scale? New amax history is also not need.

Since #19682 has been merged, you can set training=False for the layer (or model) to skip the computation of both the scaling factor and amax history.
The variable for amax history will still be retained but it should occupy a small portion of memory.

from keras.

Thanks, will test it soon.

from keras.

Hi @lingzhi98 Do you test it? It should be working as expected since the PR has been merged.

from keras.

Sorry for slow response. Have checked it, close this issue.

from keras.

Thank you!

from keras.