How to initialize and use YOLO-NAS quantized models as listed in docs? about super-gradients HOT 4 CLOSED

The doc contains details about how to do QAT and PQT, but it doesn't mention the procedure to load the then-trained model for inference.

Specifically, I want to know how to load and use the quantized YOLO-NAS FP16 and INT8 models listed in the README documentation.

The following line loads the model in FP32 precision.

yolo_nas_s = super_gradients.training.models.get("yolo_nas_s", pretrained_weights="coco")

What are the next steps I need to perform to load and use the quantized version (FP16 and INT8) as mentioned in the docs?

Edit: Removed question about QSP and QCI blocks

from super-gradients.

Join the discussion on DagsHub!

from super-gradients.

Hello, @veb-101 , I refer you to this document. Let me know if this helps!

from super-gradients.

To load the model for inference you should compile it to TensorRT. The INT8 QAT model is convertible to an ONNX with Q/DQ layers which are then compiled into actual INT8 quantization.

You can follow the code of QATTrainer to see how the FP32 model is being quantized and calibrated, then exported to ONNX.

To get INT8 model, as I mentioned, you need to compile it to TensorRT, which can be achieved using trtexec. Here's a snippet:

/usr/src/tensorrt/bin/trtexec --onnx=$1.onnx --workspace=2048 --avgRuns=100 --duration=15 --int8 --fp16 --saveEngine=$1.engine

Read more about TensorRT in the NVIDIA's docs.

from super-gradients.