This tutorial describes the steps to convert your tensorflow saved model to TensorRT engine and how to run it on video. The instructions are applicable for running the TRT engine on any Nvidia Jetson device with Jetpack installed. Also you can folow this tutorial to inference the TRT engine right on your PC with Nvidia Gpu, but you need to set up cuda,cudnn,tensorrt libraries for this.

• Convert saved model to ONNX using tf2onnx python library
• Simplify ONNX model using onnxsimplifier python library
• Copy siplified ONNX model from your PC to Jetson
• Create TensorRT engine from ONNX model on the Jetson using trtexec
• Adjust given script for inferencing(change the model shape, input names, adjust input preprocesing and output processing), and run the script

Install tf2onnx:

pavlo@pc:~$ pip3 install tf2onnx

Convert tf saved model to onnx:

pavlo@pc:~$ python3 -m tf2onnx.convert --saved-model "my_tf_saved_model" --output "my_model.onnx" --opset 11

Install onnxsimplifyer:

pavlo@pc:~$ pip3 install onnxsim

Simplify onnx model:

pavlo@pc:~$ python3 -m onnxsim my_model.onnx my_model_simpl.onnx --input-shape 1,224,224,3

Now copy my_model_simpl.onnx to Jetson device

jetson@myjetsonnano:~$ /usr/src/tensorrt/bin/trtexec --onnx=/home/myjetsonnano/Desktop/Projects/trt_tutorial/my_model_simpl.onnx --shapes=\'input_1:0\':1x224x224x3 --saveEngine=/home/myjetsonnano/Desktop/Projects/trt_tutorial/engine.trt --verbose --explicitBatch

export PATH=/usr/local/cuda-10.2/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64:$LD_LIBRARY_PATH
sudo apt install python3-pip
pip3 install cyton
pip3 install pycuda
sudo apt-get libprotobuf-dev
sudo apt-get protobuf-compiler

Open run_trt_engine_on_video.py and edit input name, input shape and path to the video:

add data preprocessing if needed:

Now run the script!

P.S if your model has 2 outputs, you should uncoment commented lines