justincdavis / trtutils Goto Github PK

Add simple builders and associated interfaces in Python. Have the builder be able to target different GPU/DLA on a per layer basis, but only implement builder for all GPU, all DLA, or custom setup.

Additionally, add a CLI tool which will use these builders to sort of mimic trtexec.

Add a CLI and potentially a CLI rerouter which can call trtexec for the user. Option to find trtexec and give path to user as well as the forwarder.

While PyTorch models can have their precision changed at runtime or input size altered since they are dynamic graphs, a TensorRT compiled engine cannot do such a switch. This means that if you want to swap between precision levels at runtime, multiple TensorRT engines must be compiled ahead-of-time and then loaded when needed during runtime.

Since the TRTEngine provides a simple interface for using compiled engines as a Python object, it is possible to use those to "simulate" the runtime dynamic behavior of a PyTorch model.

It will take as input the list of TRTEngines which were compiled and then it would have methods to switch dtype, device, input size, etc.. whatever can be utilized from TRTEngine