Comments (11)
As mentioned in #12 , I got some problem with SSD TRT plan distribution on TRTIS. And yes, there are several custom plugins in the SSD sample such as NMS, Normalize, PriorBox and so on. While serving with the engine plan serialized from the SSD sample, it seems that TRTIS did not deserialize these part correctly.
It would be great if the TRT plugin mechanism were fully supported in TRTIS. And I think a complete tutorial with SSD TRT plan might be helpful, as detection task is quite common in these days.
from server.
Has anyone tried using libnvinfer_plugin.so
in LD_PRELOAD
? I am unable to get it to work even though it has all the plugins.
Still stuck at
tensorrt_1 | E0601 15:02:09.585748 34 logging.cc:43] getPluginCreator could not find plugin Normalize_TRT version 1 namespace
tensorrt_1 | E0601 15:02:09.585764 34 logging.cc:43] Cannot deserialize plugin Normalize_TRT
and similar loglines for NMS_TRT and PriorBox_TRT
from server.
As a work-around for TRTIS not yet having TRT plugin support, you should be able to use LD_PRELOAD in many cases. For example, assuming your TRT plugins are compiled into trtplugins.so.
$ LD_PRELOAD=trtplugins.so trtserver --model-store=/tmp/models ...
You can load multiple plugin libraries with LD_PRELOAD. The limitation of this approach is that the plugins must be managed separately from the model store itself. And more seriously, if there are plugin name conflicts across multiple plugin libraries there is no way to handle it. But if you have just a single plugin library this should get you unblocked.
from server.
How would you suggest I compile plugins into trtplugins.so?
from server.
The TensorRT documentation has description and examples on how to create TensorRT plugin libraries: https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#extending
from server.
I am still facing the same issue:
LD_PRELOAD=libnvinfer_plugin.so /opt/tensorrtserver/bin/trtserver --model-store=/ScriptsNModels/TestModelZoo
E0612 20:28:44.454662 46 logging.cc:43] getPluginCreator could not find plugin Normalize_TRT version 1 namespace
E0612 20:28:44.454670 46 logging.cc:43] Cannot deserialize plugin Normalize_TRT
E0612 20:28:44.454697 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454715 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454736 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454742 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454763 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454777 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454795 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454813 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454831 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454838 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454857 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454864 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454900 46 logging.cc:43] getPluginCreator could not find plugin NMS_TRT version 1 namespace
E0612 20:28:44.454907 46 logging.cc:43] Cannot deserialize plugin NMS_TRT
Note that I have a serialized plan file for SSD from running sampleSSD in TensorRT code.
Can anyone also provide a config.pbtxt for SSD?
from server.
I have implemented IPluginFactory with plugin and set
ICaffeParser* parser = createCaffeParser(); parser->setPluginFactoryExt(&pluginFactory);
and generate tensorrt model . when I employ it to server, it occur error
ERROR: Not a valid serialized plugin
And I also implemented plugin with IPluginV2 IPluginCreator. and generate lib.so
and use LD_PRELOAD=trtplugins.so but also occur the same error!
from server.
We're working on validating TRT plugin support in TRTIS. It appears to work in some cases but not all. We'll update here once we learn more.
from server.
We will be adding support for the default provided plugins shipped with TensorRT for now. We may add custom plugins support in the future if possible.
This means that plugins mentioned here will be supported by default in TRTIS in the future releases.
from server.
Support for default plugins as mentioned here is added on master and will be in 19.07. Please retry your model/s to let me know if it is not fixed for you.
from server.
I've successfully imported custom ops imported from onnx parser and registered there via LD_PRELOAD workaround.
from server.
Related Issues (20)
- Questions about serving PyTorch LLM in Python backend with token streaming using "Decoupled Mode" HOT 4
- Multiple CPU instances results in decreasing of infer speed. HOT 1
- Inference in Triton ensemble model is much slower than single model in Triton HOT 2
- repeated answer:When I use vllm with Qwen-7b-chat the generated text is x lnot end until the maength, with the repeated answer HOT 1
- Calling index search inside Triton python backend HOT 1
- TensorRT 8.6.3.1 package in Python PyPy for Triton Nvidia Inference Server version > 24.01 HOT 3
- Unable to use pytoch library with libtorch backend when using triton inference server In-Process python API HOT 4
- model analyser stucks HOT 2
- Question to huggingface model using triton
- Questions about input and output shape in model configuration when batch size is 1 HOT 2
- Model Management HOT 1
- passing input data HOT 1
- Python backend status zombie but Tritonserver `v2/health` still return 200 OK HOT 1
- Onnxruntime backend doesn't load model when container is running on Ubuntu HOT 1
- Cant build python+onnx+ternsorrtllm backends r24.04 HOT 3
- increase chunk size for streaming with tensorrtllm_backend
- trt accelerator
- Can't build the Docker image r24.04 on Azure Nvidia VMI HOT 3
- Build error when building new image on top of the `nvcr.io/nvidia/tritonserver:24.04-py3-sdk` container image from NGC HOT 2
- Feature Questions
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from server.