Comments (5)
atobiszei
commented on June 4, 2024
I found out that in the Triton image there are 2 versions of OpenVINO, and one of them is missing libraries from OpenVINO:
root@8bc8eab2d6ce:/# find -name "*openvino*" | grep -v 2330 | grep -v 23\.3\.0 | grep -v LICENSE | grep -v "libopenvino_c\|libopenvino.so"
./opt/tritonserver/backends/openvino
./opt/tritonserver/backends/openvino/libopenvino_intel_gna_plugin.so
./opt/tritonserver/backends/openvino/libopenvino_tensorflow_lite_frontend.so
./opt/tritonserver/backends/openvino/libtriton_openvino.so
./opt/tritonserver/backends/openvino/libopenvino_onnx_frontend.so
./opt/tritonserver/backends/openvino/libopenvino_auto_batch_plugin.so
./opt/tritonserver/backends/openvino/libopenvino_pytorch_frontend.so
./opt/tritonserver/backends/openvino/libopenvino_paddle_frontend.so
./opt/tritonserver/backends/openvino/libopenvino_intel_gpu_plugin.so
./opt/tritonserver/backends/openvino/libopenvino_tensorflow_frontend.so
./opt/tritonserver/backends/openvino/libopenvino_gapi_preproc.so
./opt/tritonserver/backends/openvino/libopenvino_auto_plugin.so
./opt/tritonserver/backends/openvino/libopenvino_hetero_plugin.so
./opt/tritonserver/backends/openvino/libopenvino_intel_cpu_plugin.so
./opt/tritonserver/backends/onnxruntime/libopenvino_onnx_frontend.so
./opt/tritonserver/backends/onnxruntime/libonnxruntime_providers_openvino.so
./opt/tritonserver/backends/onnxruntime/libopenvino_ir_frontend.so
./opt/tritonserver/backends/onnxruntime/libopenvino_intel_cpu_plugin.so
./opt/tritonserver/backends/onnxruntime/libopenvino_tensorflow_frontend.so
So this problem most likely affects also TF Lite, PaddlePaddle & Pytorch model formats.
Culprit is most likely here:
https://github.com/triton-inference-server/onnxruntime_backend/blob/48cc4f132a451a8dfebe501583d88acb5243dc38/tools/gen_ort_dockerfile.py#L311
as not all libraries are copied.
from server.
krishung5
commented on June 4, 2024
@tanmayv25 for vis.
from server.
tanmayv25
commented on June 4, 2024
@atobiszei The openVINO backend in Triton does not support models saved in savedModel format. Read about Triton's OpenVINO backend here: https://github.com/triton-inference-server/openvino_backend?tab=readme-ov-file#openvino-backend
You'd have to convert savedModel using model optimizer tool into OpenVINO IR model (.xml and .bin files). Then place these files into the model directory instead of TF savedmodel dir.
from server.
atobiszei
commented on June 4, 2024
@tanmayv25
This paragraph states otherwise:
https://github.com/triton-inference-server/openvino_backend#loading-non-default-model-format.
When I removed ONNX backend from Triton image & tuned shape parameters in config it worked fine.
from server.
tanmayv25
commented on June 4, 2024
Thanks for the correction. It seems the feature to load savedmodel has been added recently.
We need to revisit the Triton image to make sure that there are no conflicting dependencies. The openVINO backend should be using its own installation of openVINO library instead of the one held in onnxruntime.
This could also help us installing different OV between OV and ONNXRuntime backends.
from server.
Related Issues (20)
- No trtllm tag in ngc for 24.05 HOT 4
- No 24.05-trtllm-python-py3 in NGC Repo HOT 2
- YOLOv8n-poses is giving me a negative output error HOT 2
- Automatically unload (oldest) models when memory is full HOT 2
- Specific structure for ensemble model may causes deadlock
- Windows 10 docker build Error "Could not locate a complete Visual Studio instance" HOT 2
- A Confusion about prefetch HOT 2
- What is the correct way to run inference in parallel in Triton?
- Support histogram custom metric in Python backend HOT 2
- Backend support for .keras files?
- triton-inference-server cannot be started HOT 1
- Incorrect asset tritonserver2.35.0-jetpack5.1.2-update-2.tgz HOT 1
- How does Triton implement one instance to handle multiple requests simultaneously? HOT 1
- ONNX backend with TensorRT optimizer sometimes fails to start HOT 1
- Any example of triton-vllm with c++ client?
- Tritonserver for FIL backend not starting HOT 1
- Why is my model in ensemble receiving out-of-order input HOT 2
- Add TT-Metalium as a backend
- unexpected datatype TYPE_INT64 for inference input ,expecting TYPE_INT32 HOT 1
- triton malloc fail HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from server.