Comments (4)
@mustafakasap
If you really want text, you can output it as a byte array. Hopefully the next model you have in your ensemble will be able to consume it - or perhaps it could just deal with a boolean?
BTW - what about 0? It's neither positive nor negative.
Here's an example that would return byte arrays with text - I think it's as close as it gets to strings:
@dali.pipeline_def(batch_size=8, num_threads=min(mp.cpu_count(), 4), device_id=0)
def pipe():
input0 = dali.fn.external_source(device="cpu", name="DALI_INPUT_0")
sum = dali.fn.reductions.sum(input0)
# Conver the strings to numpy arrays of uint8 and wrap the arrays into constant nodes
pos = dali.types.Constant(np.array(list(bytes("Positive", "utf8"))))
neg = dali.types.Constant(np.array(list(bytes("Negative", "utf8"))))
is_positive = sum >= 0
is_negative = 1 - is_positive
return is_positive * pos + is_negative * neg```
from dali_backend.
Hi @mustafakasap,
In the 2ns and 3rd cases the input0
is pruned as it is not connected to any output (one of the optimization DALI does building the processing graph). You may need to return it as it is just to make it works.
From the examples you provided I'm not fully sure if DALI is the right tool for the processing you want. One of the possible solutions is to:
- write a custom DALI plugin
- write a custom TRITON backend in python
- make this a part of the model
from dali_backend.
Hello, @mustafakasap
Unfortunately, what you wrote cannot be achieved for multiple reasons:
- You have an
if
statement which looks like it's trying to act on a result of an operator - that's not how DALI operates. The variablesum
is not a value, but rather aDataNode
object. The functionpipe
only defines what the pipeline and does not take part in the execution - the only thing that's preserved until run-time are the connections between operators and DataNodes. str
is not a valid return type of DALI operators - they can only return tensors with the following element types: int8/16/32/64, uint8/16/32/64, float16, float32, float64.
from dali_backend.
Thank you Michal for the responses. Let me provide more context about what I want to achieve (rather than above simplified case):
I will be using above Dali model connected to the output of yoloV3 (or any other ml model) in an ensemble model. so whatever arrives into my dali model is a 1D array of structs with bounding box coordinates, confidence etc. I will process this 1D array, iterate over each struct, construct a python string (which will be Json output of bounding boxes etc) and return this encoded JSon string as 1D uint array...
Here: https://github.com/triton-inference-server/server/blob/main/docs/examples/model_repository/simple_identity/config.pbtxt seems Triton has string datatype (which is actually uint8 array with -1 dimension)
With below cases, I tried to simulate the above scenario but seems this is not possible in Dali (and sure understand that Dali's design purpose is complete different).
Now I am thinking of writing my own custom dali plugin (like here: https://github.com/triton-inference-server/dali_backend/tree/main/docs/examples/dali_plugin) which will convert i.e. the 1D array (yolov3 output) into 1D array (json inference result string). and use this plugin function in above dali model to solve my issue. Again, would this be the right approach? or should I always handle/parse the raw output from yolov3 inside the Triton Inference client app?
CASE1 (Works fine but seems I cant iterate over the struct array of input0):
Pipeline:
@dali.pipeline_def(batch_size=1, num_threads=min(mp.cpu_count(), 4), device_id=0)
def pipe():
input0 = dali.fn.external_source(device="cpu", name="DALI_SIMPLE_1_INPUT_0")
output0 = dali.types.Constant(np.array(list(bytes("Hello World!", "utf8"))))
identity = dali.fn.reductions.sum(input0) * 0 + 1
return identity * output0
Client Output:
(running inference client in Verbose mode, so below output is directly from inference client API)
infer, metadata ()
model_name: "dali_simple_01"
inputs {
name: "DALI_SIMPLE_1_INPUT_0"
datatype: "UINT8"
shape: 1
shape: 16
}
outputs {
name: "DALI_SIMPLE_1_OUTPUT_0"
}
raw_input_contents: "\000\001\002\003\004\005\006\007\010\t\n\013\014\r\016\017"
model_name: "dali_simple_01"
model_version: "1"
outputs {
name: "DALI_SIMPLE_1_OUTPUT_0"
datatype: "INT64"
shape: 1
shape: 12
}
raw_output_contents: "H\000\000\000\000\000\000\000e\000\000\000\000\000\000\000l\000\000\000\000\000\000\000l\000\000\000\000\000\000\000o\000\000\000\000\000\000\000 \000\000\000\000\000\000\000W\000\000\000\000\000\000\000o\000\000\000\000\000\000\000r\000\000\000\000\000\000\000l\000\000\000\000\000\000\000d\000\000\000\000\000\000\000!\000\000\000\000\000\000\000"
Here if I decode the above raw_output_contents into string (from asnumpy...), I get : Hello World!
CASE2 (Fails, dont know exactly why but instead of multiplying the result byte array with "is_positive" constant above, I just tried multiplying with integer 1 (ok it is not tensor) ):
Pipeline:
@dali.pipeline_def(batch_size=1, num_threads=min(mp.cpu_count(), 4), device_id=0)
def pipe():
input0 = dali.fn.external_source(device="cpu", name="DALI_SIMPLE_1_INPUT_0")
output0 = dali.types.Constant(np.array(list(bytes("Hello World!", "utf8"))))
return 1 * output0
Client Output:
infer, metadata ()
model_name: "dali_simple_01"
inputs {
name: "DALI_SIMPLE_1_INPUT_0"
datatype: "UINT8"
shape: 1
shape: 16
}
outputs {
name: "DALI_SIMPLE_1_OUTPUT_0"
}
raw_input_contents: "\000\001\002\003\004\005\006\007\010\t\n\013\014\r\016\017"
Traceback (most recent call last):
File "/myworkspace/projects/tritonserver/samples/dali/simple/client.py", line 25, in <module>
results = triton_client.infer(model_name='dali_simple_01',
File "/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py", line 1146, in infer
raise_error_grpc(rpc_error)
File "/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py", line 62, in raise_error_grpc
raise get_error_grpc(rpc_error) from None
tritonclient.utils.InferenceServerException: [StatusCode.UNKNOWN] DALI error: [/opt/dali/dali/pipeline/graph/graph_descr.cc:451] Operator node with name DALI_SIMPLE_1_INPUT_0 not found.
Stacktrace (14 entries):
[frame 0]: /opt/tritonserver/backends/dali/dali/libdali.so(+0x7edbe) [0x7f2e868b3dbe]
[frame 1]: /opt/tritonserver/backends/dali/dali/libdali.so(dali::OpGraph::Node(std::string const&)+0x89) [0x7f2e86938929]
[frame 2]: /opt/tritonserver/backends/dali/dali/libdali.so(daliGetOperatorBackend+0x2f) [0x7f2e869ea52f]
[frame 3]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x50cc4) [0x7f2e874a1cc4]
[frame 4]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x4e6eb) [0x7f2e8749f6eb]
[frame 5]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x4ebec) [0x7f2e8749fbec]
[frame 6]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x2d832) [0x7f2e8747e832]
[frame 7]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x2db96) [0x7f2e8747eb96]
[frame 8]: /opt/tritonserver/backends/dali/libtriton_dali.so(TRITONBACKEND_ModelInstanceExecute+0x150) [0x7f2e8746d210]
[frame 9]: /opt/tritonserver/bin/../lib/libtritonserver.so(+0x30c859) [0x7f2fb83f0859]
[frame 10]: /opt/tritonserver/bin/../lib/libtritonserver.so(+0x109ec0) [0x7f2fb81edec0]
[frame 11]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7f2fb7c39de4]
[frame 12]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f2fb80b7609]
[frame 13]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f2fb7927293]
CASE3 (Fails, instead of above non tensor identity integer value, I tried to construct my own constant tensor like the "is_positive" one but still this fails...):
Pipeline:
@dali.pipeline_def(batch_size=1, num_threads=min(mp.cpu_count(), 4), device_id=0)
def pipe():
input0 = dali.fn.external_source(device="cpu", name="DALI_SIMPLE_1_INPUT_0")
output0 = dali.types.Constant(np.array(list(bytes("Hello World!", "utf8"))))
identity = dali.types.Constant(1)
return identity * output0
Client Output:
infer, metadata ()
model_name: "dali_simple_01"
inputs {
name: "DALI_SIMPLE_1_INPUT_0"
datatype: "UINT8"
shape: 1
shape: 16
}
outputs {
name: "DALI_SIMPLE_1_OUTPUT_0"
}
raw_input_contents: "\000\001\002\003\004\005\006\007\010\t\n\013\014\r\016\017"
Traceback (most recent call last):
File "/myworkspace/projects/tritonserver/samples/dali/simple/client.py", line 25, in <module>
results = triton_client.infer(model_name='dali_simple_01',
File "/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py", line 1146, in infer
raise_error_grpc(rpc_error)
File "/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py", line 62, in raise_error_grpc
raise get_error_grpc(rpc_error) from None
tritonclient.utils.InferenceServerException: [StatusCode.UNKNOWN] DALI error: [/opt/dali/dali/pipeline/graph/graph_descr.cc:451] Operator node with name DALI_SIMPLE_1_INPUT_0 not found.
Stacktrace (14 entries):
[frame 0]: /opt/tritonserver/backends/dali/dali/libdali.so(+0x7edbe) [0x7f2f9afdadbe]
[frame 1]: /opt/tritonserver/backends/dali/dali/libdali.so(dali::OpGraph::Node(std::string const&)+0x89) [0x7f2f9b05f929]
[frame 2]: /opt/tritonserver/backends/dali/dali/libdali.so(daliGetOperatorBackend+0x2f) [0x7f2f9b11152f]
[frame 3]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x50cc4) [0x7f2f9bbc8cc4]
[frame 4]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x4e6eb) [0x7f2f9bbc66eb]
[frame 5]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x4ebec) [0x7f2f9bbc6bec]
[frame 6]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x2d832) [0x7f2f9bba5832]
[frame 7]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x2db96) [0x7f2f9bba5b96]
[frame 8]: /opt/tritonserver/backends/dali/libtriton_dali.so(TRITONBACKEND_ModelInstanceExecute+0x150) [0x7f2f9bb94210]
[frame 9]: /opt/tritonserver/bin/../lib/libtritonserver.so(+0x30c859) [0x7f30ccb17859]
[frame 10]: /opt/tritonserver/bin/../lib/libtritonserver.so(+0x109ec0) [0x7f30cc914ec0]
[frame 11]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7f30cc360de4]
[frame 12]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f30cc7de609]
[frame 13]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f30cc04e293]
from dali_backend.
Related Issues (20)
- Using custom DALI plugins with Triton HOT 3
- ModuleNotFoundError: No module named 'numpy' HOT 10
- Failed to open serialized model file: models/dali/1/model.dali HOT 2
- Memory increase when decoding exception occurs HOT 2
- dali backend device parameter setting question HOT 1
- layout parameter to external_source causes assert error HOT 2
- DALI backend not releasing device memory HOT 8
- How to provide mean & stddev to dali.fn.normalize HOT 3
- Error when executing Mixed operator decoders__Image when sending image binary to dali in triton HOT 9
- how to use the numpy data in the DALI HOT 3
- Batching does not improve performance with dali HOT 10
- Can dali backend support default values or optional input? HOT 2
- Unexpected large memory needed for gpu resize HOT 4
- Error in thread 31: nvJPEG error (5): The user-provided allocator functions, for either memory allocation or for releasing the memory, returned a non-zero code. HOT 6
- Cannot compile dali_backend with older version of triton HOT 2
- how to provide batch input data for dali pipeline whicn input shapes [-1] HOT 1
- if I want to crop from different start point, how can I build pipe to do this? HOT 2
- Test issue
- Connecting InputOperator with no explicit inputs to Triton HOT 12
- Could not serialize dali.fn.python_function HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dali_backend.