Code Monkey home page Code Monkey logo

Comments (4)

mzient avatar mzient commented on June 25, 2024 2

@mustafakasap
If you really want text, you can output it as a byte array. Hopefully the next model you have in your ensemble will be able to consume it - or perhaps it could just deal with a boolean?
BTW - what about 0? It's neither positive nor negative.
Here's an example that would return byte arrays with text - I think it's as close as it gets to strings:

@dali.pipeline_def(batch_size=8, num_threads=min(mp.cpu_count(), 4), device_id=0)
def pipe():
    input0 = dali.fn.external_source(device="cpu", name="DALI_INPUT_0")
    sum = dali.fn.reductions.sum(input0)
    
    # Conver the strings to numpy arrays of uint8 and wrap the arrays into constant nodes
    pos = dali.types.Constant(np.array(list(bytes("Positive", "utf8"))))
    neg = dali.types.Constant(np.array(list(bytes("Negative", "utf8"))))
    
    is_positive = sum >= 0
    is_negative = 1 - is_positive
    return is_positive * pos + is_negative * neg```

from dali_backend.

JanuszL avatar JanuszL commented on June 25, 2024 1

Hi @mustafakasap,

In the 2ns and 3rd cases the input0 is pruned as it is not connected to any output (one of the optimization DALI does building the processing graph). You may need to return it as it is just to make it works.
From the examples you provided I'm not fully sure if DALI is the right tool for the processing you want. One of the possible solutions is to:

from dali_backend.

mzient avatar mzient commented on June 25, 2024

Hello, @mustafakasap
Unfortunately, what you wrote cannot be achieved for multiple reasons:

  1. You have an if statement which looks like it's trying to act on a result of an operator - that's not how DALI operates. The variable sum is not a value, but rather a DataNode object. The function pipe only defines what the pipeline and does not take part in the execution - the only thing that's preserved until run-time are the connections between operators and DataNodes.
  2. str is not a valid return type of DALI operators - they can only return tensors with the following element types: int8/16/32/64, uint8/16/32/64, float16, float32, float64.

from dali_backend.

mustafakasap avatar mustafakasap commented on June 25, 2024

Thank you Michal for the responses. Let me provide more context about what I want to achieve (rather than above simplified case):

I will be using above Dali model connected to the output of yoloV3 (or any other ml model) in an ensemble model. so whatever arrives into my dali model is a 1D array of structs with bounding box coordinates, confidence etc. I will process this 1D array, iterate over each struct, construct a python string (which will be Json output of bounding boxes etc) and return this encoded JSon string as 1D uint array...

Here: https://github.com/triton-inference-server/server/blob/main/docs/examples/model_repository/simple_identity/config.pbtxt seems Triton has string datatype (which is actually uint8 array with -1 dimension)

With below cases, I tried to simulate the above scenario but seems this is not possible in Dali (and sure understand that Dali's design purpose is complete different).

Now I am thinking of writing my own custom dali plugin (like here: https://github.com/triton-inference-server/dali_backend/tree/main/docs/examples/dali_plugin) which will convert i.e. the 1D array (yolov3 output) into 1D array (json inference result string). and use this plugin function in above dali model to solve my issue. Again, would this be the right approach? or should I always handle/parse the raw output from yolov3 inside the Triton Inference client app?

CASE1 (Works fine but seems I cant iterate over the struct array of input0):
Pipeline:

          @dali.pipeline_def(batch_size=1, num_threads=min(mp.cpu_count(), 4), device_id=0)
          def pipe():
              input0 = dali.fn.external_source(device="cpu", name="DALI_SIMPLE_1_INPUT_0")
          
              output0 = dali.types.Constant(np.array(list(bytes("Hello World!", "utf8"))))
              
              identity = dali.fn.reductions.sum(input0) * 0 + 1
          
              return identity * output0

Client Output:
(running inference client in Verbose mode, so below output is directly from inference client API)

        infer, metadata ()
        model_name: "dali_simple_01"
        inputs {
          name: "DALI_SIMPLE_1_INPUT_0"
          datatype: "UINT8"
          shape: 1
          shape: 16
        }
        outputs {
          name: "DALI_SIMPLE_1_OUTPUT_0"
        }
        raw_input_contents: "\000\001\002\003\004\005\006\007\010\t\n\013\014\r\016\017"
        
        model_name: "dali_simple_01"
        model_version: "1"
        outputs {
          name: "DALI_SIMPLE_1_OUTPUT_0"
          datatype: "INT64"
          shape: 1
          shape: 12
        }
        raw_output_contents: "H\000\000\000\000\000\000\000e\000\000\000\000\000\000\000l\000\000\000\000\000\000\000l\000\000\000\000\000\000\000o\000\000\000\000\000\000\000 \000\000\000\000\000\000\000W\000\000\000\000\000\000\000o\000\000\000\000\000\000\000r\000\000\000\000\000\000\000l\000\000\000\000\000\000\000d\000\000\000\000\000\000\000!\000\000\000\000\000\000\000"

Here if I decode the above raw_output_contents into string (from asnumpy...), I get : Hello World!

CASE2 (Fails, dont know exactly why but instead of multiplying the result byte array with "is_positive" constant above, I just tried multiplying with integer 1 (ok it is not tensor) ):
Pipeline:

        @dali.pipeline_def(batch_size=1, num_threads=min(mp.cpu_count(), 4), device_id=0)
        def pipe():
            input0 = dali.fn.external_source(device="cpu", name="DALI_SIMPLE_1_INPUT_0")
        
            output0 = dali.types.Constant(np.array(list(bytes("Hello World!", "utf8"))))
        
            return 1 * output0

Client Output:

          infer, metadata ()
          model_name: "dali_simple_01"
          inputs {
            name: "DALI_SIMPLE_1_INPUT_0"
            datatype: "UINT8"
            shape: 1
            shape: 16
          }
          outputs {
            name: "DALI_SIMPLE_1_OUTPUT_0"
          }
          raw_input_contents: "\000\001\002\003\004\005\006\007\010\t\n\013\014\r\016\017"
          
          Traceback (most recent call last):
            File "/myworkspace/projects/tritonserver/samples/dali/simple/client.py", line 25, in <module>
              results = triton_client.infer(model_name='dali_simple_01',
            File "/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py", line 1146, in infer
              raise_error_grpc(rpc_error)
            File "/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py", line 62, in raise_error_grpc
              raise get_error_grpc(rpc_error) from None
          tritonclient.utils.InferenceServerException: [StatusCode.UNKNOWN] DALI error: [/opt/dali/dali/pipeline/graph/graph_descr.cc:451] Operator node with name DALI_SIMPLE_1_INPUT_0 not found.
          Stacktrace (14 entries):
          [frame 0]: /opt/tritonserver/backends/dali/dali/libdali.so(+0x7edbe) [0x7f2e868b3dbe]
          [frame 1]: /opt/tritonserver/backends/dali/dali/libdali.so(dali::OpGraph::Node(std::string const&)+0x89) [0x7f2e86938929]
          [frame 2]: /opt/tritonserver/backends/dali/dali/libdali.so(daliGetOperatorBackend+0x2f) [0x7f2e869ea52f]
          [frame 3]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x50cc4) [0x7f2e874a1cc4]
          [frame 4]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x4e6eb) [0x7f2e8749f6eb]
          [frame 5]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x4ebec) [0x7f2e8749fbec]
          [frame 6]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x2d832) [0x7f2e8747e832]
          [frame 7]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x2db96) [0x7f2e8747eb96]
          [frame 8]: /opt/tritonserver/backends/dali/libtriton_dali.so(TRITONBACKEND_ModelInstanceExecute+0x150) [0x7f2e8746d210]
          [frame 9]: /opt/tritonserver/bin/../lib/libtritonserver.so(+0x30c859) [0x7f2fb83f0859]
          [frame 10]: /opt/tritonserver/bin/../lib/libtritonserver.so(+0x109ec0) [0x7f2fb81edec0]
          [frame 11]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7f2fb7c39de4]
          [frame 12]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f2fb80b7609]
          [frame 13]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f2fb7927293]

CASE3 (Fails, instead of above non tensor identity integer value, I tried to construct my own constant tensor like the "is_positive" one but still this fails...):
Pipeline:

          @dali.pipeline_def(batch_size=1, num_threads=min(mp.cpu_count(), 4), device_id=0)
          def pipe():
              input0 = dali.fn.external_source(device="cpu", name="DALI_SIMPLE_1_INPUT_0")
          
              output0 = dali.types.Constant(np.array(list(bytes("Hello World!", "utf8"))))
         
              identity = dali.types.Constant(1)

              return identity * output0

Client Output:

          infer, metadata ()
          model_name: "dali_simple_01"
          inputs {
            name: "DALI_SIMPLE_1_INPUT_0"
            datatype: "UINT8"
            shape: 1
            shape: 16
          }
          outputs {
            name: "DALI_SIMPLE_1_OUTPUT_0"
          }
          raw_input_contents: "\000\001\002\003\004\005\006\007\010\t\n\013\014\r\016\017"
          
          Traceback (most recent call last):
            File "/myworkspace/projects/tritonserver/samples/dali/simple/client.py", line 25, in <module>
              results = triton_client.infer(model_name='dali_simple_01',
            File "/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py", line 1146, in infer
              raise_error_grpc(rpc_error)
            File "/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py", line 62, in raise_error_grpc
              raise get_error_grpc(rpc_error) from None
          tritonclient.utils.InferenceServerException: [StatusCode.UNKNOWN] DALI error: [/opt/dali/dali/pipeline/graph/graph_descr.cc:451] Operator node with name DALI_SIMPLE_1_INPUT_0 not found.
          Stacktrace (14 entries):
          [frame 0]: /opt/tritonserver/backends/dali/dali/libdali.so(+0x7edbe) [0x7f2f9afdadbe]
          [frame 1]: /opt/tritonserver/backends/dali/dali/libdali.so(dali::OpGraph::Node(std::string const&)+0x89) [0x7f2f9b05f929]
          [frame 2]: /opt/tritonserver/backends/dali/dali/libdali.so(daliGetOperatorBackend+0x2f) [0x7f2f9b11152f]
          [frame 3]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x50cc4) [0x7f2f9bbc8cc4]
          [frame 4]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x4e6eb) [0x7f2f9bbc66eb]
          [frame 5]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x4ebec) [0x7f2f9bbc6bec]
          [frame 6]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x2d832) [0x7f2f9bba5832]
          [frame 7]: /opt/tritonserver/backends/dali/libtriton_dali.so(+0x2db96) [0x7f2f9bba5b96]
          [frame 8]: /opt/tritonserver/backends/dali/libtriton_dali.so(TRITONBACKEND_ModelInstanceExecute+0x150) [0x7f2f9bb94210]
          [frame 9]: /opt/tritonserver/bin/../lib/libtritonserver.so(+0x30c859) [0x7f30ccb17859]
          [frame 10]: /opt/tritonserver/bin/../lib/libtritonserver.so(+0x109ec0) [0x7f30cc914ec0]
          [frame 11]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7f30cc360de4]
          [frame 12]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f30cc7de609]
          [frame 13]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f30cc04e293]

from dali_backend.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.