For example, the pipeline is as follows, the input is HWC, <div class="snippet-cli

Hello <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-ur

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

how to generate inputs with different shape as batch to DALI_backend?,about triton-inference-server/dali_backend

Comments (3)

szalpal commented on July 4, 2024

Because of the dynamic batching, Triton assumes, that all samples within a batch have the same shape. Therefore, if Triton receives two infer requests: [1, 640, 480, 3] and [2, 640, 480, 3], it will batch them together into [3, 640, 480, 3].

To work this around, I suggest one of the following approaches:

Just send the different sized images in different batches. Ofc, this will only work if you expect to have similar sizes of images.
The easy situation is when you are using fn.decoders.image inside DALI's pipeline, i.e. you are sending encoded images to DALI Backend. Then all you need to do (in the triton client) is to pad all encoded buffers with zeros (every decoder will just discard them), something like:

def array_from_list(arrays):
    """
    Convert list of ndarrays to single ndarray with ndims+=1
    :param arrays List of ndarrays
    """
    lengths = list(map(lambda x, arr=arrays: arr[x].shape[0], [x for x in range(len(arrays))]))
    max_len = max(lengths)
    arrays = list(map(lambda arr, ml=max_len: np.pad(arr, ((0, ml - arr.shape[0]))), arrays))
    for arr in arrays:  # Sanity check
        assert arr.shape == arrays[0].shape, "Arrays must have the same shape"
    return np.stack(arrays)

If you need to input decoded images, things will get hacky:
- First of all, in the Client you need to read the actual sizes of images and then pad them (with any value)
- Create an extra input (fn.external_source) in DALI pipeline, which will get the shape of every image before padding
- Use fn.crop operator to crop the images within DALI pipeline

# Client code
def batch_from_list(images):
   """
   :param images: List of images
   :return: Tuple: (Padded images, image shapes before padding)
   """
   ...
   return batch, shapes

# DALI pipeline code
@dali.pipeline_def(batch_size=32, num_threads=8, device_id=0)
def pipe():
   images = fn.external_source(name="IMAGE_ARRAY")
   shapes = fn.external_source(name="SHAPES", device="cpu")  # Note the device="cpu", this is necessary
   images = fn.crop(images, crop=shapes, ...)
   ...
   return images

Should you have any more questions, don't hesitate to ask.

from dali_backend.

Edwardmark commented on July 4, 2024

@szalpal ,hi, I try to inference with encoded images, and save the decoded ones to output, but the result is not true, when different shape image is feed to pipeline.
The config.pbtxt is as follows:

# https://github.com/triton-inference-server/dali_backend/issues/18
name: "dali_det_pre"
backend: "dali"
max_batch_size: 32
input [
  {
    name: "IMAGE_RAW"
    data_type: TYPE_UINT8
    dims: [ -1 ]
  }
]
output [
  {
    name: "DALI_OUTPUT_0" # original image
    data_type: TYPE_UINT8
    dims: [-1, -1, 3]
  },
  {
    name: "DALI_OUTPUT_1" # resized image
    data_type: TYPE_FP32
    dims: [3, 640, 640]
  },
  {
    name: "DALI_OUTPUT_2" # ratio
    data_type: TYPE_FP32
    dims: [2]
  }
]
dynamic_batching {
  preferred_batch_size: [ 4, 8, 16, 32 ]
  max_queue_delay_microseconds: 100
}

The dali pipeline is as follows:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import nvidia.dali as dali
import nvidia.dali.fn as fn
import nvidia.dali.types as types
import argparse
import numpy as np
import os

pipe = dali.pipeline.Pipeline(batch_size=32, num_threads=8)
with pipe:
    expect_output_size = (640., 640.)
    images_raw = fn.external_source(device='cpu', name="IMAGE_RAW")
    images = fn.image_decoder(images_raw, device="mixed", output_type=types.RGB)
    raw_shapes = fn.shapes(images, dtype=types.INT32)
    norm_images = fn.resize(
        images,
        mode='not_larger',
        size=expect_output_size,
    )
    resized_shapes = fn.shapes(norm_images, dtype=types.INT32)
    ratio = fn.slice(resized_shapes / raw_shapes, 0, 2, axes=[0])
    norm_images = fn.crop_mirror_normalize(norm_images, mean=[0.], std=[255.], output_layout='CHW')
    norm_images = fn.pad(norm_images, axis_names="HW", align=expect_output_size)
    pipe.set_outputs(images, norm_images, ratio)
pipe.serialize(filename="1/model.dali")

The client is as follows:

    image_data, im_paths = load_images(FLAGS.img_dir if FLAGS.img_dir is not None else FLAGS.img,
                             max_images=FLAGS.batch_size * FLAGS.n_iter)

    image_data = array_from_list(image_data)
    print("Images loaded")


    inputs = generate_inputs(FLAGS.input_name, image_data.shape, "UINT8")
    outputs = generate_outputs(FLAGS.output_names)

    # Initialize the data
    inputs[0].set_data_from_numpy(image_data)
    # Test with outputs
    results = triton_client.infer(model_name=model_name, inputs=inputs, outputs=outputs)

    #import pdb;pdb.set_trace()
    org_im = results.as_numpy(FLAGS.output_names[0])

but the result is strage,
with shape as (5, 2521, 3361, 3), so how to recover the real orginal image?

from dali_backend.

szalpal commented on July 4, 2024

@Edwardmark

I don't fully understand, what would you like to achieve. Could you describe, what would you like for the DALI pipeline to do? This way we can verify, if DALI pipeline if written correctly

from dali_backend.

how to generate inputs with different shape as batch to DALI_backend? about dali_backend HOT 3 CLOSED

Comments (3)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent