Questions about serving PyTorch LLM in Python backend with token streaming using "Decoupled Mode" about server HOT 4 CLOSED

Thanks for your questions!

1. Correct.

2. The max batch size for each model instance will be 8.

3. It is for demonstration purposes only. It is mainly meant to show that you can send responses after returning from execute function. It could be a workaround to the memory usage issue you brought up in (1) (i.e. you can have a single model instance with multiple threads inside to avoid extra memory usage).

4. That's correct.

5. The warning is mainly about not accepting too many requests and spawning a lot of threads (i.e. having a limit on the number of requests that you want to accept). I think it is not required to set the daemon flag. You don't need to join the threads before returning None. As shown in the example, the threads continue to send responses after returning None from the execute function.

6. I think it is hard to have a general recommendation for your scenario since it is dependent on a lot of parameters. It would be good to experiment with these different options to see which one performs best for your model. Would love to learn more which route worked best for you.

Let us know if you have any additional questions!

from server.

Hi @Tabrizian , thank you very much for your detailed reply and insights! I have a few follow-up questions:

1. Regarding quetsion 5, if I don't join the threads before returning None in execute(), then in order to prevent too many requests from being accepted and to prevent too many threads from being spawned, I would then have to keep track of the number of requests that are still on-going/the number of threads that are in still in flight, is that correct? Is the self.inflight_thread_count member variable in the square_model.py file (https://github.com/triton-inference-server/python_backend/blob/main/examples/decoupled/square_model.py) a way to keep track of this?
2. In general, regarding the amount of memory a model needs during inference, the amount of memory inference takes is not just the size of the weights but also the size of the intermediate layer tensor output/activation right? In other words, do I have to leave some additional space outside of the raw weights memory size for the intermediate layer tensor output/activation during inference?

Thanks a lot for your time and help again!

from server.

1. correct. You can also probably use semaphores so that you block until you have acquired enough resources: https://docs.python.org/3/library/threading.html#semaphore-objects

2. Yes, it would be good to run your model thorough model analyzer to better understand the memory requirements.

from server.

1. correct. You can also probably use semaphores so that you block until you have acquired enough resources: https://docs.python.org/3/library/threading.html#semaphore-objects
2. Yes, it would be good to run your model thorough model analyzer to better understand the memory requirements.

I see, thank you very much for your answers and insights!

from server.