Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method about kserve HOT 12 OPEN

Hello

The same issue applies for me as @serdarildercaglar mentioned above. It will be very helpful for us to fix this please..

from kserve.

@sivanantha321 Could you please help me with this problem? Please let me know if you can address it.

Will look into it

from kserve.

hi guys,
I'm having the same problems as @serdarildercaglar has. I urgently need a solution. Thx

from kserve.

@sivanantha321 Could you please help me with this problem? Please let me know if you can address it.

from kserve.

Hi @bunyaminkeles. Could you fix the issue? Is there any improvement on your side?
I am about to launch my project to production but I couldn't fix this issue. If I cannot employ workers, I cannot use multi-process.

from kserve.

from kserve.

For now, you can try with ray serve for take advantage of multiple workers. https://kserve.github.io/website/latest/modelserving/v1beta1/custom/custom_model/#parallel-model-inference. I recommend using kserve 0.11 as 0.12 release seems broken with ray serve

from kserve.

I used ray serve and it was successful in multiprocessing. However, when I use ray serve, as the number of replicas increases, resource consumption increases, and whether the server is busy or not, these replicas are constantly running, which negatively affects the cost. I will manage with ray serve until the Workers' problem is solved.
Is there any work planned to solve the problem depending on the increase in the number of Workers?

Thank you very much. Sincerely best wishes.

from kserve.

@serdarildercaglar are you using multiprocessing mainly to increase the gpu utilization ? Just curious about the motivation

from kserve.

Thanks for response @yuzisun.
Yes,
I need to increase the number of workers to process requests to the model simultaneously. When I increase the number of workers using Fastapi and send requests at the same time or close to the same time, the gpu processes the requests simultaneously. If the number of workers is 1, it processes the requests one by one and returns a response. Therefore, the response time becomes very long.
Since we are using kubernetes and kserve in our project, it is vital that I set the number of workers to 2 or more.

from kserve.

Thanks for response @yuzisun. Yes, I need to increase the number of workers to process requests to the model simultaneously. When I increase the number of workers using Fastapi and send requests at the same time or close to the same time, the gpu processes the requests simultaneously. If the number of workers is 1, it processes the requests one by one and returns a response. Therefore, the response time becomes very long. Since we are using kubernetes and kserve in our project, it is vital that I set the number of workers to 2 or more.

Why not setting the replica to 2 or more as that’s how kubernetes scales? The worker count is mainly for saving expensive compute resource like gpu to scale up within the container, but at some point it is bounded by the resource limit of the container and you can‘t scale as much as it can with kubernetes replicas.

from kserve.

• Increasing the number of “workers” can process incoming requests simultaneously based on the number of “workers” without creating more computational resources. Suppose the model generates prediction using CPU. For example, let the number of “workers” be 3. When a request arrives, the model runs normally. When 3 requests arrive at the same time, the same model can process requests with 3 workers using the available resources. If I create 3 replicas, 3 replicas will always consume resources when a single request comes or when 3 requests come.
  The best solution for me with my limited resources is to use workers for CPU or GPU.

• Another issue: When using CPU, when I make the number of workers more than 2, the response does not return and waits until timeout. But when I deploy a model in onnx format, workers for CPU works fine. For a normal transformers model, workers does not work.

I may not have been able to explain it fully because of the language barrier. Thank you for trying to help.

from kserve.