Deploying HuggingFace model/pipeline using uvicorn-gunicorn-fastapi-docker on Google Cloud Run about uvicorn-gunicorn-fastapi-docker HOT 2 OPEN

If you use def for the fastapi function, it creates a new thread (from a threadpool) for each incoming request. The model has a single copy in GPU.

If you use async def with N workers, it creates a total of N forks. Each request is handled by one of these N forks. For each of the N workers, there is a copy of the model in the GPU. Workers don't share memory or other resources.

To decide the number of workers: N = number of threads + 1.
You also need more than enough GPU to fit N copies of the model.

So if you are GPU limited, that's your criteria to decide the number of workers.

What I wrote above is based on what I observed in a few tests. It might well be incorrect.

from uvicorn-gunicorn-fastapi-docker.

I would also recommend using Gunicorn instead of Uvicorn to run the app

from uvicorn-gunicorn-fastapi-docker.