jeremyjordan / ml-monitoring Goto Github PK

Hello @jeremyjordan,

I've been following your fastapi ml-monitoring repository as a template for my own project and it's been super helpful! Thanks so much for setting this up. Unfortunately, I'm experiencing a lot of trouble getting prometheus to scrape my Counter metric and latency as well. Interestingly, when I run your wine-quality application and add a Counter metric though, it seems to be working fine, but mine which pretty much follows your same approach (only difference being that I set up my application using application factory design pattern) doesn't seem to be working. It seems like histogram and summary are going through though.

Do you have any insight as to what the issue could be? Would really appreciate your guidance as I've been trying to figure this out for 3 days.

Here is my monitoring.py file: https://github.com/rileyhun/fastapi-ml-example/blob/main/app/core/monitoring.py

Reproducible example:

git clone https://github.com/rileyhun/fastapi-ml-example.git

docker build -t ${IMAGE_NAME}:${IMAGE_TAG} -f Dockerfile .
docker tag ${IMAGE_NAME}:${IMAGE_TAG} rhun/${IMAGE_NAME}:${IMAGE_TAG}
docker push rhun/${IMAGE_NAME}:${IMAGE_TAG}

minikube start --driver=docker --memory 4g --nodes 2
kubectl create namespace monitoring
helm install prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring

kubectl apply -f deployment/wine-model-local.yaml
kubectl port-forward svc/wine-model-service 8080:80

python api_call.py

Hi Jeremy,

I'm following your template for a POC, and it's been very helpful. I'm creating a REST API for an NLP model (Multinomial Naive Bayes) and I'm not sure how to monitor this particular model when the predictions are classes instead of float values like the wine quality prediction model. How would the prometheus instrumentation be used to capture metrics for classification models?

Thanks,

Riley

Hello again @jeremyjordan,

We are trying to decrease the latency of our BERT model prediction service that is deployed using FastAPI. The predictions are called through the /predict endpoint. We looked into the tracing and found one of the bottlenecks is the prometheus-fastapi-instrumentator. About 1% of the requests do timeout because they exceed 10s.

We also discovered that some metrics are not getting reported on 4 requests/second. Some requests took 30-50 seconds, with the starlette/fastapi taking long times. So it seems that under high usage, the /metrics endpoint doesn't get enough resources, and hence all /metrics requests wait for some time and fail eventually. So having separate container for metrics could help. Or if possible to have metrics delayed/paused under high load. Any insight/guidance would be much appreciated.