This repository showcase the usage of the Prometheus by using GKE with :

• the HipsterShop

The following tools need to be install on your machine :

• jq
• kubectl
• git
• gcloud ( if you are using GKE)
• Helm

PROJECT_ID="<your-project-id>"
gcloud services enable container.googleapis.com --project ${PROJECT_ID}
gcloud services enable monitoring.googleapis.com \
cloudtrace.googleapis.com \
clouddebugger.googleapis.com \
cloudprofiler.googleapis.com \
--project ${PROJECT_ID}

ZONE=us-central1-b
gcloud containr clusters create isitobservable \
--project=${PROJECT_ID} --zone=${ZONE} \
--machine-type=e2-standard-2 --num-nodes=4

git clone https://github.com/isItObservable/Episode-5---PromQL
cd Episode-5---PromQL

cd hipstershop
./setup.sh

helm install prometheus stable/prometheus-operator

kubectl get svc
kubectl edit svc prometheus-grafana

change to type NodePort

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: prometheus
    meta.helm.sh/release-namespace: default
  labels:
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 7.0.3
    helm.sh/chart: grafana-5.3.0
  name: prometheus-grafana
  namespace: default
  resourceVersion: "89873265"
  selfLink: /api/v1/namespaces/default/services/prometheus-grafana
spec:
  clusterIP: IPADRESSS
  externalTrafficPolicy: Cluster
  ports:
  - name: service
    nodePort: 30806
    port: 80
    protocol: TCP
    targetPort: 3000
  selector:
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/name: grafana
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

Deploy the ingress by making sure to replace the service name of your grafan

cd ..\grafana
kubectl apply -f ingress.yaml

Get the login user and password of Grafana

• For the password :

kubectl get secret --namespace default prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode

• For the login user:

kubectl get secret --namespace default prometheus-grafana -o jsonpath="{.data.admin-user}" | base64 --decode

Get the ip adress of your Grafana

kubectl get ingress grafana-ingress -ojson | jq  '.status.loadBalancer.ingress[].ip'

The tutorial will be using metrics expose by the Kube state metric and node exporter.

Let's focus on the metric kube_pod_status_phase Open Grafana, click on Explore Let's just look on the label exposed on this metric

The label exposed on this metric are :

• endpoint
• instance
• job
• namespace
• phase
• pod
• service

Let's try to to count the number of Failed pods per services in the hipster-shop namespace over the last 5minutes

let's start by using label filtering

kube_pod_status_phase{phase="Failed", namespace="hipster-shop"}

Now let's use the function to count the number of running pods per namespaces

count(kube_pod_status_pahse{phase="Running"} ) by(namespace)

Now that we have counter, we can use rate to have the number of pods in failure /s over the last 10m

rate(sum(kube_pod_status_pahse{phase="Failed"} ) by(namespace) [10m])

Let's pick a Counter to utilize the rate function

let's take the example of the metric : node_disk_io_time_seconds_total

rate(node_disk_io_time_seconds_total [5m]) 

let's use a Gauge metric to use histogram_quantile The metric that we could use is : etcd_lease_object_counts_bucket

histogram_quantile(0.95, sum(rate(etcd_lease_object_counts_bucket[5m])) by (le))

count the number of objects created in our cluster we can use for that metrics expose in etcd

sum(rate(etcd_object_counts [5m] ) ) by (resource)

If you are not confortable of building your PromQL, there is a solution that will help you designing your Promql : PromLens