this document show steps to filter stackdriver logs, stream them to cloud pub/sub and trigger a cloud function to further process them.
For the sake of the demo we will be creating a kubernetes cluster which we will be using to filter its logs and ship them to Cloud PubSub.
-
create a k8s cluster called log-demo
zone=us-west1 gcloud container clusters create log-demo-cluster --zone $zone
-
create a pubsub topic where we will send our cluster logs
gcloud pubsub topics create log-demo-cluster-errors
-
create a function to process the pubsub messages(logs)
Here, we are simply printing back the filtered log but it's up to you to process the log as you wish and perhaps store it later in a storage service like cloud BigTable, BigQuery, GCS. you can write your function in any of the following: python, nodejs, GO.
cat<< EOF > main.py
def hello_pubsub(data, context):
"""Background Cloud Function to be triggered by Pub/Sub.
Args:
data (dict): The dictionary with data specific to this type of event.
context (google.cloud.functions.Context): The Cloud Functions event
metadata.
"""
import base64
if 'data' in data:
payload = base64.b64decode(data['data']).decode('utf-8')
else:
payload = 'EMPTY'
print('I am a log entry, do somehting with me: {}!'.format(payload))
EOF
-
deploy the function while specifying our previously created topic as the trigger
gcloud functions deploy hello_pubsub --source `pwd` --runtime python37 --trigger-topic log-demo-cluster-errors
-
create a pubsub log export sink while filtering logs specific to our gke cluster which has severity
warning
and above:NOTE
: changePROJECT_ID
to your project idgcloud beta logging sinks create pub-sub-sink \ pubsub.googleapis.com/projects/[PROJECT_ID]/topics/log-demo-cluster-errors \ --log-filter='resource.type="container" AND resource.labels.cluster_name="log-demo-cluster" AND severity>=WARNING'
-
get the pubsub sink writer identity service account
sink_sa=$(gcloud beta logging sinks describe pub-sub-sink --format="(writerIdentity)" | awk '{print $2}')
-
Grant role of cloud pub sub publisher to the service account gcloud projects add-iam-policy-binding rad-tests --member=$sink_sa --role=roles/pubsub.publisher
-
Create some errors withing the cluster (deploy many replicas)
kubectl run nginx --image=nginx --replicas=20
-
Get the logs from our cloud function execution:
gcloud functions logs read hello_pubsub