mozilla / docker-etl Goto Github PK

This ETL script will periodically fetch bugs from Web Compatibility > Knowledge Base component on bugzilla, as well as their core bugs dependencies and breakage reports and then store them into BQ.

I've noticed some occasional unclassified reports that were missed by the broken_site_report_ml job.

Currently we select reports for classification based on a condition where reported_at is after the last successful classification run:

Thinking this could have happened because at the time of running the job, the missed reports were not in BQ table, and got inserted with a delay. So to fix this, we could look for reports don't have a label and with reported_at before the last successful classification run.

DAP defines a Collector role which polls the leader for available data. Since we need metrics in BQ for further processing we need to implement this role.

This ETL script will periodically send a batch of reports to bugbug http service for classification and store results in BQ.

https://app.circleci.com/pipelines/github/mozilla/docker-etl/92/workflows/6717bfcd-03d2-4cfc-9ce5-f68e9f7906db/jobs/257

#!/bin/bash -eo pipefail
# Store service account
echo $MAU_DASHBOARDS_KEY > ${HOME}/gcloud-service-key.json

# Initialize gcloud CLI
gcloud auth activate-service-account --key-file=${HOME}/gcloud-service-key.json
gcloud --quiet config set project $MAU_DASHBOARDS_PROJECT
gcloud --quiet config set compute/zone $GOOGLE_COMPUTE_ZONE

ERROR: (gcloud.auth.activate-service-account) There was a problem refreshing your current auth tokens: invalid_grant: Invalid grant: account not found