Hi all,
Are there any plans to add bitbucket support to this project? I may be able to help with this too, if it's not too much work.

Thanks

There are several conditionals based on VCS--if <github>: ... else if <gitlab>: ...

This is already inelegant and will quickly become unmaintainable as we add VCSes.

Running the python3 schedule.py --query_file=changes.sql --table=changes --access_token=${token} part of setup.sh initially fails with the following exception.

Traceback (most recent call last):
  File "/Users/segan3/Library/Python/3.8/lib/python/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/Users/segan3/Library/Python/3.8/lib/python/site-packages/grpc/_channel.py", line 923, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/segan3/Library/Python/3.8/lib/python/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Failed to find a valid credential. The request to create a transfer config is supposed to contain an authorization code."
	debug_error_string = "{"created":"@1607609612.827897000","description":"Error received from peer ipv4:216.58.207.234:443","file":"src/core/lib/surface/call.cc","file_line":1063,"grpc_message":"Failed to find a valid credential. The request to create a transfer config is supposed to contain an authorization code.","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "schedule.py", line 105, in <module>
    app.run(create_or_update_scheduled_query)
  File "/Users/segan3/Library/Python/3.8/lib/python/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/Users/segan3/Library/Python/3.8/lib/python/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "schedule.py", line 100, in create_or_update_scheduled_query
    response = client.create_transfer_config(parent, transfer_config)
  File "/Users/segan3/Library/Python/3.8/lib/python/site-packages/google/cloud/bigquery_datatransfer_v1/gapic/data_transfer_service_client.py", line 811, in create_transfer_config
    return self._inner_api_calls["create_transfer_config"](
  File "/Users/segan3/Library/Python/3.8/lib/python/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
    return wrapped_func(*args, **kwargs)
  File "/Users/segan3/Library/Python/3.8/lib/python/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func
    return retry_target(
  File "/Users/segan3/Library/Python/3.8/lib/python/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/Users/segan3/Library/Python/3.8/lib/python/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/Users/segan3/Library/Python/3.8/lib/python/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Failed to find a valid credential. The request to create a transfer config is supposed to contain an authorization code.

This was from running it with an auth token from my personal account, which had the Owner role in the project.

There are probably better solutions, but I managed to get around it by first creating a similar scheduled query manually to there get option to grant access to the needed scope.

If you rerun setup.sh on an existing project where an event-handler secret is already existing, it will try to generate a new one instead of honoring the existing one.
Because of this mocking data will fail and you'll get an exception from the event-handler.

Exception: Unverified Signature
at index (/app/event_handler.py:50)
at dispatch_request (/usr/local/lib/python3.7/site-packages/flask/app.py:1935)
at full_dispatch_request (/usr/local/lib/python3.7/site-packages/flask/app.py:1949)
at reraise (/usr/local/lib/python3.7/site-packages/flask/_compat.py:39)
at handle_user_exception (/usr/local/lib/python3.7/site-packages/flask/app.py:1820)
at full_dispatch_request (/usr/local/lib/python3.7/site-packages/flask/app.py:1951)
at wsgi_app (/usr/local/lib/python3.7/site-packages/flask/app.py:2446)

Hi 4Keys!
My team needs some help setting this up for private repos. What appears to be simple has taken over a week and no success in sight yet.
Is there anyway one of you can help us out with 30mins of your time?

Ketan

In multiple parts of the project, an env.sh file is referenced.
I think an example of the file should be provided, to simplify the full setup and later usages.

Deployment definition change in recent PR (#34) was not reflected in the data generator, causing new dashboard setups to fail: #37

• Write script
• Update documentation

reference:

• #50
• #46

Currently if you want to re-run the scripts that populate the derived tables, you have to find the query history and execute it again manually. We should have an easy python script or set of gcloud commands to run the scripts and update the tables.

In METRICS.md it says "the median amount of time between the deployment which caused the failure" but in the slq query it uses "time_created" which is the time the incident was created. Please see below screenshot

Looks like it just got easier -- any reason we shouldn't proceed? @dinagraves @donmccasland

As seen in #47, using Python to set up the scheduled queries is fragile. I had a better time calling the gcloud CLI directly from bash. Let's replace /queries/schedule.py with a bash script.

It's important to preserve the ability to re-run the schedule setter-upper any time (independent of setup.sh), so this script should be an independent module that setup.sh will invoke.

Enumerate all the manual prompts in the setup so the user knows what to expect.

Per the findings in the 2018-2019 SODR Report, there are now four buckets of performers. Accordingly, the Quick Check tool offers four buckets when reporting results.

I suggest we use the same rubric, with four buckets for color-coding the dashboard.

Hey @dinagraves,

Thanks for all the work with this project, we are in the process of implementing it in our organization.

I'm getting an error when running setup.sh (following you're youtube video and was hoping you could point me in the right direction. My environment are as follows

Operating System

Running on Ubuntu 20.04.1 LTS

gcloud

I want to make use of an already existing project called "bcc-fourkeys" in our gcloud instance
I confirmed that billing is enabled and that I have owner access

executing setup.sh

When I attempt to run setup.sh I get the following error

Please let me know if you need an more information.
Greatly appreciate the help.

Kind Regards,
Philip

When an exception is raised in the event handler due to an unauthorized caller (thrown here or here), the application currently returns HTTP status 500.

It would be more correct to return a 403 (Forbidden).

The setup/*_schema.json files--other than events_raw_schema.json--aren't actually used during setup.

Let's remove these files, but add the table definitions to documentation for reference.

djsanders@

Error running setup.sh

• Selected "No" when asked to create project
• Supplied existing project ID
• Selected "Yes" when asked to create example project (helloworld)

Would you like to create a separate new project to test deployments for the four key metrics? (y/n):y
Setting up project for Helloworld...
+ gcloud projects create --folder=
ERROR: (gcloud.projects.create) Missing required argument [PROJECT_ID]: an id or a name must be provided for the new project
+ gcloud beta billing projects link --billing-account=billingAccounts/xxxxxx-xxxxxx-xxxxxx
ERROR: (gcloud.beta.billing.projects.link) argument PROJECT_ID: Must be specified.
Usage: gcloud beta billing projects link PROJECT_ID --billing-account=ACCOUNT_ID [optional flags]
  optional flags may be  --help

The requirements file for event_handler includes the lines:

pytest==5.3.0; python_version > "3.0"
pytest==4.6.6; python_version < "3.0"

...since we require python >= 3.6, and since pytest is now at version 6, I propose we update this to:

pytest~=6.0.0

Does that sound okay to you, @dinagraves

(this is what I'm using for the event_handler tests)

Also, we could get faster installs and slimmer prod builds if we isolate testing dependencies. AIUI, PIP doesn't support "dev dependencies" like e.g. npm, but I did find this: https://stackoverflow.com/questions/17803829/how-to-customize-a-requirements-txt-for-multiple-environments

...instead of hardcoding it like here:
https://github.com/GoogleCloudPlatform/fourkeys/blob/fe7d7d8114b2b1786b1312b5543276108baa2f24/event_handler/sources.py#L45

I noticed in some places where gcloud is run that flags are not after the command:

For example:

export WEBHOOK=$(gcloud run --platform managed --region ${FOURKEYS_REGION} services describe event-handler --format=yaml | grep url | head -1 | sed -e 's/ *url: //g')

Running the above, I get this error:

`ERROR: (gcloud.run) unrecognized arguments: --platform (did you mean '--format'?)

To search the help text of gcloud commands, run:
gcloud help -- SEARCH_TERMS`

Modifications that seem to work:

export WEBHOOK=$(gcloud run services describe event-handler --platform managed --region ${FOURKEYS_REGION} --format=yaml | grep url | head -1 | sed -e 's/ *url: //g')

There is sometimes a delay in the bigquery job execution causing the datastudio dashboard to load before the bigquery scripts have finished populating the tables. This results in an empty dashboard.

The setup script should wait until the bigquery jobs have finished to open the Datastudio config.

The INSTALL.md refers to a "hipster store" sample, which was subsequently changed to a hello world sample. Should be updated to reflect the latest state of the setup script.

Adding a screenshot of the dashboard that includes the sample data will help users understand visualize the end state and motivate them to go through the process of setting up the dashboard.

The link configured on the text "Apps Script" is broken, it returns a 404.

Link to directory of the file with the error
https://github.com/GoogleCloudPlatform/fourkeys/tree/main/connector

This may or may not be a good idea. Let’s explore. See: #75 (comment)

...this will then render the GCE api service unnecessary, so remove that resource.

• Add information to documentation about BQ authorization flow
• Add note on documentation about having certificates installed on python version

I'm getting...

python3 schedule.py --query_file=changes.sql --table=changes --access_token=ya29.a0AfH6SMB8FMz2I_ZuM76BZ44PtNIO37BZLyTwr7ZcWQFA-t9geupxv12zDGJRSZHzlRTIdSIY6mPrh-4n7OH8k1U_lA9hSuE7VqpHLJSOWTx-iGWUBhZ2hAbXVb_Oqz9h7mJaWG8TCrEFfAZddXyuLzEgNA2AFiuqWJP3F4e2pICRa-7p7XVKyQ
Traceback (most recent call last):
  File "/home/davidstanke/.local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/davidstanke/.local/lib/python3.6/site-packages/grpc/_channel.py", line 923, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/davidstanke/.local/lib/python3.6/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.FAILED_PRECONDITION
        details = "BigQuery Data Transfer service account is not found, this could be fixed by re-enabling BigQuery Data Transfer service API for project fourkeys-007886. Please ask the project owner to re-enable the API at https://console.cloud.google.com/apis/library/bigquerydatatransfer.googleapis.com."
        debug_error_string = "{"created":"@1611155717.256131935","description":"Error received from peer ipv4:172.217.8.10:443","file":"src/core/lib/surface/call.cc","file_line":1063,"grpc_message":"BigQuery Data Transfer service account is not found, this could be fixed by re-enabling BigQuery Data Transfer service API for project fourkeys-007886. Please ask the project owner to re-enable the API at https://console.cloud.google.com/apis/library/bigquerydatatransfer.googleapis.com.","grpc_status":9}"

Per the docs, the setup script should have prompted me to configure BigQuery interactively, but that didn't seem to happen(?).

I'm investigating.

When I tried the command to get the event handler endpoint in INSTALL.md, I received the following error:

ERROR: (gcloud.run) unrecognized arguments: --platform (did you mean '--format'?)

It looks like the syntax for gcloud run services has changed as of version 322.0.0 and --platform managed --region ${FOURKEYS_REGION} now need to come after the positional arguments.

The following syntax works:

gcloud run services describe event-handler --platform managed --region ${FOURKEYS_REGION} --format=yaml | grep url | head -1 | sed -e 's/  *url: //g'

When the -b flag is used with setup/cleanup.sh, it will delete any projects named "fourkeys-*" ... this includes projects that were created by setup.sh, but also may include other projects, like fourkeys-foo, fourkeys-bar, or (eek!) fourkeys-testing.

The cleanup script script should filter more narrowly for projects to delete.

I can't seem to figure out the Datastudio Dashboard template step here. Launching the Datastudio URL, it gives me a message "The linked connectorid was invalid".

I also tried looking for a template in Datastudio community but can't find any similar.

Cloud Run configs should be set globally. The lack of a region tag caused the service account binding to fail:

+ gcloud iam service-accounts create cloud-run-pubsub-invoker --display-name 'Cloud Run Pub/Sub Invoker'
Created service account [cloud-run-pubsub-invoker].
+ gcloud run --platform managed services add-iam-policy-binding github-worker --member=serviceAccount:[email protected] --role=roles/run.invoker
ERROR: (gcloud.run.services.add-iam-policy-binding) Error parsing [service].
The [service] resource is not properly specified.
Failed to find attribute [region]. The attribute can be set in the following ways: 
- provide the argument [--region] on the command line
- set the property [run/region]
- specify from a list of available regions in a prompt
+ gcloud run --platform managed services add-iam-policy-binding cloud-build-worker --member=serviceAccount:[email protected] --role=roles/run.invoker
ERROR: (gcloud.run.services.add-iam-policy-binding) Error parsing [service].
The [service] resource is not properly specified.
Failed to find attribute [region]. The attribute can be set in the following ways: 
- provide the argument [--region] on the command line
- set the property [run/region]

The statistics works as it should if each merge to main branch is deployed to production separately. If there are more merges to main branch before the deploy to production, either the lead time or the deployment frequency will be incorrect.

Option 1: Create deployment event for each merge to main branch. Will include all commits in lead time, but will generate higher number of deployment events.
Option 2: Create one deployment event for last merge to main branch. Will only include commits for last merge in lead time calculation, but will generate correct number of deployment events.

Example to hopefully make it more clear (using option 2):
Create branch a, make 2 commits, merge in to master. Create branch b, make 2 commits, merge to master.

Git log afterwards:

git log --decorate=no --date-order --reverse --pretty=oneline

6bd52a46ab919384b043a40750e0aed8e0e0d43b Branch a, commit 1
5a93561896c7c04758df9fe05eaaa4f7154e53f6 Branch a, commit 2
807b8acfc1007e544941944df182d10e6f9f52fd Merge pull request #3 from org-name/branch-a
964198b26178b4203f14a77c77080af70a445750 B - 1
bb13c56e5e9b214682c28647adc787ff061183e2 B - 2
21e706a11d41e467fe055dbdb5fe21a609427a20 Merge pull request #4 from org-name/branch-b

Create deployment with GitHub API for last merge:

curl -u $GITHUB_USER:$GITHUB_TOKEN \
  -X POST \
  -H "Accept: application/vnd.github.v3+json" \
  https://$HOSTNAME/api/v3/repos/org-name/test-four-keys/deployments \
  -d '{"ref":"21e706a11d41e467fe055dbdb5fe21a609427a20"}'

curl -u $GITHUB_USER:$GITHUB_TOKEN \
  -X POST \
  -H "Accept: application/vnd.github.v3+json" \
  https://$HOSTNAME/api/v3/repos/org-name/test-four-keys/deployments/8/statuses \
  -d '{"state":"success"}'

Resulting BigQuery contents in deployments table:

  {
    "source": "github",
    "deploy_id": "8",
    "time_created": "2020-12-09 11:44:48 UTC",
    "repository": "org-name/test-four-keys",
    "changes": [
      "21e706a11d41e467fe055dbdb5fe21a609427a20",
      "964198b26178b4203f14a77c77080af70a445750",
      "bb13c56e5e9b214682c28647adc787ff061183e2"
    ]
  }

Note that the commits related to branch a are not included.

In this case only the changes on branch b will be included in the lead time dashboard, but deployment frequency will show 1 deployment. If another deployment event had been created for merge commit of branch a, the frequency would be too high.

Add more data to show in the README and the METRICS.md.

Currently, the script uses random numbers and probability to create incidents. There should be at minimum one incident for the dashboard. Otherwise, the dashboard will not display the incident related charts.

Add instructions about the datastudio configuration to the setup.

Also, return the URL to the terminal.

Hi Dina,

I have attended your Four Keys presentation during cdCon2020. I liked the tool. I would like to know if it is possible to run the dashboard on the local setup? I would move it to GCP later. Thanks

regards,
Jayesh

Currently points to https://github.com/dinagraves/four-keys-playground

Update it to https://github.com/GoogleCloudPlatform/fourkeys

Today I found I can't browse our dashboards created before.
When opening the dashboards, following pop-up appeared.

And clicking "承認(Approval)", then following pop-up appeared.

What happened on "connector" ? (Honestly, I'm not familiar with "connector")
Maybe is it related to "publishing connector" ? -> #42 (comment)

Or anything wrong with me ?

Update Nox to only run applicable session on the files that have code changes in a PR.

The python script should have unit tests.

We can make the unit tests go faster, by running all of the python versions in parallel.

Per this doc, nox allows specifying python version on the command line, e.g.: python3 -m nox --python=3.7

We can use Cloud Build's parallel steps to test py 3.6, 3.7, and 3.8 (as specified in the noxfile) simultaneously.

This issue is just for the record. (sorry, to push you @dinagraves)
Would be cool to get more insight into the data connector and how it crunshes numbers. 😀

Hi @dinagraves

I followed the installation instructions and also generated some mock data. At last I connect the "Four Keys Dashboard" Connector with my project and it gave me a "Four Keys" datasource and the shared public template.
But when opening the dashboard I do see some numbers from June but not my mocked data.

Honestly I also can't figure out how the datasource/connector is "calculating"/"transforming" the actual data in the Bigquery tables (changes, deployments, incidents) into the dashboard.
Could you probably shed some light into this?

Thanks anyways for this cool project and work you've done on this!
best regards,
Jo

Hi,

We are using Jira instead of Github issues to track our bugs or change failures. What could be a good way to integrate Jira as source?
As a metric we could use the time between opening and resolving a Jira issue with type bug (in our case).
I was thinking about polling Jira once daily to import all issues with a specific type "bug" into the four_keys "incidents" table directly. I guess we would need to deactivate the scheduled query for incidents since it would truncate other created entries.
What do you think?

BR, Jo

use underscores! (see https://www.terraform-best-practices.com/naming)

...also look for any other opportunities for best practices.

Is there a linter?

If the user chooses to create a new project via the setup script, there is sometimes a delay before it is listed in the gcloud projects list command. The result is that we are unable to fetch the project number, which in turn means that the rest of the setup script will fail.

The script should wait until there is a valid project number, retrying the gcloud command until it returns the number, or a maximum of 5 minutes.

I have added my webhook to my gitlab repo, and now looking at how to review my recent project history.

Is there a mechanism to scan a git repo (any) and generate corresponding events_raw so start with 6-9 months of historical way of working on the four metrics? Or would that miss too much info to be of use?

Thinking this would be useful to maximise the info contained on any git repo (regardless of gitlab, github, bitbucket, etc...).

Include more information in the INSTALL.md. Currently there is information on how to do a variety of things in the README (eg run the data generator at will) which should also be included in the INSTALL.md.