#WHAT: Create dashboard that represents four golden signals of monitoring (latency, traffic, errors, and saturation) for the productcatalogservice

#WHY: Creates opinionated monitoring for this service

#HOW: Terraform resources added to initialization terraform scripts

#WHAT: Create dashboard that represents four golden signals of monitoring (latency, traffic, errors, and saturation) for the three services: paymentservice, emailservice, and shippingservice

#WHY: Creates opinionated monitoring for these services

#HOW: Terraform resources added to initialization terraform scripts. Should use a terroform templatefile to provision the same dashboards for each of these services

Description: A user may get an "Invalid Snapshot Position" Error when following these steps to create a snapshot in the Stackdriver Sandbox User Guide. This made it impossible to see any values coming from the application, since the snapshot was never activated.

Error: The error text reads: "Invalid snapshot position: Loaded script contained x lines. Please ensure that the snapshot was set in the same code version as the deployed source." A picture is shown below.

Fix: This error is occurring because the service is deployed using container images that were created from the source code that is earlier than the currently checked-in version on GitHub. One possible fix is that the service should be deployed using updated container images from the most recent version.

#WHAT: Create two SLOs for the frontend service (1 for availability and 1 for performance) and an alerting policy on budget burn for each one. SLO defined as 99.9% availability/performance over 30 day rolling window. Alert when 10% budget expended over 1 hour.

#WHY: Demonstrate best practice alerting on SLO error budget burn through terraform for this service

#HOW: Terraform resources added into initialization terraform scripts

The Error Reporting section has two issues:

1. It seems that we need to manually enable the Error Reporting API to get it to work. This should be added to the instructions in the docs.
2. This demonstration would be more powerful if the user were able to manufacture some errors and see them reported on the UI. The docs could have some "how to break it" instructions, e.g. stopping nodes through the GCP console or a particular pattern using the load generator.

When Sandbox creation completes, the log shows a list of key URLs for next steps.
There should be a command to retrieve this URLs at any time after project creation.

We need to add istio support, but likely having Anthos Service Mesh is even better option to illustrate how it helps observe the apps.

#WHAT: Create two SLOs for the checkoutservice (1 for availability and 1 for performance) and an alerting policy on budget burn for each one. SLO defined as 99.9% availability/performance over 30 day rolling window. Alert when 10% budget expended over 1 hour.

#WHY: Demonstrate best practice alerting on SLO error budget burn through terraform for this service

#HOW: Terraform resources added into initialization terraform scripts

The Stackdriver Sandbox website does not link to the official user guide, and only has set-up instructions through step 2 (run ./install.sh). If a user were approaching from that website, they may be confused with how to proceed from there.

It would be helpful to include a link from that website to the User Guide, which has more detailed documentation. For example, add to the website "Step 3: Follow Set-Up instructions in the User Guide" (along with a link to the User Guide).

Full Cloud Shell output:

./install.sh
Checking Prerequisites...
Checking for billing accounts
using billing account: Personal Billing Account
Make sure Terraform is installed
Initialize terraform state

Initializing the backend...

Initializing provider plugins...

The following providers do not have any version constraints in configuration,so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

• provider.null: version = "~> 2.1"
  Terraform has been successfully initialized!
  You may now begin working with Terraform. Try running "terraform plan" to see
  any changes that are required for your infrastructure. All Terraform commands
  should now work.
  If you ever set or change modules or backend configuration for Terraform,
  rerun this command to reinitialize your working directory. If you forget, other
  commands will detect it and remind you to do so if necessary.
  Apply Terraform automation
  null_resource.customize_manifests: Refreshing state... [id=8555647629071790355]
  data.google_billing_account.acct: Refreshing state...
  random_id.project: Refreshing state... [id=85ny-g]
  google_project.project: Refreshing state... [id=stackdriver-sandbox-4086952698]
  Error: Error reading Project "stackdriver-sandbox-4086952698": googleapi: Error 403: The caller does not have permission, forbidden

How do we want the script to behave when it is run multiple times? Currently, terraform will skip deploying a new project, but the load balancer portion will create a new load balancer.

Do we want the script to cancel if it sees tfstate files in the current directory? Or should it always deploy another sandbox instance? Or something else?

#WHAT: Create two SLOs for the shippingservice (1 for availability and 1 for performance) and an alerting policy on budget burn for each one. SLO defined as 99.9% availability/performance over 30 day rolling window. Alert when 10% budget expended over 1 hour.

#WHY: Demonstrate best practice alerting on SLO error budget burn through terraform for this service

#HOW: Terraform resources added into initialization terraform scripts

Since Stackdriver has been rebranded to Operations, the following would be useful:

• The install script links to Stackdriver Dashboard (app.google.stackdriver.com/accounts/create) at the end, but this page no longer contains any information. This should be removed from the install script and docs updated accordingly.
• All instances of project name "Stackdriver" should be renamed to "Operations" in the docs, and tool names from "Stackdriver *" to "Cloud *".
• There are a few images that may be misleading since they show "Stackdriver" in the UI where users should now expect to see "Operations"

#WHAT: Create dashboard that represents four golden signals of monitoring (latency, traffic, errors, and saturation) for the checkoutservice

#WHY: Creates opinionated monitoring for this service

#HOW: Terraform resources added to initialization terraform scripts

#WHAT: Create two SLOs for the currencyservice (1 for availability and 1 for performance) and an alerting policy on budget burn for each one. SLO defined as 99.9% availability/performance over 30 day rolling window. Alert when 10% budget expended over 1 hour.

#WHY: Demonstrate best practice alerting on SLO error budget burn through terraform for this service

#HOW: Terraform resources added into initialization terraform scripts

#WHAT: Create two SLOs for the paymentservice (1 for availability and 1 for performance) and an alerting policy on budget burn for each one. SLO defined as 99.9% availability/performance over 30 day rolling window. Alert when 10% budget expended over 1 hour.

#WHY: Demonstrate best practice alerting on SLO error budget burn through terraform for this service

#HOW: Terraform resources added into initialization terraform scripts

The emailservice appears to be in a crashloop when testing on my personal project. The error message is:

google.api_core.exceptions.PermissionDenied: 403 Stackdriver Trace API has not been used in project 1093971809799 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudtrace.googleapis.com/overview?project=1093971809799 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

Do we need to enable the API before using stackdriver sandbox? Is this not done as part of the install script?

#WHAT: Create two SLOs for the emailservice (1 for availability and 1 for performance) and an alerting policy on budget burn for each one. SLO defined as 99.9% availability/performance over 30 day rolling window. Alert when 10% budget expended over 1 hour.

#WHY: Demonstrate best practice alerting on SLO error budget burn through terraform for this service

#HOW: Terraform resources added into initialization terraform scripts

• Make sure it's available when we print its public IP
• Consider moving it to container instead of VM

Recommendation service probes fail frequently. Removing them makes the service more stable.
We should do it in both main and Next'19 branches.

#WHAT: Create two SLOs for the cartservice (1 for availability and 1 for performance) and an alerting policy on budget burn for each one. SLO defined as 99.9% availability/performance over 30 day rolling window. Alert when 10% budget expended over 1 hour.

#WHY: Demonstrate best practice alerting on SLO error budget burn through terraform for this service

#HOW: Terraform resources added into initialization terraform scripts

#WHAT: Create an HTTP uptime check on the external IP of the demo application. Alerting policy is defined as number of fails exceeds 1 for 1 minute over 20 minutes polling period. Email notification alerts to owner of the project

#WHY: Add best practice uptime check monitoring in the form of uptime check and associated alerting policy

#HOW: Terraform resources added to initialization terraform scripts

main.css includes:

background-image: url(../images/content_copy.svg);

There is no content_copy.svg in the repository. Should the image be added or the css be updated?

It would make it more clear to introduce the user to how to make an Analysis Report when going through the Trace section in the User Guide, since when the user first opens Trace, there should be no reports generated.

Between "Finally, click Analysis Reports in the navigation menu to see a list of reports that are generated" and "View one of the reports that was created (or the one you created yourself) to understand either the density or cumulative distribution of latency for the call you selected", a few sentences can be added with documentation on Report-creation.

Loadgenerator tool has option

./loadgenerator-tool setup

This option creates a firewall rule that allows accessing the VM that runs locust using external IP.
We need to remove this option and instead make sure that every time new job is created, corresponding firewall rule is created/updated as well.

There's an insufficient quota issue when running the script install.sh. It failed to create the cluster.

Error: googleapi: Error 403: Insufficient regional quota to satisfy request: resource "IN_USE_ADDRESSES": request requires '5.0' and is short '1.0'. project has a quota of '4.0' with '4.0' available. View and manage quotas at https://console.cloud.google.com/iam-admin/quotas?usage=USED&project=stackdriver-sandbox-69333807., forbidden

on 03_gke_cluster.tf line 42, in resource "google_container_cluster" "gke":
42: resource "google_container_cluster" "gke" {

#WHAT: Creates dashboard representing four golden signals of monitoring (latency, traffic, errors, and saturation) for the frontend service

#WHY: Creates opinionated monitoring of this service

#HOW: Terraform resources added to initialization terraform scripts

#WHAT: Create two SLOs for the recommendationservice (1 for availability and 1 for performance) and an alerting policy on budget burn for each one. SLO defined as 99.9% availability/performance over 30 day rolling window. Alert when 10% budget expended over 1 hour.

#WHY: Demonstrate best practice alerting on SLO error budget burn through terraform for this service

#HOW: Terraform resources added into initialization terraform scripts

#WHAT: Create dashboard that represents four golden signals of monitoring (latency, traffic, errors, and saturation) for the cartservice

#WHY: Creates opinionated monitoring for this service

#HOW: Terraform resources added to initialization terraform scripts

The Error Reporting section would be a more powerful demonstration if the user were able to manufacture some errors and see them reported on the UI. The docs could have some "how to break it" instructions, e.g. stopping nodes through the GCP console or a particular pattern using the load generator.

#WHAT: Create two SLOs for the productcatalogservice (1 for availability and 1 for performance) and an alerting policy on budget burn for each one. SLO defined as 99.9% availability/performance over 30 day rolling window. Alert when 10% budget expended over 1 hour.

#WHY: Demonstrate best practice alerting on SLO error budget burn through terraform for this service

#HOW: Terraform resources added into initialization terraform scripts

I just created a new Google account ([email protected]) to create a new sandbox. Here's the last part of Cloud Shell output. It did not create the load generator instance.

Stackdriver Sandbox deployed successfully!
Stackdriver Dashboard: https://app.google.stackdriver.com/accounts/create
Google Cloud Console Dashboard: https://console.cloud.google.com/kubernetes/workload?project=stackdriver-sandbox-347738008
Hipstershop web app address: http://34.67.163.61
Load generator web interface: [not found]

https://www.youtube.com/watch?v=QW7bnzqSawk&t=24

#WHAT: Create dashboard that represents four golden signals of monitoring (latency, traffic, errors, and saturation) for the currencyservice

#WHY: Creates opinionated monitoring for this service

#HOW: Terraform resources added to initialization terraform scripts

Getting this error while running terraform install script

• data.google_billing_account.acct: data.google_billing_account.acct: More than one matching billing account found

When running:

./loadgenerator-tool create --zone us-east1-c http://[ip] 

it creates a new VM with assumption to connect to http://http://[ip]
Instead it should verify correctness of the input and result in http://[ip]

#WHAT: Create two SLOs for the adservice (1 for availability and 1 for performance) and an alerting policy on budget burn for each one. SLO defined as 99.9% availability/performance over 30 day rolling window. Alert when 10% budget expended over 1 hour.

#WHY: Demonstrate best practice alerting on SLO error budget burn through terraform for this service

#HOW: Terraform resources added into initialization terraform scripts

#WHAT: Create dashboard that represents four golden signals of monitoring (latency, traffic, errors, and saturation) for the recommendationservice

#WHY: Creates opinionated monitoring for this service

#HOW: Terraform resources added to initialization terraform scripts

When creating a new Sandbox, the instructions contain a list of URLs of "next steps".
In dashboard URI, we need to add a project parameter so the link opens correctly.
https://app.google.stackdriver.com/accounts/create?**project=<CREATED_PROJECT?>**

#WHAT: Create dashboard that represents four golden signals of monitoring (latency, traffic, errors, and saturation) for the adservice

#WHY: Creates opinionated monitoring for this service

#HOW: Terraform resources added to initialization terraform scripts

When setting up Stackdriver Debugger following the steps in the Sandbox User Guide, I receive a message asking whether I would like to enable the API in the cloud shell.

This is a step that could be done as a part of Sandbox provisioning.

Note: following the choice to enable the API, in order to continue, the user must enable billing specifically for an API method called. If the API is enabled, it would be helpful to either add instructions to the User Guide specifying that API billing should be set-up, or find a way to do it automatically. Screenshot for this is shown below.

The earlier billing is enabled, the better, since it can take a few minutes for the system to register the updated billing information before the user can move on to the next step.

The docs/README.md#Stackdriver Monitoring section has some outdated information:

• The Stackdriver Monitoring console is no longer "a separate UI from the consoles for other GCP and Stackdriver products." The images should be updated too.
• The new Monitoring dashboard does not have a Resources > (Infrastructure) Kubernetes Engine. I found similar info through Monitoring > Dashboards > Kubernetes Engine > Infrastructure tab, but I'm not sure if that's the right place.
• Similarly, the route for finding Metrics Explorer is Monitoring > Metrics Explorer instead of going through Resources
• In log query creation, resource is under Kubernetes Container > stackdriver-sandbox > default > server instead of GKE Container > stackdriver-sandbox > default + server from dropdown menu
• In Logging, it looks like "Create Export" is now "Create Sink"

After Sandbox is created, the user should receive instructions on "what's next" which should include 101 overview of all Stackdriver components.

We need to manually enable the Error Reporting API to get it to work in Sandbox. This should be automatically enabled by the provisioning

Currently readme.md contains both the explanation what is Sandbox, as well as instructions for developers on how to build and deploy it using skaffold. The latter should move to contributing.md that is only relevant to folks interested in contributing to this repo as opposed to readme.md that targets everyone, including folks who discover Sandbox and only want to use it as finished product.

#WHAT: Terraform support to create example dashboard for User Experience, contains metrics that are relevant to user interaction with the application

#WHY: We want to provide dashboard that reveals metrics about the demo application as whole

#HOW: Additional terraform config file created alongside initial provisioning terraform