cloudspannerecosystem / autoscaler Goto Github PK

Currently logs are written with severity DEFAULT, and with no structure.
Change to structured logging and specify severity.
https://cloud.google.com/functions/docs/monitoring/logging#writing_structured_logs

Need for a solution to cater for regional outages - i.e. multi-regional resilience.

Add a "margin" parameter to each metric.

For example:
high_prio_cpu = 65%
high_prio_cpu_margin = 5%

With the margin, the autoscaler will only scale up if the current high CPU measurement is over 70%, or will only scale down if the high CPU measurement is under 60%.
The margin effectively creates two bands. The metric is permitted to exist between those two bands without triggering an autoscale.

Why do this?
If we rely only on a specific number, for example 65%, the autoscaler will ALWAYS scale up or down when the cool down period runs out.

An alternative to a "margin" is to explicitly have min and max for each metric.
The advantage of having a margin instead of min/max is that the user only needs to think about a single number, and we can have a sensible default for the margin.

Hi Team,
We recently deployed the autoscaler components on our GCP GKE clusters to autoscale our regional(europe-west2) Spanner instance, all the deployments & cronJobs are working as expected, but the scaler deployment is not able to down/up scale the spanner nodes. - getting "Error 7 PERMISSION DENIED" in pod logs

We have created scaler & poller serviceAccounts with different names (not poller-sa & scaler-sa as suggested in the gke terraform documentation) & given predefined roles to them - roles/spanner.viewer to poller svcAcct & roles/spanner.admin to scaler svcAcct (using workload identities annotated with gcp service accounts in deployment yaml for both)

Plz suggest if we are missing something, or is this a known issue?

From https://github.com/cloudspannerecosystem/autoscaler/tree/master/poller page

The links in following paragraph are all 404, thus it lacks of instructions on how to configure custom metrics.

Cloud Spanner metric and filter that should be used when querying for data. The Autoscaler will automatically add the filter expressions for Spanner instance resources, instance id and project id.

Dataflow is considered as medium priority https://cloud.google.com/spanner/docs/cpu-utilization
so when there is high Dataflow load (import/export) and low high priority load autoscaler just doesn't work at all.

while it is possible to fix with custom metric

      "metrics" = [
        {
          "name"                     = "cpu_utilization_total"
          "filter"                   = "metric.type=\"spanner.googleapis.com/instance/cpu/utilization\""
          "regional_threshold"       = 90
          "multi_regional_threshold" = 90
        },

probably would be great if such rule will be added into default set

It can be desirable to scale on thresholds that are lower than the defaults. Currently those custom thresholds need to be set in the code. Doing so changes the threshold for all Cloud Spanner instances being autoscaled with that deployment of the Poller. If the custom thresholds could be included in the configuration JSON objects they could be set on a per instance basis and code changes wouldn't be required.

Example configuration:

[
    {
        "projectId": "my-spanner-project",
        "instanceId": "spanner1",
        "scalerPubSubTopic": "projects/my-spanner-project/topics/spanner-scaling",
        "minNodes": 1,
        "maxNodes": 3,
        "metrics": [
           {  name: "high_priority_cpu",
              regional_threshold: 30,
              multi_regional_threshold: 20
           }
       ]
    }
]

The autoscaler could be more responsive if it was able to scale up or down Spanner instance when a Cloud Alerting threshold is triggered. That would allow the autoscaler to act outside of the polling interval enabling it to scale up as soon as the scale up threshold is passed.

Looks like the App Engine Terraform resource can now create a Firestore db, so we should update the Autoscaler Terraform. This will clean up some steps in the deployment process and simplify the installation process.

Reference:
https://cloud.google.com/firestore/docs/solutions/automate-database-create#create_a_database_with_terraform
https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/app_engine_application

It can be desirable to scale on metrics other than CPU and Storage. Currently to add new metrics to scale on they need to be defined in the Poller function. If the custom metric could be included in the configuration JSON objects they could be set on a per instance basis and code changes wouldn't be required.

Example configuration:

[
    {
        "projectId": "my-spanner-project",
        "instanceId": "spanner1",
        "scalerPubSubTopic": "projects/my-spanner-project/topics/spanner-scaling",
        "minNodes": 1,
        "maxNodes": 3,
        "metrics": [
        { "name":  "high_priority_cpu",
           filter: 'metric.type="spanner.googleapis.com/instance/cpu/utilization_by_priority" AND metric.label.priority="high"',
           reducer: 'REDUCE_SUM',
           aligner: 'ALIGN_MAX',
           period: 60,
           regional_threshold: 30
           multi_regional_threshold: 20
           }
       ]
    }
]

Changes made to the Cloud Scheduler configuration as per the documentation are reset by Terraform on the next run. To support idempotency it would be optimal to make this configurable, with either Terraform or the direct Cloud Scheduler configuration specified as the source of truth.

The following terraform variables:

• dashboard_threshold_high_priority_cpu_percentage
• dashboard_threshold_rolling_24_hr_percentage
• dashboard_threshold_storage_percentage

only have defaults within the monitoring module, but not in the root modules.
Therefore, they cause the terraform scripts to stop and ask for values, which most users will not know what they are for.

Following the steps to Deploy a distributed Autoscaler tool for Cloud Spanner and forgot to enable Cloud Scheduler API beforehand.

Retried after enabling Cloud Scheduler API and it failed with the following error:

module.forwarder.google_storage_bucket.bucket_gcf_source: Creating...
╷
│ Error: googleapi: Error 409: You already own this bucket. Please select another name., conflict
│
│   with module.forwarder.google_storage_bucket.bucket_gcf_source,
│   on ../../modules/forwarder/main.tf line 40, in resource "google_storage_bucket" "bucket_gcf_source":
│   40: resource "google_storage_bucket" "bucket_gcf_source" {

This indicates that the deployment script is not idempotent.

I think the issue is this bucket:

I have an organization policy which requires uniform bucket access. I cannot use this module without the uniform_bucket_level_access flag set true. Will submit a PR.

Currently, the following fields are hard coded into the code:

• OVERLOAD_THRESHOLD for high priority CPU - currently at 90%
• regional_threshold for high priority CPU - currently at 65%

There are a couple others, but these are the main ones that are likely to impact Spanner performance.
It would be good to have these configurable for each spanner instance.

Spanner is a P0 dependency for some of our services and we already notice latency spikes at 65% high priority CPU utilization. Based on this, 90% is a too high of an overload threshold for us.

We currently manually change these thresholds ourselves when we deploy the autoscaler to 55% target and 75% overload.

When deploying the Autoscaler in distributed mode with its state stored in Spanner, logs show the following error:

The Cloud Firestore API is not available for Firestore in Datastore Mode

Because of this error, the scaling operation is not completed.

To get proper monitoring of the autoscaler, the tool should

• trace of internal processes (scaling decisions,.....) and different calls (http, pubsub, grpc) with the instance tagged
  • this would the benefit to get errors rate and latency distributions metrics for those
• have metrics for scaling events per instance with the the related GCM (high cpu, storage...) also, both as tags:
  • successful scaling count
  • scaling "abort" with the reason as a tag (bigger than max size, lower than min size)

I would suggest to use a open source library like OpenTelemetry for this.

Do we have option to identify minNodes and MaxNodes automatically as per load?

Rather expecting user to provide minNodes and MaxNodes, its better to identify nodes size automatically as per intelligence identified during load throughput KPI.

With regards,
Awadhesh Gupta

Long-running components (such as is the case when the autoscaler is deployed to Kubernetes, particularly with the scaler component) are observed to consume an increasing amount of resources over time -- this is primarily memory consumption. A suspected memory leak may be the cause of this, which is currently being investigated.

Currently the IAM permissions for the Poller component service account are multiply assigned in different Terraform source files, and overlapping IAM roles are applied (roles/spanner.viewer and roles/monitoring.viewer). Using a custom role would enable use of a simplified and more granular set of permissions.

Hi Team,
Need quick help in checking what will be implications of storing the state in the same spanner instance that we are autoscaling by the tool.

Plz feel free to close this once answered.

Thanks!

The autoscaler is only tracking the last time a scaling action is taken. It would also be helpful to capture the operation id for the last action taken, so we could log the status of the operation as it continues to monitor the Spanner instance.

It must be changed to functions-framework": "^2.0.0",

It would be great if there were tools that could collect all the schedules and configurations for a Spanner instance. This would be helpful in understanding the configuration where multiple methods are being used and in troubleshooting scenarios where the autoscaler isn’t working as intended.

It would be nicer if these were implemented as Terraform modules so they could be integrated into existing Terraform working directories. I've done so in a private fork for the per-project case and it was not very complex. Obviously, it might be more complex for the other working directories and then there's the matter of updating Google's docs, etc. so I'm not saying it'd be trivial; something to think about.

Setup GitHub Actions to run linter and unit tests.

Currently the High priority CPU aligner is align_max and reducer is reduce_sum. According to the documentation you point to in GCP monitoring alerts for spanner (https://cloud.google.com/spanner/docs/monitoring-cloud#high-priority-cpu) the high priority cpu aligner should be mean, while the aggregator (which I'm understanding to mean the reducer) should be max. This is different than the rolling 24 hour version.

The impact of this is that (in my case) I am seeing scale up events in my multi-region spanner instance when two locations sum to greater than 45% (which I don't think is ideal, if I am understanding those numbers).

Currently state is tracked in Firestore. This feature would optionally allow the state to be tracked in a Spanner database.

Hi!

First of all, this repository is great and can be help me. Today I work into in project shared into my company, and to create a new resource ever need declare an team prefix, however the autoscaler don't provide this into your terraform configs.

So, would it be possible to provide an prefix variable for terraform? If the answer is yes, I can pull this implementation and send a PR.

Context:
https://cloud.google.com/blog/products/databases/get-more-out-of-spanner-with-granular-instance-sizing

For the custom roles created via Terraform:

• spannerAutoscalerMetricsViewer
• spannerAutoscalerCapacityManager

Because a delete operation actuated by terraform destroy is only a soft delete, this can result in an inability to (re)build during a certain time window (7-37 days):

"You can't create a role_id (spannerAutoscalerCapacityManager) which has been marked for deletion."

To work around this, a random string may be appended to the role name(s), circumventing the restriction.

Enable Dependabot for library update tracking.

modules.spanner.google_spanner_instance_iam_member.spanner_admin_iam should also be suppressed when terraform_spanner is not enabled because it depends on the instance that is suppressed.

Previously we used to adjust each parameter in the data for pubsub_target of poller_job (e.g. minSize, maxSize, scalingMethod, etc.).

However, the data is currently a fixed value.

We want to adjust and use each parameter of data, so could you please modify it to be configurable by variables as before?

Using spanner persistent state, when a scaler manages multiple instances, the spannerAutoscaler table has only one record. Is it possible to separate multiple lines of processing. Row id can be set to projects/${spanner.projectId}/instances/${spanner.instanceId}
e.g
id
projects/project-1/instances/spanner-1
projects/project-1/instances/spanner-2

Poller function fails with below error

{ insertId: "000000-c7f61764-5b66-477e-8c8e-2448fc1cddff" jsonPayload: { message: "An error occurred in the Autoscaler poller function TypeError: spanners is not iterable at checkSpanners (/workspace/index.js:348:25) at processTicksAndRejections (internal/process/task_queues.js:95:5) at async exports.checkSpannerScaleMetricsPubSub (/workspace/index.js:360:5)" payload: { } } labels: { execution_id: "e9blyz99v02m" } logName: "****" receiveTimestamp: "2021-10-19T04:47:20.303390670Z" resource: { labels: { function_name: "tf-poller-function" project_id: "****" region: "****" } type: "cloud_function" } severity: "ERROR" timestamp: "2021-10-19T04:47:09.633Z" trace: "projects/*****" }

In the latest version of gcloud the command to create a Firestore database now uses --location instead of --region

It would be good to have spanner configs set to be in dry-run mode, where it would only log the intention.

When using Firestore to hold state in the centralized deployment model, the service account for the scaler component does not have cross-project permissions to write the state to the Firestore DB.

When the autoscaler scales up or down a Cloud Spanner instance it would be nice if it could notify users that it made a change to a Cloud Spanner instance.

Useful notification channels could be:

• Chat services such as Slack or Google Chat
• Email

Useful information could be:

• Spanner Instance
• Observed Metric, value and threshold
• Number of nodes added or removed

Currently, the default metrics have hard coded defaults. They should also use the metricsDefaults for consistency and to make the code easier to maintain.

https://github.com/cloudspannerecosystem/autoscaler/blob/master/terraform/modules/spanner/main.tf
uses authoritative resources and if there is already assigned roles - they will be silently overwritten

the non-authoritative resources should be used instead (and probably in other places), i.e.
google_spanner_instance_iam_member
google_project_iam_member

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_project_iam

resource "google_spanner_instance_iam_binding" "spanner_metadata_get_binding" {
  instance = var.spanner_name
  role = "roles/spanner.viewer"
  project      = var.project_id

  members = [
    "serviceAccount:${var.poller_sa_email}",
  ]
  depends_on = [google_spanner_instance.main]
}

resource "google_spanner_instance_iam_binding" "spanner_admin_binding" {
  # Allows scaler to change the number of nodes of the Spanner instance
  instance = var.spanner_name
  role = "roles/spanner.admin"
  project      = var.project_id

  members = [
    "serviceAccount:${var.scaler_sa_email}",
  ]
  depends_on = [google_spanner_instance.main]
}

resource "google_project_iam_binding" "poller_sa_cloud_monitoring" {
  # Allows poller to get Spanner metrics
  role    = "roles/monitoring.viewer"
  project = var.project_id

  members = [
    "serviceAccount:${var.poller_sa_email}",
  ]
}

When the number of target Spanner instances being autoscaled grows large, it can be difficult to interpret the logs from the poller and scaler components. Possibilities for addressing this include some combination of:

• Adding a logline prefix and/or other identifying field
• Moving to more structured logging
• Using Log Analytics to enable structured querying
• Creating a custom dashboard to be able to drill down into logs

Cloud Scheduler is currently the only supported scheduler mechanism. While Cloud Scheduler is serverless, which has the benefits of having a low maintenance overhead there are some limitations:

• Publishing Pub/Sub messages cross project
• Lack of VPC service controls
• Requires AppEngine

These could be worked around with an alternative scheduler, perhaps a Spring base scheduler or Kubernetes Cron Job.

We are seeing an issue with a promise not triggering an error upon quota limits. Our calls all return without an error in the promise (We should see an error since this will fail due to insufficient quota). Our usage:

const spannerInstance = spannerClient.instance(spanner.instanceId);
  spannerInstance.setMetadata(metadata)
    .catch(err => {
      console.log("----- " + spanner.projectId + "/" + spanner.instanceId + ": Unsuccessful scaling attempt.\n\tThis may be caused by a previous ongoing scaling operation.");
      console.debug(err);
    });

However, this approach will successfully show the error:

spannerInstance.setMetadata(metadata, function(err, operation, apiResponse) {
    if (err) {
        console.log('err on func')
        return
    }
 
    operation
        .on('error',  function(err) {
                console.log(err)
        })
        .on('complete', function() {
                console.log('we finished!')
        });
  });

Output:

% node app.js
GoogleError: Project ************* cannot add 4 nodes in region us-central1.
    at Operation._unpackResponse (/Users/******/Desktop/projects/testjs/node_modules/google-gax/build
/src/longRunningCalls/longrunning.js:133:31)
    at /Users/*******/Desktop/projects/testjs/node_modules/google-gax/build/src/longRunningCalls/longr
unning.js:119:18 {
  code: 8
}

Currently the Workload Identity Terraform module does not support Kubernetes versions greater than 1.23, while 1.24 or higher is now the default version across all GKE release channels. For now we pin the version to 1.23, though this version will transition to end-of-life on 2023-07-31. An update to the Workload Identity module is being tracked in the issue workload-identity compatibility.

Given,

spanners[sIdx] = {...nodesDefaults, ...spanners[sIdx]};
if(spanners[sIdx].minNodes && spanners[sIdx].minSize != spanners[sIdx].minNodes) {
    log(`DEPRECATION: minNodes is deprecated, remove minNodes from your config and instead use: units = 'NODES' and minSize = ${spanners[sIdx].minNodes}`,
       'WARNING');
    spanners[sIdx].minSize = spanners[sIdx].minNodes;
}

Because this logic does not care that minSize was specified and because minNodes is provided by nodesDefaults if a payload config does not provide minNodes then the default minNodes overrides the provided minSize.

Same all applies to maxSize/maxNodes.

Also note that the warning for the maxNodes case accidentally mentions minNodes.

Many time teams have a dashboard that tracks important Spanner metrics. It would be helpful if the we had a starter dashboard users could deploy to get started.

Good metrics to include on the dashboard would be the scaling metrics and node count grouped by instance:

• High CPU Utilization
• Rolling CPU Utilization
• Storage Utilization
• Node Count

When a Spanner instance reaches a size > 1000 PUs, then the STEPSIZE method does not scale-in the instance if stepSize<1000.

For example:
An instance currently has 4000 PUs and stepSize 100
The instance needs to be scaled-in.
The STEPWISE method will suggest 3900 PUs. However, this number will be rounded up to 4000 PUs and scaling-in will not proceed.

Currently the Spanner Autoscaler has dependencies on a number of Google Cloud services, adjacent to Cloud Spanner:

• Cloud Scheduler
• Cloud Functions
• Pub/Sub

For organisations/teams that cannot or do not desire to use these services, a deployment option to run the Autoscaler within a Google Kubernetes Engine (GKE) cluster could be introduced. This would enable customers that already run GKE to use the Autoscaler without taking on additional GCP service dependencies.