terraform-google-modules / terraform-google-github-actions-runners Goto Github PK

Awesome module, can't thank you guys enough for spending the time to put this together! Do you have any plans to publish this to the terraform registry?
Thanks again folks!

TL;DR

The gh-runner-mig-vm module (v3.1.0) creates a secret in the Secret Manager with name gh-token and value gh_token. A second module, with other labels, image, service account, replicas, etc. will attempt to create the same secret, which will result in an error:

Error: Error creating Secret: googleapi: Error 409: Secret [projects/xxxxxxxxx/secrets/gh-token] already exists.
with module.my_module_name.google_secret_manager_secret.gh-secret
on .terraform/modules/my_module_name/modules/gh-runner-mig-vm/main.tf line 75, in resource "google_secret_manager_secret" "gh-secret":

Terraform Resources

Terraform - gh-runner-mig-vm

GitHub - gh-runner-mig-vm

Detailed design

Ideally, add a new optional variable, gh_token_key, that defines what name should be given to the secret to avoid conflicts. Another alternative would be to add a variable, similarly to create_network and create_subnetwork, create_gh_token so that it's possible to control whether to create the secret or attempt to use an existing one with the gh-token name.

Additional information

No response

TL;DR

GH OIDC identity pool provider currently has the github.com issuer encoded as expected. With the 3.5 server release workload identity is supported in GHES. The issuer_uri now has to be configurable

Detailed design

The current gh_oidc terraform setup config assumes the github.com issuer for the tokens: https://github.com/terraform-google-modules/terraform-google-github-actions-runners/blob/v3.0.0/modules/gh-oidc/main.tf#L37

With the 3.5 server release, the GHES can issue its own tokens that are not accepted by the current gh_oidc-setup identity pool provider with the error:

{
  "error":"invalid_grant",
  "error_description":"The issuer in ID Token https://<server domain>/_services/token does not match 
                           the expected ones: https://token.actions.githubusercontent.com."}

I believe having the issuer_uri configurable as a var in the TF module would resolve this.

Enabling the GCP APIs from within the module seems like a good idea, until you consider that this module is likely to be used from within other modules and terraform environments, environments that are likely to already have at least some of these APIs enabled (compute for example).

The problem comes when you try to destroy or remove this module.
If this module includes terraform code for enabling the APIs, then destroying this module would implicitly disable those google endpoints even if they're in use by other parts of the terraform code base for that GCP project.

I would humbly suggest that it may be better to leave enabling of those APIs outside of the module and maybe include a blurb in the documentation with examples for enable them from terraform instead. Alternatively, a parameter could be passed into this module to prevent it from trying to manage those API endpoints.

Hello again!

I've gotten to a place where I've been able to make some changes, and I'm ready to begin submitting them upstream. I see a 'test' directory and some things inside of it but it isn't clear to me how I'm supposed to run those tests.

If anyone has a moment, a section in the Readme about how to run the test suite would be appreciated :)

Thanks again for your work on this, I appreciate it!

TL;DR

Add a new example showing how to deploy actions runner controller with the GKE runner module using the helm provider to install ARC.

Terraform Resources

No response

Detailed design

No response

Additional information

No response

TL;DR

Create a variable to pass a boolean to the mig_template module to use the enable_shielded_vm variable.

Terraform Resources

https://github.com/harrison-carter/terraform-google-vm/blob/master/modules/instance_template/main.tf#L171-L178

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance#nested_shielded_instance_config

Detailed design

No response

Additional information

No response

TL;DR

Ideally, GitHub actions should use their own service account to authenticate.

Therefore it would be helpful to include creation of the SA in the oidc submodule.

Terraform Resources

No response

Detailed design

No response

Additional information

No response

TL;DR

Add Docker to the startup-script for gh-runner-mig-vm

Terraform Resources

No response

Detailed design

No response

Additional information

No response

TL;DR

It seems that there is race condition in shutdown script, when script is trying to invoke gcloud command, but it is not allowed to run.

In shutdown logs I can see:

Mar 22 13:12:02 github-runner-g0vk GCEMetadataScripts[1609]: 2022/03/22 13:12:02 GCEMetadataScripts: Found shutdown-script in metadata.
Mar 22 13:12:03 github-runner-g0vk GCEMetadataScripts[1609]: 2022/03/22 13:12:03 GCEMetadataScripts: shutdown-script: internal error, please report: running "google-cloud-sdk.gcloud" failed: cannot create transient scope: DBus error "org.freedesktop.systemd1.TransactionIsDestructive": [Transaction for snap.google-cloud-sdk.gcloud.506272c3-22a2-4337-a998-723a7f1f3ce1.scope/start is destructive (poweroff.target has 'start' job queued, but 'stop' is included in transaction).]
Mar 22 13:12:03 github-runner-g0vk GCEMetadataScripts[1609]: 2022/03/22 13:12:03 GCEMetadataScripts: shutdown-script exit status 46
Mar 22 13:12:03 github-runner-g0vk GCEMetadataScripts[1609]: 2022/03/22 13:12:03 GCEMetadataScripts: Finished running shutdown scripts.

Expected behavior

I expect github runner VM to self unregister from github.

Observed behavior

Only some of the VMs (~15%) get unregistered on shutdown. The rest VMs are dangling in the GitHub's runners list.

Terraform Configuration

n/a

Terraform Version

Terraform v1.1.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v4.4.0
+ provider registry.terraform.io/hashicorp/google-beta v4.4.0

Additional information

No response

I use pycharm as my IDE currently, which creates files inside of an .idea directory, and without updating .gitignore git will want to commit these files.

I'd like to update the .gitignore file to exclude these files if you don't mind, it will benefit any other devs who also use Pycharm.

TL;DR

WorkloadIdentityPool pools are unable to recreated after they are deleted since they have the ability to be "undeleted". Therefore if you run terraform destroy and terraform apply you will always get Error creating WorkloadIdentityPool: googleapi: Error 409: Requested entity already exists

The fix would be to check if the pool is in a deleted state and call undelete instead of create during terraform apply

Expected behavior

No response

Observed behavior

No response

Terraform Configuration

module "gh_oidc" {
  source                = "terraform-google-modules/github-actions-runners/google//modules/gh-oidc"
  project_id            = var.project_id
  pool_id               = var.pool_name
  pool_display_name     = var.pool_name
  provider_id           = var.provider_name
  provider_display_name = var.provider_name
  attribute_mapping = {
    "attribute.aud" : "assertion.aud",
    "attribute.actor" : "assertion.actor"
    "attribute.repository" : "assertion.repository",
    "google.subject" : "assertion.sub"
  }
  sa_mapping = {
  }

### Terraform Version

```sh
Terraform v1.0.8
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v3.90.1
+ provider registry.terraform.io/hashicorp/google-beta v4.5.0

Your version of Terraform is out of date! The latest version
is 1.1.2. You can update by downloading from https://www.terraform.io/downloads.html

Additional information

No response

README.md has TF 0.12 as a requirement, however the versions.tf file in all of the modules specify 0.13 or ~0.13

TL;DR

Any chance we could get a patch to the GKE runner to support Orgs in the same way that you've done it for MIG?

Fyi @martinhale @bmaximuml

Terraform Resources

No response

Detailed design

No response

Additional information

No response

TL;DR

As far as I can see in the google provider terraform docs these resources are now in the google provider.

Can you remove the google-beta provider or is there some other reason for keeping it?

Terraform Resources

No response

Detailed design

No response

Additional information

No response

TL;DR

Thank you team for the Simple GKE setup. works great!
Looking to see if it's straight forward to run windows runners on this setup. Reading about it is possible but not sure if it's supported

Additionally, is there a way to clean up the pod/runner after it runs so I can guarantee clean builds?

Terraform Resources

No response

Detailed design

No response

Additional information

No response

TL;DR

Based on current code level there is no possibility to specify a Runner Group for Enterprise organization accounts. Currently all registered new runners are automatically assigned to Default Runners Group.

Hence, the suggestion is to add an option to register runner and assign to the group(s) as part of this module for MIG native VMs and containers.
Documentation upon groups can be found here.

Thanks

Terraform Resources

# Add variable

variable "gh_runner_group" {
  type        = string
  description = "GitHub runner group"
  default     = "Default"
}

Detailed design

The change would most likely need to happen in startup scripts (VMs) and entrypoint.sh (containers) for /runner/config.sh parameters. Something like that:

RUNNER_ALLOW_RUNASROOT=1 /runner/config.sh --unattended --replace --work "/runner-tmp" --url "$REPO_URL" --token "$ACTIONS_RUNNER_INPUT_TOKEN" --runnergroup "$ACTIONS_RUNNER_GROUP"

The cooldown period for the MIG isn't currently parameterized, I've needed to include that though and I'd like to upstream it for others if I haven't made too big a mess of this.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): Update module github.com/gruntwork-io/terratest to v0.47.1

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Issue

When I used the gh-oidc module, I got the error. I think the manner of GitHub OIDC was updated.

ERROR: gcloud crashed (OAuthError): ('Error code invalid_grant: Specified identity issuer https://token.actions.githubusercontent.com not found in the configured identity providers.', '{"error":"invalid_grant","error_description":"Specified identity issuer https://token.actions.githubusercontent.com not found in the configured identity providers."}')

Though I haven't completely understood the specification of GitHub OIDC, google_iam_workload_identity_pool_provider.oidc.issue_uri should be https://token.actions.githubusercontent.com.

https://github.com/terraform-google-modules/terraform-google-github-actions-runners/blob/v1.1.0/modules/gh-oidc/main.tf#L35-L38

  oidc {
    issuer_uri        = "https://token.actions.githubusercontent.com"
  }

TL;DR

Google provider v5.0 was released few days ago, but the modules in this repo don't support v5.

Terraform Resources

No response

Detailed design

Additional information

By Terraform Google Provider 5.0.0 Upgrade Guide | Guides | hashicorp/google | Terraform | Terraform Registry, it seems like label behavior change may affect our modules.

version constraint lines:

• https://github.com/terraform-google-modules/terraform-google-github-actions-runners/blob/master/modules/gh-oidc/versions.tf#L23-L27
• https://github.com/terraform-google-modules/terraform-google-github-actions-runners/blob/master/modules/gh-runner-gke/versions.tf#L23
• https://github.com/terraform-google-modules/terraform-google-github-actions-runners/blob/master/modules/gh-runner-mig-vm/versions.tf#L23-L27
• https://github.com/terraform-google-modules/terraform-google-github-actions-runners/blob/master/modules/gh-runner-mig-container-vm/versions.tf#L23

TL;DR

When running the gcloud builds submit --config=cloudbuild.yaml command in the Simple GKE hosted runner instructions, I'm erroring out with E: Unable to locate package liblttng-ust0.

Expected behavior

The above error should not happen, the build should be successful.

Observed behavior

The build is not successful due to the above.

Terraform Configuration

Does not apply.  I used the exact Terraform configuration as specified in the GKE simple setup successfully.

Terraform Version

08:43 $ terraform version
Terraform v1.6.0
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v5.33.0
+ provider registry.terraform.io/hashicorp/google-beta v4.85.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.31.0
+ provider registry.terraform.io/hashicorp/random v3.6.2

Your version of Terraform is out of date! The latest version
is 1.8.5. You can update by downloading from https://www.terraform.io/downloads.html

Additional information

Gist of log can be found here

Hi, I need to setup runner to be shared accross multiple repos in organizaton, is current module support this?

It looks like this sets up a MIG, an autoscaler, an instance template, and even a secret in GCP SM but... it doesn't seem to do anything with the secret?

The startup script variable defaults to blank, I was thinking the module would include a startup script that would download the values from the secret, and install+configure a github actions runner.

Apologies if I've missed the section that does this, I have gone looking but I can't find it.

TL;DR

It would be good to have an example or module for using Autopilot to host runners.

Terraform Resources

No response

Detailed design

No response

Additional information

No response

README requirements stipulate Terraform v0.12 while modules require usage of v0.13

Requirements: https://github.com/terraform-google-modules/terraform-google-github-actions-runners#requirements

Module version example: https://github.com/terraform-google-modules/terraform-google-github-actions-runners/blob/master/modules/gh-runner-mig-vm/versions.tf#L18

TL;DR

Runners deployed with MIG not added to github repository. Can't ssh to gcp instances.

Expected behavior

Runners will be added to the repository.

Observed behavior

The repository does not have instances assigned after terraform deployment finished. SSH to instances is not working.

Terraform Configuration

module "runner-mig-dind" {
  source         = "terraform-google-modules/github-actions-runners/google//modules/gh-runner-mig-vm"
  create_network = true
  project_id     = var.project_id
  repo_name      = var.repo_name
  repo_owner     = var.repo_owner
  gh_token       = var.gh_token
}

---
tfvars:

project_id = "my-project"
repo_name  = "my-repo"
repo_owner = "my-org"
gh_token     = "my-PAT"

Terraform Version

Terraform v1.1.3
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v4.9.0
+ provider registry.terraform.io/hashicorp/google-beta v4.9.0

Your version of Terraform is out of date! The latest version
is 1.1.4. You can update by downloading from https://www.terraform.io/downloads.html

Additional information

same code working for module version v2.0.1

It might be useful in some cases to register multiple action runners on a large compute machine to parallelize jobs & utilise the remaining capacity of the instance while the instance-group is scaling up (which takes some time).

I have made & tested the changes in my fork (https://github.com/raj-saxena/terraform-google-github-actions-runners/pull/2/files).
Example -

module "runner-mig" {
  source                     = "github.com/raj-saxena/terraform-google-github-actions-runners//modules/gh-runner-mig-vm?ref=1468894a76e090779f9a0a7a8d004722cf50a585"
  create_network             = true
  project_id                 = var.project_id
  repo_owner                 = var.repo_owner
  gh_token                   = var.gh_token
  gh_runner_instances_per_vm = 3
}

@bharathkkb Would this be something that you are interested in? If yes, I'll create a PR soon based on your reply.

TL;DR

We are getting New Advisory Notification in GCP because there is no attribute_condition set:

Would be great if the Example Usage in the Readme: https://github.com/terraform-google-modules/terraform-google-github-actions-runners/tree/master/modules/gh-oidc would include this.

Expected behavior

The Readme https://github.com/terraform-google-modules/terraform-google-github-actions-runners/tree/master/modules/gh-oidc should include the attribute_condition in the examples.

Or go even further and make attribute_condition a required field.

Observed behavior

We are getting Important information about Workload Identity Federation and Identity Pool Configurations notifcations in GCP because we are missing an attribute_condition.

Terraform Configuration

module "gh_oidc" {
  source      = "terraform-google-modules/github-actions-runners/google//modules/gh-oidc"
  project_id  = var.project_id
  pool_id     = "example-pool"
  provider_id = "example-gh-provider"
  sa_mapping = {
    "foo-service-account" = {
      sa_name   = "projects/my-project/serviceAccounts/[email protected]"
      attribute = "attribute.repository/${USER/ORG}/<repo>"
    }
  }
}

### Terraform Version
irrelevant


### Additional information

_No response_

The gh-runner-mig-vm module doesn't seem to use the machine_type variable

The use of google_project_iam_binding here ensures that in the given project, no other role/user can be in the role "roles/storage.objectViewer". This is undesirable in my case and seems like a not ideal thing to do as it seems likely to conflict with other terraform used to manage roles.
Proposed fix incoming

TL;DR

It would be better to add a static ip address for the mig-vm which can set as output.
This ip address can be used to whitelist in apps or infra where ip restriction is in place and needs to be accessed via github actions.
An example can be to be access k8s cluster in which api server is ip restricted.

Terraform Resources

https://github.com/terraform-google-modules/terraform-google-vm/tree/v11.1.0/modules/mig#:~:text=no-,stateful_ips,-Statful%20IPs%20created

Detailed design

No response

Additional information

No response

Right now the module takes the Github Token as a direct parameter, and then deposits it into a GCP-SM Secret.

Putting the Github token in plaintext in our terraform repo is poor form, so I've put it into a GCP SM already outside of this module, I then use a data source to extract that secret and pass it into the module. It doesn't seem necessary for the module to then create a new GCP SM secret to put the key into.

Ideally, I could just pass in the name (and GCP Project) of the GCP SM secret containing the Github Token.

Again I just wanna say that I'm loving the module, I hope you don't mind my filing issues, I do intend to upstream my fixes if I patch anything myself. Cheers!

Hi!

Thanks for such clean work on these modules. Just wondering if the actions run on the VMs also have access to the secrets, mainly GITHUB_TOKEN, via the service account associated with the VM. If yes, then is there a way to use these modules without storing the GITHUB_TOKEN in Secrets Manager?

I can maybe (not tested yet) harden the setup a little by running the runner agent in a docker container with network isolation, but then we would lose the other benefits of the service account, and was wondering if there is a way around the complexity.

Also filed https://github.com/orgs/community/discussions/48405 for a related discussion.

Thanks.

Right now this module doesn't support Terraform version 0.14, so this dependency needs to be bumped

TL;DR

As mentioned in #123:

This is still an issue today. We stumbled upon this problem with the mig & mag-template dependency and had to downgrade our complete googleProvider to <5.0.0. What's the Point sticking with mig ~>7.0? There were 4 major updates since (11.x.x)

Expected behavior

When using a google Provider I expect to use the newest google provider versions and not being restricted by a sub-sub-subdependency inside of the VM Templating.

Observed behavior

When installing with other Providers and modules there ist a Dependency Hell going on.

Could not retrieve the list of available versions for provider hashicorp/google: no available releases match the given constraints >= 3.43.0, >= 3.50.0, >= 3.53.0, >=
│ 3.64.0, ~> 4.0, >= 4.28.0, < 5.0.0, 5.26.0, < 6.0.0

Terraform Configuration

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 4.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 4.0"
    }
  }
  required_version = ">= 0.13"
}

Terraform Version

Terraform v1.8.3
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v4.84.0 <- would use >5.0
+ provider registry.terraform.io/hashicorp/google-beta v4.84.0 <- would use >5.0
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/random v3.6.2
+ provider registry.terraform.io/hashicorp/time v0.11.1

Additional information

No response

Is GHES(Github Enterprise Server) supported?
Looking inside startup script of gh-runner, it seems like it's not supported.

Is it supported in other modules? Or is there any support plan in near future?

There's a ton of examples already documented and I can't tell you how much I appreciate what's already there, thank you!

That said, I'm looking to connect a self-hosted runner to an entire Github Org, it would be cool if we could use this against an Org.

If I'm able to find an incantation that works I'll gladly submit a PR :D