Code Monkey home page Code Monkey logo

google-github-actions / get-secretmanager-secrets Goto Github PK

View Code? Open in Web Editor NEW
132.0 9.0 26.0 2.28 MB

A GitHub Action for accessing secrets from Google Secret Manager and making them available as outputs.

Home Page: https://cloud.google.com/secret-manager

License: Apache License 2.0

JavaScript 11.25% TypeScript 82.99% Shell 5.76%
github-actions actions gcp google-cloud-platform secrets google-cloud secret-manager google-secret-manager

get-secretmanager-secrets's People

Contributors

averikitsch avatar bharathkkb avatar danbamikiya avatar dependabot[bot] avatar google-github-actions-bot avatar sethvargo avatar verbanicm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

get-secretmanager-secrets's Issues

New release needed

TL;DR

The latest release v0.2.2 is many months old and doesn't support WIF. We would like to have a new release so we can pin the version we use instead of using @main

Support secrets containing json

TL;DR

It would be nice if we could get specific keys from json secrets directly into an output variable.

Detailed design

- id: 'secrets'
  uses: 'google-github-actions/get-secretmanager-secrets@v0'
  with:
    secrets: |-
      token:my-project/docker-registry:token
      username:my-project/docker-registry:username

We're in the process of moving our secrets from Hashicorp Vault to Google Secret Manager. In Vault, we store the secrets as json. If we simply copy over the data to Google Secret Manager, we get our json in the output variable we specify in the plugin, which then needs further processing to get to a specific key.

It would be nice if we could specify the exact key we're interested in in the secrets input.

Accept a string of secret names as input to the action in order to dynamically get an unknown number of secrets.

TL;DR

I want to call the action and pass a string that has comma-separated secret names. That way, I don't know which or how many secrets I'm fetching in advance.

Detailed design

I support a repository for reusable workflows. Different users of my workflows require different secrets from GSM. I can get the names of those secrets as a string input that has them listed as comma-separated values. I would like for a way to then pass that list to your action and get the secrets

E.g.

inputs:
  secrets_to_fetch:
    type: string
    description: "Comma-separated secret names. e.g. 'my_secret,your_secret' " 

Then, when you call the workflow you pass "secret_one,secret_two,secret_three" as a value for "secrets_to_fetch". This should then create three entries under the "secrets" block of the action, get those secrets and store them in their respective names. I suppose it would require you to pass projectID as well which should then translate to something along the lines of:

 - name: Get Secrets from Secret Manager
        id: get-secrets
        uses: google-github-actions/get-secretmanager-secrets@v1
        with:
          project_id: your-project-id
          secrets: ${{ inputs.secrets_to_fetch }}

I tried constructing the list myself and saving it as a variable, but neither $GITHUB_ENV nor $GITHUB_OUTPUT seem to like the '\n' character at the end of every variable. It looked something like

secret_one:<project-id>/secret_one\nsecret_two:<project-id>/secret_two

The only way I could get around this restriction is to use "::set-output name=secret1::value1". But "set-output" is deprecated so I have reservations about using it. Further, it still only seems to fetch a single secret but I'm not sure why. I'm also not sure if I should try and add a bunch of white spaces after the newline character (\n) in order to keep the expected formatting of the workflow files.

Overall, there are many unknowns and I haven't had much success with dynamically getting secrets based on input. Is this possible currently and if not are there plans to add it as a feature?

Additional information

No response

New method to fetch the latest enabled version

TL;DR

At present, we employ the most recent version of the system, irrespective of its enable status. Consequently, if version 2 is disabled and version 1 is enabled, it would result in an error.

Detailed design

No response

Additional information

No response

Specify secrets without project identifier

TL;DR

Is it possible to introduce an even shorter version of specifying the secret? Without the project identifier? The first step in using this action is authenticating using google-github-actions/auth - I figured it should be possible to derive the project identifier from that step?

This would help us in making generic workflow files that we can use in different GCP projects, without having to change all the project identifiers.

Detailed design

No response

Additional information

No response

Possible format error retrieving secret

TL;DR

The secret retrieved by this action seems to be corrupted or different from the one returned by the gcloud secrets versions access command.

Expected behavior

The secret should work

Observed behavior

The secret is not valid

Action YAML

I cannot paste the action because it's a private repository, anyway here are the relevant parts:

Working

- name: Setup deploy variables
id: secrets
run: |
  ACCESS_KEY_ID=$(gcloud secrets versions access 1 --secret=access_key_id)
  SECRET_ACCESS_KEY=$(gcloud secrets versions access 1 --secret=secret_access_key)
  echo "::add-mask::$ACCESS_KEY_ID"
  echo "::add-mask::$SECRET_ACCESS_KEY"
  echo "SECRET_ACCESS_KEY=$SECRET_ACCESS_KEY" >> $GITHUB_OUTPUT
  echo "ACCESS_KEY_ID=$ACCESS_KEY_ID" >> $GITHUB_OUTPUT
- name: Deploy 
  run: |
    # requires SECRET_ACCESS_KEY and ACCESS_KEY_ID
    pnpm run deploy
  env:
    SECRET_ACCESS_KEY: ${{ steps.secrets.outputs.SECRET_ACCESS_KEY }}
    ACCESS_KEY_ID: ${{ steps.secrets.outputs.ACCESS_KEY_ID }}

Not working

- name: Retrieve required env variables
  id: secrets
  uses: google-github-actions/get-secretmanager-secrets@v1
  with:
    secrets: |-
      access_key_id:projects/000000000/secrets/access_key_id/versions/1
      secret_access_key:projects/000000000/secrets/secret_access_key/versions/1
- name: Deploy 
  run: |
    # requires SECRET_ACCESS_KEY and ACCESS_KEY_ID
    pnpm run deploy
  env:
    SECRET_ACCESS_KEY: ${{ steps.secrets.outputs.secret_access_key }}
    ACCESS_KEY_ID: ${{ steps.secrets.outputs.access_key_id }}

Log output

Except for the error in the deploy command (reporting that the something was wrong with the authorizations), the only visible difference is that the env variables set with the output generated by the action are separated by a blank line:

When it works

GCP_PROJECT: XXXXXX
SECRET_ACCESS_KEY: ***
ACCESS_KEY_ID: ***

When it does not work

GCP_PROJECT: XXXXXX
SECRET_ACCESS_KEY: ***

ACCESS_KEY_ID: ***

Additional information

No response

Update to use Node 16

TL;DR

Node 12 is deprecated

Detailed design

  • Cut a release with current features with node12 support
  • Update action.yml to uses: 'node16'
  • Update all tests to use 16 in the matrix
  • Update actions/checkout@v2 -> actions/checkout@v3 everywhere
  • Update README to note node 16 requirement for self-hosted runners (example)
  • Cut a release with node16 support

Additional information

No response

Is there a way to output the secret into a json file?

TL;DR

Is there a way to output the secret into a JSON file to the project root like gcloud beta cli?

gcloud beta secrets versions access --secret=default latest > config.json

Detailed design

I have a firebase project where my secrets are stored as a JSON file in the project root directory.

I would like to use https://github.com/FirebaseExtended/action-hosting-deploy to manage my deployment to firebase hosting and inject the secret as a JSON file into the root during the build process.

Additional information

No response

System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. Parameter ''using: node16' is not supported

TL;DR

Hi all,

after the recent change (3496e55) to switch to Node16 by default,
we are getting:

Error: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. 
(Parameter ''using: node16' is not supported, use 'docker' or 'node12' instead.')
   at GitHub.Runner.Worker.ActionManifestManager.ConvertRuns(IExecutionContext executionContext, TemplateContext templateContext, TemplateToken inputsToken, String fileRelativePath, MappingToken outputs)
   at GitHub.Runner.Worker.ActionManifestManager.Load(IExecutionContext executionContext, String manifestFile)
   
Error: Fail to load google-github-actions/get-secretmanager-secrets/main/action.yml

in our CI pipeline.

Anyone else having this issue?
Any way we can fix it on our side?

Thank you!

Expected behavior

No response

Observed behavior

No response

Action YAML

name: Docker build check
on: [workflow_dispatch, pull_request]
concurrency:
  group: docker-build-${{ github.ref }}
  cancel-in-progress: true
jobs:
  docker-build:
    runs-on: [self-hosted, preemptible]
    env:
      NODE_ENV: test
    steps:
      - name: setup node
        uses: actions/setup-node@v2
        with:
          node-version: 'lts/*'
      - uses: actions/checkout@v2
      - uses: google-github-actions/setup-gcloud@master
        with:
          project_id: ${{ secrets.xxx }}
          service_account_key: ${{ secrets.xxx }}
          export_default_credentials: true
          credentials_file_path: .xxx.json
      - id: secrets
        uses: google-github-actions/get-secretmanager-secrets@main
        with:
          secrets: |-
            CMS_API_TOKEN:xxx/INTEGRATION_XXX
            SHEETS_CMS_API_KEY:xxx/SHEETS_XXX
      - name: write .env
        run: |
          cat << EOF > .env
          BUTTERCMS_API_TOKEN=${{ steps.secrets.outputs.BUTTERCMS_API_TOKEN }}
          GCP_PROJECT_ID=${{ secrets.GCP_PROJECT_ID }}
          SHEETS_TO_CMS_API_KEY=${{ steps.secrets.outputs.SHEETS_TO_CMS_API_KEY }}
          EOF
      - name: docker login
        run: |-
          gcloud --quiet auth configure-docker
      - name: write git-commit.txt
        run: echo "${GITHUB_SHA}" > git-commit.txt
      - name: docker build
        run: |-
          docker build \
            --build-arg GCP_PROJECT_ID=${{ secrets.GCP_PROJECT_ID }} \

Log output

No response

Additional information

No response

Support impersoante service account

TL;DR

I would like to get Google Cloud Secret Manager secrets with an impersonate service account.

Detailed design

    - id: 'secrets'
      uses: 'google-github-actions/get-secretmanager-secrets@v1'
      with:
        impersonate_service_account: "[email protected]"
        secrets: |-
          token:my-project/docker-registry-token

Additional information

No response

Issue with locating problem in step to get secret

TL;DR

Getting secrets from Google Secret Manager by CLI works but not with this GitHub Action app.

Expected behavior

I except the step to return a secret in an output, but the step that is supposed to gets the secret from the Google Secretsmanager is throwing an error instead.

Observed behavior

When using the google-github-actions/get-secretmanager-secrets GitHub Action, I encounter following error:

Error: google-github-actions/get-secretmanager-secrets failed with: Error: Permission 'secretmanager.versions.access' denied for resource 'projects/123456789012/secrets/this-is-my-secret/versions/latest' (or it may not exist).

When running gcloud command directly, the workflow can retrieve the secret in question.

Action YAML

name: my-test-workflow

on:
  workflow_dispatch:

env:
  gcloud_project: secrets-project

jobs:
  test-workflow:
    runs-on: [self-hosted, gke-cluster]

      - name: Checkout
        uses: actions/[email protected]

      - name: Create environment variables
        run: |
          echo gcloud_version=$(/google-cloud-sdk/bin/gcloud --format=json version|jq -r '."Google Cloud SDK"') >> $GITHUB_ENV

      - name: Set up Cloud SDK
        uses: google-github-actions/[email protected]
        with:
          project_id: ${{ env.gcloud_project }}
          version: ${{ env.gcloud_version }}

      - name: Get Google Secretsmanager secret by CLI
        id: "cli-secrets"
        run: |
          echo "::set-output name=my-secret::$(gcloud secrets versions access latest --secret='this-is-my-secret')"

      - name: Get Google Secretsmanager secret by APP
        id: "app-secrets"
        uses: "google-github-actions/[email protected]"
        with:
          secrets: |-
            my-secret:123456789012/this-is-my-secret

      - name: Write the secrets content
        if: always()
        run: |
          echo "${{ steps.cli-secrets.outputs.my-secret }}"
          echo "${{ steps.app-secrets.outputs.my-secret }}"

Additional information

GitHub runners are running in GCP (inside one GKE cluster).
For authentication, I am using Workload Identity Federation.

Authentication is not required, since it's authenticating automatically. One thing I noted:

  • the project_id be default is set to the GCP project the cluster is setup in
  • the project_id of the secret is in another project, but I specify the project_id of the secret when running setup step

Also, the image (alpine based) that we use as our base does actually have gcloud installed already at a certain version (plus terraform and some other). Therefore we do get the same version and set it to the environmental vars.

Turned on ACTIONS_STEP_DEBUG in the repository to try get more information on what happens inside the step, but it doesn't give me any more usable output.

Temporarily Disable Output Masking

Question

Is it possible to temporarily disable the output masking? I am facing an issue where empty secrets are being passed to my K8s workflow and it is impossible to debug it if I can't validate that I am getting the secrets properly in the output.

    - name: Setup Cloud SDK
      uses: google-github-actions/setup-gcloud@master
      with:
        service_account_key: ${{ secrets.GCLOUD_AUTH }}
        export_default_credentials: true

    - name: Get Secrets from GSM
      uses: google-github-actions/get-secretmanager-secrets@main
      with:
        secrets: |-
          ALLOWED_USERS:pokegambler/ALLOWED_USERS/${{ env.SECRET_VERSION }}
          ANNOUNCEMENT_CHANNEL:pokegambler/ANNOUNCEMENT_CHANNEL/${{ env.SECRET_VERSION }}
          DEFAULT_GUILDMODE:pokegambler/DEFAULT_GUILDMODE/${{ env.SECRET_VERSION }}
          DISCORD_WEBHOOK_CHANNEL:pokegambler/DISCORD_WEBHOOK_CHANNEL/${{ env.SECRET_VERSION }}
          DISCORD_WEBHOOK_TOKEN:pokegambler/DISCORD_WEBHOOK_TOKEN/${{ env.SECRET_VERSION }}
          IMG_UPLOAD_CHANNEL:pokegambler/IMG_UPLOAD_CHANNEL/${{ env.SECRET_VERSION }}
          MONGO_CLUSTER_STRING:pokegambler/MONGO_CLUSTER_STRING/${{ env.SECRET_VERSION }}
          OFFICIAL_SERVER:pokegambler/OFFICIAL_SERVER/${{ env.SECRET_VERSION }}
          TOKEN:pokegambler/TOKEN/${{ env.SECRET_VERSION }}
          WHITELIST_GUILDS:pokegambler/WHITELIST_GUILDS/${{ env.SECRET_VERSION }}

    - name: Deploy to GKE
      run: |-
         kubectl delete secret pokegambler-$(echo ${{ env.TAG }}) --ignore-not-found
         kubectl create secret generic pokegambler-$(echo ${{ env.TAG }}) \
           --from-literal=TOKEN=${{ steps.secrets.outputs.TOKEN }} \
           --from-literal=DISCORD_WEBHOOK_CHANNEL=${{ steps.secrets.outputs.DISCORD_WEBHOOK_CHANNEL }} \
           --from-literal=DISCORD_WEBHOOK_TOKEN=${{ steps.secrets.outputs.DISCORD_WEBHOOK_TOKEN }} \
           --from-literal=ANNOUNCEMENT_CHANNEL=${{ steps.secrets.outputs.ANNOUNCEMENT_CHANNEL }} \
           --from-literal=OWNER_ID=${{ steps.secrets.outputs.OWNER_ID }} \
           --from-literal=WHITELIST_GUILDS=${{ steps.secrets.outputs.WHITELIST_GUILDS }} \
           --from-literal=DEFAULT_GUILDMODE=${{ steps.secrets.outputs.DEFAULT_GUILDMODE }} \
           --from-literal=IMG_UPLOAD_CHANNEL=${{ steps.secrets.outputs.IMG_UPLOAD_CHANNEL }} \
           --from-literal=ADMIN_CMD_LOG_CHANNEL=${{ steps.secrets.outputs.ADMIN_CMD_LOG_CHANNEL }} \
           --from-literal=OFFICIAL_SERVER=${{ steps.secrets.outputs.OFFICIAL_SERVER }} \
           --from-literal=MONGO_CLUSTER_STRING=${{ steps.secrets.outputs.MONGO_CLUSTER_STRING }}
         ./kustomize build deployments/${{ env.TAG }} | kubectl apply -f -
         kubectl rollout status deployment/pokegambler-$(echo ${{ env.TAG }})
         kubectl get services -o wide

I am using this to set secrets through Kustomize and all of them are returning as empty strings. Please let me know if I'm doing something wrong.

Also, in case of a successful mask, would it be github's asterisk styled masking?

Update
I've validated that the secrets are stored correctly using the python client locally.

"read ETIMEDOUT" and "socket hang up" errors

TL;DR

It has been about two weeks that I've been getting "read ETIMEDOUT" and "socket hang up" errors more and more frequently while loading one of the many secrets saved via Secret Manager.

google-github-actions/get-secretmanager-secrets failed with: failed to access secret "projects/xxxxxxxxxxx/secrets/xxxxxxxxx/versions/latest": socket hang up
google-github-actions/get-secretmanager-secrets failed with: failed to access secret "projects/xxxxxxxxxxx/secrets/xxxxxxxxx/versions/latest": read ETIMEDOUT

That specific GitHub workflow has remained unchanged for at least 5 months.

Expected behavior

No response

Observed behavior

No response

Action YAML

deploy:
    name: Deploy
    runs-on: ubuntu-20.04
    permissions:
      actions: write
      checks: read
      contents: read
      deployments: read
      id-token: write
      pull-requests: read
      statuses: read
    needs: [build-web, build-api]
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Set VPN secrets
        uses: dominicwatson/github-action-envsubst@v1
        with:
          files: .github/github.ovpn
        env:
          VPN_CA: ${{ secrets.VPN_CA}}
          VPN_CERT: ${{ secrets.VPN_CERT }}
          VPN_KEY: ${{ secrets.VPN_KEY }}
          VPN_TLS: ${{ secrets.VPN_TLS }}

      - name: Connect to VPN
        run: |-
          sudo apt-get -qq update && sudo apt-get -qq install openvpn -y
          sudo openvpn --config .github/github.ovpn --daemon

      - uses: chrnorm/deployment-action@releases/v1
        name: Create GitHub deployment
        id: deployment
        with:
          token: ${{ secrets.PAT }}
          target_url: https://xxxxxxxxxxx
          environment: xxxxxxxxxxx
          ref: master

      - id: "auth"
        name: "Authenticate to Google Cloud"
        uses: "google-github-actions/auth@v0"
        with:
          token_format: "access_token"
          workload_identity_provider: "xxxxxxxxxx"
          audience: "xxxxxxxx"
          service_account: "[email protected]"

      - uses: "google-github-actions/get-gke-credentials@v0"
        name: Get GKE credentials
        with:
          cluster_name: ${{ env.GKE_CLUSTER }}
          location: ${{ env.GKE_ZONE }}

      - name: Get secrets from GCP Secret Manager
        id: secrets
        uses: google-github-actions/[email protected]
        with:
          secrets: |-
            REDIS_URL:projects/xxxxxxxxx/secrets/REDIS_URL
            JWT_SECRET:projects/xxxxxxxxx/secrets/JWT_SECRET
          
      - name: Set URL, secrets and config variables
        uses: dominicwatson/github-action-envsubst@v1
        with:
          files: helm/values.yaml
        env:
          REDIS_URL: ${{ steps.secrets.outputs.REDIS_URL_INTEGRATION }}
          JWT_SECRET: ${{ steps.secrets.outputs.JWT_SECRET }}

Log output

Run google-github-actions/[email protected]
Error: google-github-actions/get-secretmanager-secrets failed with: failed to access secret "projects/xxxxxxxxx/secrets/xxxxxxxxx/versions/latest": socket hang up


Run google-github-actions/[email protected]
Error: google-github-actions/get-secretmanager-secrets failed with: failed to access secret "projects/xxxxxxxxx/secrets/xxxxxxxxx/versions/latest": read ETIMEDOUT

Additional information

No response

Allow to filter masking of secrets

TL;DR

Add an option to filter masking of secrets with a regex.

Detailed design

Getting a multi line JSON secret (a credentials json as an example) masks single characters like { and causes issues in setting outputs in composite actions with this warning : Warning: Skip output 'result' since it may contain secret.

This is due to the fix added in #9

We don't want to disable masking because we still want to mask some part of that multi line JSON (the private key in case of credentials json) but want to skip other parts like the braces {, }, the project_id, etc.

Rather than hardcoding these, I propose we add an optional input mask_filter through which we can skip masking for some lines when going through a multi-line secret line by line.

Additional information

No response

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.