Code Monkey home page Code Monkey logo

drone-gcf's Introduction

Build Status Coverage Status

Drone-GCF - a Drone.io plugin to deploy Google Cloud Functions

Overview

The plugin supports deploying, listing, calling, and deleting of multiple functions at once. See below for an example drone.yml configuration that deploys several functions to GCP Cloud Functions and later deletes two of them. A service account is needed for authentication to GCP and should be provided as json string via drone secrets. In the configuration below, the json of the service account key file is stored in the drone secret called token.

Deploying Cloud Functions

Example .drone.yml file (drone.io 1.0 format):

kind: pipeline
name: default

steps:
  - name: test
    image: golang
    commands:
      - "go test -v"
    when:
      event:
        - push


  - name: deploy-cloud-functions
    image: oliver006/drone-gcf
    settings:
      action: deploy
      project: myproject
      token:
        from_secret: token
      runtime: go111
      env_secret_db_password:
        from_secret: db_password_prod
      env_secret_user_api_key:
        from_secret: user_api_key_prod
      functions:
        - TransferFileToGCS:
          - trigger: http
            memory: 2048MB
            allow_unauthenticated: true
            gen2: true
            environment:
              - ENV_VAR: env_var_value_123
        - HandleEvents:
          - trigger: topic
            trigger_resource: "projects/myproject/topics/mytopic"
            memory: 512MB
        - ProcessEmails:
          - trigger: http
            memory: 512MB
            runtime: python37
            source: ./python/src/functions/
            vpcconnector: vpc-connector
            env_vars_file: ".env.yaml"
        - ProcessSecrets:
            - trigger: http
              runtime: python37
              source: ./python/src/functions/
              secrets:
                /mnt/path: gcpsm_secrets:latest
                TOP_SECRET: gcpsm_top_secret:1

    when:
      event: push
      branch: master


  - name: delete-cloud-functions
    image: oliver006/drone-gcf
    settings:
      action: delete
      functions:
        - TransferFileToGCS
        - HandleEvents
    when:
      event: tag

The plugin supports several types of Google Cloud Function triggers:

  • http - triggered for every request to an HTTP endpoint, no other parameters are needed. (see gcloud output for URL of the endpoint).
  • bucket - triggered for every change in files in a GCS bucket. Supply the name of the bucket via trigger_resource.
  • topic - triggered for every message published to a PubSub topic. Supply the name of the topic via trigger_resource.
  • event - triggered for every event of the type specified via trigger_event of the resource specified via trigger_resource.

See the output of gcloud functions deploy --help for more information regarding the setup of triggers.

When deploying a function, there is the option to deploy as a public function. This can be configured by setting allow_unauthenticated to true. This adds the --allow-unauthenticated flag described here to the deploy command. Please note that this expects a boolean value, either true or false.

By default, the plugin will use the GCP project ID of the service account provided but you can override it by setting the project parameter.

The runtime can be set on a per-function basis or for all functions at once. In the example above, the runtime is set to go111 for all functions and then overwritten with python37 for just ProcessEmails. This will result in the plugin deploying three functions, two in Golang and one in Python.
If no runtime setting is provided at all, the plugin will fail.

By default, the function is deployed without specifying a Generation Version. To use Cloud Functions Second Generation, set gen2 to true on each function. This adds the --gen2 flag as described here to the deploy command. Please note that this expects a boolean value, either true or false.

Similarly, you can set the source location of each function in case you keep the code in separate folders.

There are three ways to set environment variables when deploying cloud functions:

  • from a secret
  • putting the value directly into the drone.yml file
  • Google Secret Manager

To pull in an environment variable value from a secret, add an entry to the settings that starts with env_secret_ followed by the name as which the variable will be made available to the cloud function. In the config example above, drone will pull the values from the secrets db_password_prod and user_api_key_prod and make them available as DB_PASSWORD and USER_API_KEY. (env variable names will be upper-case) \

Environment variables from secrets will be made available to all functions that you deploy within one drone step. If, for whatever reasons, this is not acceptable and you need to keep them separate then you have to use multiple drone steps, one for each function.

If you run into issues when setting environment variables with special characters in their values, there's a setting you can use to specify a delimiter string to be used as separation between variables. Normally, gcloud would use a comma (,), but we've set the default to something more unlikely to cause any issue (:|:). If you still need to change it, you can use the environment_delimiter setting.

Using Google Secret Manager allows two ways to make secret available to the functions.

  • Mounting a secret as a volume, making it available as a file. /mnt/secrets: gcpsm_secret:latest, where the key is the mount point, and the value is the secret name followed by the version.
  • As an environment variable. ENV_NAME: gcpsm_secret:1, where the key is the name of the variable and the value is the secret name followed by the version.

Calling Cloud Functions

You can also trigger a cloud function by using call as the action. Optionally, you can supply a json data string that will be passed to the function.

  - name: call-cloud-function
    image: oliver006/drone-gcf
    settings:
      action: call
      project: myproject
      token:
        from_secret: token
      functions:
        - UpdateDatabase:
          - data: '{"key": "value"}'
    when:
      event: tag

drone-gcf's People

Contributors

anagas avatar aromaniuk-postclick avatar booya avatar hemarc avatar james-mcgoodwin avatar konrness avatar leanazulyoro avatar marinx avatar neil-berg avatar nesth avatar oliver006 avatar raybb avatar sduncan-nyt avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

drone-gcf's Issues

Plugin doesn't recognize functions

Hello all, I'm trying to use this plugin, but I do not seam to get it correctly configured.
This is my .drone.yml:

---
kind: pipeline
name: deploy-staging

steps:
  - name: restore_cache
    image: plugins/s3-cache
    settings:
      pull: true
      root: drone_cache
      endpoint: https://storage.googleapis.com
      access_key:
        from_secret: gcs_access_key
      secret_key:
        from_secret: gcs_secret_key
      restore: true

  - name: npm_install
    image: node:12-alpine
    commands:
      - npm install
    depends_on:
      - restore_cache

  - name: rebuild_cache
    image: plugins/s3-cache
    settings:
      pull: true
      root: drone_cache
      endpoint: https://storage.googleapis.com
      access_key:
        from_secret: gcs_access_key
      secret_key:
        from_secret: gcs_secret_key
      rebuild: true
      mount:
        - node_modules
    depends_on:
      - npm_install

  - name: npm_run_build
    image: node:12-alpine
    commands:
      - npm run build
    depends_on:
      - npm_install

  - name: deploy_cf
    depends_on:
      - npm_run_build
    image: oliver006/drone-gcf
    settings:
      action: deploy
      project: my-project-staging
      token:
        from_secret: token
      runtime: nodejs10
      env_secret_sql_user:
        from_secret: sql_user_stg
      env_secret_sql_password:
        from_secret: sql_password_stg
      env_secret_instance_connection_name:
        from_secret: instance_connection_name_stg
      env_secret_firestore_project_id:
        from_secret: firestore_project_id_stg
      env_secret_firestore_collection_name:
        from_secret: firestore_collection_name_stg
      functions:
        - MyFunctionName:
          - trigger: http
            memory: 2048MB
            runtime: nodejs10
            region: us-east1
            allow_unauthenticated: true
            environment:
              - LOGGING: true
                NODE_ENV: production
trigger:
  branch:
    - develop
  event:
    - push

When I run this pipeline, the "deploy_cf" fails with this:

2020/07/14 19:34:27 Drone-GCF Plugin  version: [not-tagged]   hash:    date: 2020-07-10-16:51:55
2020/07/14 19:34:27 Missing or invalid runtime [] for function: [{"MyFunctionName":[{"allow_unauthenticated":true
2020/07/14 19:34:27 Missing or invalid runtime [] for function: "environment":[{"LOGGING":true
2020/07/14 19:34:27 Missing or invalid runtime [] for function: "NODE_ENV":"production"}]
2020/07/14 19:34:27 Missing or invalid runtime [] for function: "memory":"2048MB"
2020/07/14 19:34:27 Missing or invalid runtime [] for function: "region":"us-east1"
2020/07/14 19:34:27 Missing or invalid runtime [] for function: "runtime":"nodejs10"
2020/07/14 19:34:27 Missing or invalid runtime [] for function: "trigger":"http"}]}]
2020/07/14 19:34:27 parseConfig() err: Didn't find any functions

It seams to mess up when parsing the yaml? It's not recognizing the "runtime" field, and doesn't even seam to recognize that there are any functions.
I'm sorry if this is a dumb syntax error from my part, but I don't see it.

How to set runtime service account?

Hi @oliver006 ,

Thanks for providing this plugin!
I wanted to know how to set the runtime service account for a cloud function that I deployed using your plugin?

`kind: pipeline
name: default

steps:

  • name: deploy-cloud-functions-dev
    image: oliver006/drone-gcf
    settings:
    action: deploy
    project: xxxxxx
    runtime: python38
    token:
    from_secret: gcf-credentials-dev

    functions:
    - dbt_run_http:
    - trigger: http
    memory: 512MB
    runtime: python38
    source: ./
    entry_point: dbt_run_http`

The service account for the token specified here is not the one that shows up in the Google Cloud Function UI. It routes to the default service account for runtime. Is there any way to specify here the service account it uses while running?

Thanks in advance!

[Feature request] env-vars

Hi, I'd like to provide PR directly but I'm a total golang newbie so I probably won't succeed. It would be nice to have support for Function's environment variables though, are you planning on doing that?

Add option to set delimeter for environment variables

Hi, me again, I have this case in which I'm attempting to set an env variable in the CF which contains comma (",") characters.
This is what comes up in the log:

2020/07/15 20:11:49 Drone-GCF Plugin  version: [not-tagged]   hash:    date: 2020-07-10-16:51:55
2020/07/15 20:11:49 Using project ID: xxxxxxxxxxx
Google Cloud SDK 290.0.1
alpha 2020.04.24
beta 2020.04.24
bq 2.0.56
core 2020.04.24
gsutil 4.49
kubectl 2020.04.24
Activated service account credentials for: [[email protected]]
ERROR: (gcloud.functions.deploy) argument --set-env-vars: Bad syntax for dict arg: [XNuWskz]. Please see `gcloud topic flags-file` or `gcloud topic escaping` for information on providing list or dictionary flag values with special characters.
Usage: gcloud functions deploy (NAME : --region=REGION) [optional flags]
  optional flags may be  --allow-unauthenticated | --clear-env-vars |
                         --clear-labels | --clear-max-instances |
                         --clear-vpc-connector | --egress-settings |
                         --entry-point | --env-vars-file | --help |
                         --ignore-file | --ingress-settings | --max-instances |
                         --memory | --region | --remove-env-vars |
                         --remove-labels | --retry | --runtime |
                         --service-account | --set-env-vars | --source |
                         --stage-bucket | --timeout | --trigger-bucket |
                         --trigger-event | --trigger-http | --trigger-resource |
                         --trigger-topic | --update-env-vars | --update-labels |
                         --vpc-connector

For detailed information on this command and its flags, run:
  gcloud functions deploy --help
2020/07/15 20:11:50 runConfig() err: error: exit status 2

if you go and check those gcloud topic flags-file or gcloud topic escaping, you'll find two posibilities to allow the use of commas and other special characters in the values for a "dictionary flag", which happens to be the case for ---set-env-vars.
To sum up, what needs to happen is that when the command is building it's flags, it should allow to build it like this:
--set-env-vars=^DELIM^, maybe pass that DELIM in a environment_delimiter setting?

That would allow us to have commas (and any other character) in the values for our environment variables. like so:

- name: deploy-cloud-functions
    image: oliver006/drone-gcf
    settings:
      action: deploy
      project: xxxxxx
      token:
        from_secret: token
      functions:
        - MyFunction:
          - trigger: http
            runtime: nodejs10
            memory: 2048MB
            environment_delimiter: ";"
            environment:
              - SOME_ENV: "some,value,with,commas"
                SOME_OTHER_ENV: "someValueWithoutCommas"

which should result in this flag:
--set-env-vars=^;^SOME_ENV=some,value,with,commas;SOME_OTHER_ENV=someValueWithoutCommas

Receiving Error: 'Missing action' when action is set to deploy

Hi @oliver006 ,
I am trying to use this plugin to deploy cloud functions in drone, however I keep receiving the following error in drone:
parseConfig() err: Missing action even though action is set. Please see my .drone.yml file below:

  deploy_cf_stg:
    image: oliver006/drone-gcf
    settings:
      action: deploy
      project: nyt-dssor-stg
      token:
        from_secret: gcf_credentials_stg
      runtime: python37
      functions:
        - status_check:
            - trigger: http
              source: cloud_functions
              entry_point: run_status_check
              function_name: status_check
              runtime: python37
              verbosity: info

Thanks in advance for your help!

[feature request] support the --allow-unauthenticated flag

Hello,

I would like to use the --allow-unauthenticated flag that is available in the CLI (docs here).

If it were like the other flags currently support, which require a key and value, I would know what needs to change.

However, this flag doesn't take in any value. As such, I'm not sure what the best way to approach this is (I'm not terribly well versed in go).

From what I gather, there are 2 options:

  1. Add support forallow-unauthenticated as a setting
  2. Add support for arbitrary key/value pair to be used as a setting (perhaps a bit more future-proof?)
  • Note: there are other unsupported flags such as --clear-max-instances and --clear-vpc-connector

In either case, that comes to the question of how to support flags that don't accept any value.

These two options are relatively frictionless:

  1. only accept the input of "true"
  2. have a setting called inputless_args (or preferably something better) that takes in an array of strings that will be added as flags.
1     functions:
        - myAwesomeFunction:
            - trigger: http
              allow-unauthenticated: true
              clear-max-instances: true
2     functions:
        - myAwesomeFunction:
            - trigger: http
              inputless_args:
                - [allow-unauthenticated, clear-max-instances]

Let me know your thoughts!

Thanks so much for sharing project with the world!

[bug] deploying with multiple environment variables doesn't work

 - name: deploy-cloud-func
    image: oliver006/drone-gcf:v1.6.0
    settings:
      action: deploy
      project: my-cloud-function-testing
      token:
        from_secret: google_credentials_dev
      runtime: nodejs10
      functions:
        - slack:
            - trigger: http
              environment:
                - firstkey: firstvalue
                - secondkey: secondvalue

When I use the above, only the first environment variable is deployed.
I believe it's because of these lines.

According to the docs, the flag should be used as --set-env-vars=[KEY=VALUE,โ€ฆ] but it seems we are using :|: as a separator.

It seems like it might be as simple as switching out the separator but maybe there's a reason that separator was being used in the first place?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.