The purpose of this repo is to demonstrate the setup and architecture needed to send GitHub webhooks to AWS Lambda functions for processing.
In this example, the goal is for Lambda to perform the following any time a new repository is created in a GitHub organization:
- Create a default branch, if non exists
- Commit a placeholder
README.md
to the branch
- Commit a placeholder
- Enforce code reviews (CR) on the default branch
- Require a pull request (PR) before merging
- Require 1 approval on any PR
- Create an issue in the repo noting what was enabled
GitHub Webhooks were chosen based on the following features:
- Rapid and easy integration with existing organizations and repositories
- Event-based triggers when repositories are created
Serverless GitHub Actions looks promising as well, but as of this commit is not available yet. When that becomes available I will likely refactor this example accordingly.
The AWS Cloud Development Kit (CDK) lets you build and configure resources in the AWS Cloud using expressive, imperative programming concepts. Unlike declarative Infrastructure as Code, the AWS CDK lets you add considerable flexibility to your application stacks.
To ensure easy deployment and maintenance of this solution, all AWS components have been built into a single CDK app that can be deployed using several command-line interface (CLI) commands.
In this example, the CDK was chosen because of the following advantages:
- Easy to pick up and use for teams with programming knowledge
- Does not require any infrastructure as code experience
- Stores all infrastructure configuration as code in version control
- Reusable across multiple GitHub organizations, customers, regions, etc.
GitHub webhooks can be configured to respond to specific events (such as creation of a repository in an organization). When such an event occurs, the details of the event are sent to a specified destination (URL). The destination must be able to receive the event and perform some processing based on the inputs.
In this solution, Amazon API Gateway acts as the destination for webhook events. Within the supplied CDK app, a REST API is configured to receive the webhooks and proxy them to AWS Lambda for processing. This solution was chosen due to the limited administrative burden the service places on users. Setup is one-time and highly scalable.
Another option being considered is Amazon EventBridge, but from some cursory research it doesn't appear to support GitHub webhooks directly (yet). It would still require an API endpoint.
AWS Lambda is used as the backend processing that will enforce the needed security rules on newly-created repositories. When a GitHub webhook is sent to the REST API endpoint, the event data is then proxied to a Lambda function. Functions are effectively containers to run arbitrary code in response to events, without needing to manage the underlying hardware.
This solution was chosen due to the low cost and significantly reduced maintenance compared to managing servers (virtual or otherwise).
- An active AWS account
- If you do not have one, follow these instructions to create an activate an account
- The AWS CLI installed on your workstation
- Follow these instructions to install and configure the AWS CLI
- The AWS CDK installed on your workstation
- Follow these instructions to install and configure the AWS CDK
- Docker installed on your workstation
- Follow these instructions to install and configure Docker Desktop
A personal access token (PAT) is used to replace a password when making requests to GitHub programmatically.
- Note: It is highly recommended to do this using a service account. Creating tokens for personal user accounts to configure automation can cause problems over time.
A webhook secret can be used to help secure your webhooks and the endpoints that receive them. This is used by GitHub to calculate a signature for the request, which you can compare within your endpoint's code or authentication mechanisms.
- Follow these instructions to grant an access token.
- Save the token value someplace safe.
- Make up a password to be used as a webhook secret and store it someplace safe.
AWS Secrets Manager is used to store secrets or other sensitive information that needs to be used by applications and systems. Using the Boto3 library, the Lambda function that processes the webhook event will query Secrets Manager for the PAT and webhook secret. This removes the need to store these values within your codebase.
-
Follow these instructions to sign in to the AWS Management Console.
-
In the search bar at the top of the console, enter
Secrets Manager
First, create the secret that will hold the personal access token.
-
Select Secrets Manager from the results of the search
-
Select Store a new secret
-
Under Secret type, select Other type of secret
-
Under Key/value pairs
- In the left input field, enter
github_token
- In the right input field, enter the GitHub Personal Access Token you created previously
- In the left input field, enter
-
Select Next
-
In the Secret name input field, enter
github_token
-
Select Next
-
Select Next
-
Select Store
Next, create the secret that will hold the webhook secret.
-
Select Secrets Manager from the results of the search
-
Select Store a new secret
-
Under Secret type, select Other type of secret
-
Under Key/value pairs
- In the left input field, enter
webhook_secret
- In the right input field, enter the webhook secret you created previously
- In the left input field, enter
-
Select Next
-
In the Secret name input field, enter
webhook_secret
-
Select Next
-
Select Next
-
Select Store
Lastly, you will need to copy the Amazon Resource Name (ARN) values for both secrets. These will be used by Lambda to get the secret values later.
-
Return to the Secrets Manager console
-
Open
webhook_secret
in a new tab -
Copy the value of Secret ARN
-
In the Secrets Manager console, open
github_token
in a new tab -
Copy the value of Secret ARN
To prepare the CDK app for deployment into your AWS account, the two secret ARNs you copied previously need to be added to the app configuration. This can be done quickly by updating two string variables.
- Clone this repository to your workstation
- Open ./webhook-cdk/app.py
- On Line 9, set the value of
github_token_arn
to the ARN of thegithub_token
secret you created - On Line 13, set the value of
github_webhook_secret_arn
to the ARN of thewebhook_secret
secret you created - Save your changes
In this step, you will deploy your CDK app to your AWS account.
-
Run the following command from the command line or terminal from within your repository (make sure to change the outputs path to a path on your workstation)
cdk deploy --outputs-file /path/to/cdk-outputs.json
-
Confirm that the CDK will create the listed resources
-
Once the command completes, open
cdk-outputs.json
file -
Make note of the
PayloadURL
output
- Navigate to GitHub.com
- Open your organization
- Select the Settings tab
- Under Code, planning, and automation, select Webhooks
- Select Add webhook
- In the Payload URL input field, enter the
PayloadURL
value you copied fromcdk-outputs.json
- In the Content type drop-down menu, select
application/json
- In the Secret input field, enter the webhook secret you created previously
- Under Which events would you like to trigger this webhook?, select Let me select individual events.
- In the events:
- Deselect Pushes
- Select Repositories
- Select Add webhook
As a final step, test creating a repository in your organization. Shortly after doing so, the following should occur.
- An initial default branch is created with the placeholder
README.md
file - Protection is enabled for the default branch
- An issue is created outlining the protections that have been configured
This project makes use of the following dependencies: