A simple "hello world" reference app for the [FormidableLabs/serverless/aws][FormidableLabs/serverless/aws] Terraform module, using the [serverless][] framework targeting an AWS Lambda deploy.

• Overview
  • Audience
  • Stack
  • Naming
  • Stages
  • Environment Variables
  • User Roles
• Installation
  • Node.js
  • AWS (Deployment)
    • AWS Tools
    • Terraform
    • AWS Credentials
      • In Environment
      • Saved to Local Disk
      • AWS Vault
• Support Stack Provisioning (Superuser)
  • Bootstrap Stack
  • Service Stack
• Serverless Deployment via generated IAM Roles

Getting a serverless application into the cloud "the right way" can be a challenge. To this end, we start with a super-simple, "hello world" Express app targeting AWS Lambda using serverless. Along the way, this reference project takes care of all of the tough supporting pieces that go into a production-ready, best-practices-following cloud infrastructure like:

• Local development workflows.
• Terraform stack controlling IAM permissions and cloud resources to support a vanilla serverless application.
• Remote state management for Terraform.
• Serverless application deployment and production lifecycle management.

Using this project as a template, you can hopefully take a new serverless application and set up "everything else" to support it in AWS the right way, from the start.

A great starting point is our introduction blog post that explains existing privilege approaches, motivations for this project, and some quick example infrastructures.

This reference application is meant for developers / architects who are already familiar with AWS infrastructures (and CloudFormation), Terraform, and Serverless framework applications. This project will hopefully provide some guidance / examples to get the whole shebang all the way to a multi-environment deployment and support a team of administrators and engineers for the application.

For folks (particularly Formidables) interested in learning more about how we construct our production cloud infrastructures, head over to our learning page.

We use very simple, very common tools to allow a mostly vanilla Express server to run in localdev / Docker like a normal Node.js HTTP server and also as a Lambda function exposed via API Gateway.

Tech stack:

• express: A server.

Infrastructure stack:

• [serverless][]: Build / deployment framework for getting code to Lambda.
• [serverless-http][]: Bridge to make a vanilla Express server run on Lambda.

Infrastructure tools:

• AWS [CloudFormation][]: Create AWS cloud resources using YAML. The serverless framework creates a CloudFormation stack of Lambda-supporting resources as part of a normal deployment. This project also uses a small CloudFormation stack to bootstrap an S3 bucket and DynamoDB to handle Terraform state.
• HashiCorp [Terraform][]: Create AWS cloud resources using [HCL][]. Typically more flexible and expressive than CloudFormation. We have a simple Terraform stack that uses the [FormidableLabs/serverless/aws][FormidableLabs/serverless/aws] module to set up a production-ready set of resources (IAM, monitoring, etc.) to support the resources/stack generated by serverless.

We use a naming convention in cloud resources and yarn tasks to separate some various high level things:

• cf: AWS CloudFormation specific names.
• tf: Terraform specific names.
• sls: Serverless framework names.

Development hits a local machine, and when programmatically named, is usually referred to as:

• localdev: A development-only setup running on a local machine.

We target four different stages/environments of AWS hosted deployments:

• sandbox: A loose environment where developers can manually push / check things / break things with impunity. Typically deployed from developer laptops.
• development: Tracks feature development branches. Typically deployed by CI on merges to develop branch if using git flow workflow.
• staging: A near-production environment to validate changes before committing to actual production. Typically deployed by CI for release candidate branches before merging to master.
• production: The real deal. Typically deployed by CI after a merge to master.

Note that these are completely arbitrary groups, both in composition and naming. There a sensible set of groups if you need just some starting point. But the final group (or even one if you want) is totally up to you!

All of our yarn run <task> tasks should be run with a STAGE=<value> prefix. The default is to assume STAGE=localdev and only commands like yarn run node:localdev or yarn run lambda:localdev can run without specification successfully. For commands actually targeting AWS, please prefix like:

$ STAGE=sandbox yarn run <task>
$ STAGE=development yarn run <task>
$ STAGE=stage yarn run <task>
$ STAGE=production yarn run <task>

Note: We separate the STAGE variable from NODE_ENV because often there are build implications of NODE_ENV that are distinct from our notion of deploy target environments.

Our task runner scheme is a bash + yarn based system crafted around the following environment variables (with defaults):

• STAGE: localdev
• SERVICE_NAME: simple-reference (The name of the application/service in the cloud.)
• AWS_REGION: us-east-1

... and some minor localdev only ones:

• AWS_XRAY_CONTEXT_MISSING: LOG_ERROR (Have Xray not error in localdev)
• SERVER_PORT: 3000
• SERVER_HOST: 0.0.0.0

... and some implied ones:

• FUNCTION_NAME: The name of a given Lambda function. In this project, the main one is server.

If your project supports Windows, you will want to have a more general / permissive approach.

We rely on IAM roles to limit privileges to the minimum necessary to provision, update, and deploy the service. Typically this involves creating personalized users in the AWS console, and then assigning them groups for varying appropriate degrees of privilege. Here are the relevant ones for this reference project:

• Superuser - Support Stack: A privileged user that can create the initial bootstrap CloudFormation stack and Terraform service module that will support a Serverless application. It should not be used for Serverless deploys.
• IAM Groups - Serverless App: The [FormidableLabs/serverless/aws][FormidableLabs/serverless/aws] module provides IAM groups and support for different types of users to create/update/delete the Serverless application. The IAM groups created are:
  • tf-${SERVICE_NAME}-${STAGE}-admin: Can create/delete/update the Severless app.
  • tf-${SERVICE_NAME}-${STAGE}-developer: Can deploy the Severless app.
  • tf-${SERVICE_NAME}-${STAGE}-ci: Can deploy the Severless app.

ℹ️ Note: Our cloud infrastructure is based on an approach of a single shared AWS account (with many limited IAM users). A more secure and differently complex option is to use separate AWS accounts for different stages/environments for infrastructures/applications. We discuss these approaches more in our introductory blog post for the FormidableLabs/serverless/aws Terraform module.

In practice, many real world projects will segregate at least the ultimate production infrastructure to a separate AWS account and potentially utilize multiple infrastructures within a shared non-production AWS account. There are many ways to implement a robust production privilege approach, and this reference project implements just one of them!

First, modify the service name, and default aws region to suit your needs, in package.json env line.

First, make sure you have our version of node (determined by .nvmrc, you will need to have nvm installed):

$ nvm use

Then, yarn install the Node.js dependencies:

$ yarn install

Certain administrative / development work require the AWS CLI tools to prepare and deploy our staging / production services. To get those either do:

# Install via Python
$ sudo pip install awscli --ignore-installed six

# Or brew
$ brew install awscli

After this you should be able to type:

$ aws --version

Install [tfenv][] from Homebrew: brew install tfenv. Then, in the root of the repo, run tfenv install to download and use the pinned version of Terraform for this project.

Note that tfenv conflicts with Homebrew terraform and must be uninstalled first. You can still use tfenv to install and use the latest Terraform version in projects that don't have a .terraform-version file.

To work with this reference app, you need AWS credentials for your specific user (aka, FIRST.LAST). To create the bootstrap and service support stacks, that user will need to be a superuser. To deploy serverless applications, the user will need to be attached to given tf-${SERVICE_NAME}-${STAGE}-(admin|developer) IAM groups after the service stack is created.

Once you have a user + access + secret keys, you need to make them available to commands requiring them. There are a couple of options:

You can append the following two environment variables to any command like:

$ AWS_ACCESS_KEY_ID=INSERT \
  AWS_SECRET_ACCESS_KEY=INSERT \
  STAGE=sandbox \
  yarn run lambda:info

This has the advantage of not storing secrets on disk. The disadvantage is needing to keep the secrets around to paste and/or export into every new terminal.

Another option is to store the secrets on disk. You can configure your ~/.aws credentials like:

$ mkdir -p ~/.aws
$ touch ~/.aws/credentials

Then add a default entry if you only anticipate working on this one project or a named profile entry of your username (aka, FIRST.LAST):

$ vim ~/.aws/credentials
[default|FIRST.LAST]
aws_access_key_id = INSERT
aws_secret_access_key = INSERT

If you are using a named profile, then export it into the environment in any terminal you are working in:

$ export AWS_PROFILE="FIRST.LAST"
$ STAGE=sandbox yarn run lambda:info

Or, you can declare the variable inline:

$ AWS_PROFILE="FIRST.LAST"\
  STAGE=sandbox \
  yarn run lambda:info

The most secure mix of the two above options is to install and use aws-vault. Once you've followed the installation instructions, you can set up and use a profile like:

# Store AWS credentials for a profile named "FIRST.LAST"
$ aws-vault add FIRST.LAST
Enter Access Key Id: INSERT
Enter Secret Key: INSERT

# Execute a command with temporary creds
$ STAGE=sandbox aws-vault exec FIRST.LAST -- yarn run lambda:info

⚠️ Warning: Certain IAM role creation commands do not work with the default aws-vault setup if you have MFA set up (which you should).

The following commands that definitely need extra command support:

• yarn tf:service:apply
• yarn tf:service:_delete
• yarn lambda:deploy

We have a research ticket to better handle sessions with MFA, but in the meantime you can simply add the --no-session flag to any aws-vault commands that need it. E.g.

$ STAGE=sandbox aws-vault exec FIRST.LAST --no-session -- <ACTUAL_COMMAND>

# E.g.
$ STAGE=sandbox aws-vault exec FIRST.LAST --no-session -- STAGE=sandbox yarn tf:service:apply

In practice, it is probably easier in the meantime to just always add the --no-session flag when using aws-vault exec.

This section discusses getting AWS resources provisioned to support Terraform and then Serverless.

The basic overview is:

1. Bootstrap Stack: Use AWS CloudFormation to provision resources to manage Terraform state.
2. Service Stack: Use Terraform to provision resources / permissions to accompany a Serverless deploy.

after this, then we are ready to deploy a standard serverless application with full support!

This step creates an S3 bucket and DynamoDB data store to enable Terraform to remotely manage it's state. We do this via AWS CloudFormation.

All commands in this section should be run by an AWS superuser. The configuration for all of this section is controlled by: aws/bootstrap.yml. Commands and resources created are all prefixed with cf as a project-specific choice for ease of identification in the AWS console (vs. Terraform vs. Serverless-generated).

Create the CloudFormation stack:

# Provision stack.
$ STAGE=sandbox yarn run cf:bootstrap:create
{
    "StackId": "arn:aws:cloudformation:${AWS_REGION}:${AWS_ACCOUNT}:stack/cf-${SERVICE_NAME}-${STAGE}-bootstrap/HASH"
}

# Check status until reach `CREATE_COMPLETE`
$ STAGE=sandbox yarn run cf:bootstrap:status
"CREATE_COMPLETE"

Once this is complete, you can move on to provisioning the service stack section. The remaining commands below are only if you need to update / delete the bootstrap stack, which shouldn't happen that often.

Update the CloudFormation stack:

# Update, then check status.
$ STAGE=sandbox yarn run cf:bootstrap:update
$ STAGE=sandbox yarn run cf:bootstrap:status

Delete the CloudFormation stack:

The bootstrap stack should only be deleted after you have removed all of the -admin|-developer|-ci groups from users and deleted the Serverless and Terraform service stacks.

# **WARNING**: Use with extreme caution!!!
$ STAGE=sandbox yarn run cf:bootstrap:_delete

# Check status. (A status or error with `does not exist` when done).
$ STAGE=sandbox yarn run cf:bootstrap:status
An error occurred (ValidationError) when calling the DescribeStacks operation: Stack with id cf-SERVICE_NAME-STAGE does not exist

This step provisions a Terraform stack to provide us with IAM groups and other AWS resources to support and enhance a Serverless provision (in the next section).

All commands in this section should be run by an AWS superuser. The configuration for all of this section is controlled by: terraform/main.tf. Commands and resources created are all prefixed with tf as a project-specific choice for ease of identification.

ℹ️ Note: We use the terraform CLI program under the hood directly for all of our Terraform work. This is simple and good for learning, but in a real world infrastructure has several limitations (such as the pain of remembering to re-init environments on switching, etc.). Consequently, if you're looking to maintain multiple environments with Terraform in the real world, consider more flexible meta tools like terragrunt.

Init your local Terraform state.

This needs to be run once to be able to run any other Terraform commands.

$ STAGE=sandbox yarn run tf:service:init --reconfigure

⚠️ Warning: You need to run yarn run tf:service:init every time you change STAGE or other core environmental setup before you can mutate anything with the stack (like yarn run tf:service:apply). Failure to do so will result in bad things like incorrect stage variables applied to an old, stale stage in the underlying Terraform local disk cache.

ℹ️ Note: We suggest using the --reconfigure flag every time you run init when switching environments so that the remote state (in S3) remains the source of truth and accidental stuff you do on local disk doesn't end up corrupting things.

Plan the Terraform stack.

Terraform allows you to see what's going to happen / change in your cloud infrastructure before actually committing to it, so it is always a good idea to run a plan before any Terraform mutating command.

$ STAGE=sandbox yarn run tf:service:plan

Apply the Terraform stack:

This creates / updates as appropriate.

# Type in `yes` to go forward
$ STAGE=sandbox yarn run tf:service:apply

# YOLO: run without checking first
$ STAGE=sandbox yarn run tf:service:apply -auto-approve

# **WARNING**: If using `aws-vault`, remember `--no-session`!
$ STAGE=sandbox aws-vault exec FIRST.LAST --no-session -- STAGE=sandbox yarn tf:service:apply

Delete the Terraform stack:

The service stack should only be deleted after you have removed all of the -admin|-developer|-ci groups from users and deleted the Serverless stack.

# **WARNING**: Use with extreme caution!!!
# Type in `yes` to go forward
$ STAGE=sandbox yarn run tf:service:_delete

# YOLO: run without checking first
$ STAGE=sandbox yarn run tf:service:_delete -auto-approve

# **WARNING**: If using `aws-vault`, remember `--no-session`!
$ STAGE=sandbox aws-vault exec FIRST.LAST --no-session -- STAGE=sandbox yarn tf:service:_delete

Visualize the Terraform stack:

These are Mac-based instructions, but analogous steps are available on other platforms. First, you'll need GraphViz for the dot tool:

$ brew install graphviz

From there, you can visualize with:

# Generate SVG
$ STAGE=sandbox yarn run -s tf:terraform graph | dot -Tsvg > ~/Desktop/infrastructure.svg

In previous section, terraform provisions IAM groups to whom users should be attached to deploy severless applications tf-${SERVICE_NAME}-${STAGE}-(admin|developer|ci).

The configuration is controlled by: serverless.yml

⚠️ Prod/Real World Warning: This reference application deploys from local laptops for ease of instruction. However, our laptops are usually a different operating system than the target Lambda Linux execution environment. This is an issue for binary dependencies in node_modules which are OS-specific and zipped up and shipped with the Lambda application.

Our reference application presently does not have binary dependencies, but as a best practice for a real world Lambda application, you should not package and deploy from a different OS than your target Lambda execution environment. This means if locally deploying using an appropriate Docker setup for packaging, or using a CI/CD system that matches the Lambda OS to package and deploy the application.

This section will be detailed in more details in dedicated repository.