Lightweight AWS CloudFormation Stack, Template and Parameter Manager and Preprocessor

Author:

• Fabrizio Branca

Contributors:

• Lee Saferite
• Julian Kleinhans

Install composer first, then:

composer require aoepeople/stackformation

Grab the latest release from https://github.com/AOEpeople/StackFormation/releases/latest or use this shortcut (requires jq to be installed)

wget $(curl -s https://api.github.com/repos/AOEpeople/StackFormation/releases/latest | jq -r '.assets[0].browser_download_url')

If you want to use stackformation globally:

mv stackformation.phar /usr/local/bin/stackformation
chmod +x /usr/local/bin/stackformation

Create a .env.default file (and add it yo your gitignore: echo .env.default >> .gitignore)

AWS_ACCESS_KEY_ID=INSERT_YOUR_ACCESS_KEY_HERE
AWS_SECRET_ACCESS_KEY=INSERT_YOUR_SECRET_KEY_HERE
AWS_DEFAULT_REGION=INSERT_YOUR_DEFAULT_REGION_HERE

Create a blueprints.yml in your current directory:

blueprints:
  - stackname: my-stack
    template: my-stack.template

Create you CloudFormation template my-stack.template:

{
  "Resources": { 
    "MyResource1": { "Type": "AWS::CloudFormation::WaitConditionHandle" }
  }
}

Deploy your stack:

bin/stackformation.php deploy my-stack

Add parameters in your my-stack.template:

{
  "Parameters: {
    "MyParameter1": { "Type": "String" }
  },
  "Resources": { 
    "MyResource1": { "Type": "AWS::CloudFormation::WaitConditionHandle" }
  }
}

...and configure that parameter in the blueprint.yml file:

blueprints:
  - stackname: my-stack
    template: my-stack.template
    parameters:
      MyParameter1: 'Hello World'

TODO

TODO

Structure your blueprints including all templates and other files (e.g. userdata) in "modules". StackFormation will load all stack.yml files from following locations:

• blueprints/*/*/*/blueprints.yml
• blueprints/*/*/blueprints.yml
• blueprints/*/blueprints.yml
• blueprints/blueprints.yml
• blueprints.yml

So it's suggested to create a directory structure like this one:

blueprints/
  stack1/
    userdata/
      provisioning.sh
    blueprints.yml
    my.template
  stack2/
    blueprints.yml
  ...

All blueprints.yml files will be merged together.

To prevent stack resources from being unintentionally updated or deleted during a stack update you can use stack policies. Stack policies apply only during stack updates and should be used only as a fail-safe mechanism to prevent accidental updates to certain stack resources.

It's suggested to create a stack_policies directory below the corresponding stack directory:

blueprints/
  stack1/
    stack_policies/
    blueprints.yml
    ...
  stack2/
    stack_policies/
    blueprints.yml
    ...
  ...

You have to tell StackFormation where it could find the stack policy.

Example:

blueprints:
  - stackname: 'my-stack'
    template: 'templates/my-stack.template'
    stackPolicy: 'stack_policies/my-stack.json'

You can pull in StackFormation modules via composer. Look at the cfn-lambdahelper for an example. A custom composer installer (configured as require dependency) will take care of putting all the module files in your blueprints/ directory. This way you can have project specific and generic modules next to each other.

Please note that a "StackFormation module" will probably not come with a blueprints.yml file since this (and especially the stack parameter configuration) is project specific.

You will need to create the stack configuration for the parts you want to use. A good place would be blueprints/blueprints.yml where you reference the imported module.

Example:

blueprints:
  - stackname: 'lambdacfnhelpers-stack'
    template: 'cfn-lambdahelper/lambda_cfn_helpers.template'
    Capabilities: CAPABILITY_IAM

• Output lookup: {output:<stack>:<output>} -> output value
• Resource lookup: {resource:<stack>:<logicalResource>} -> physical Id of that resource
• Parameter lookup: {parameter:<stack>:<logicalResource>} -> parameter value (note that some parameters will not be shown if they're 'no_echo')
• Environment variable lookup: {env:<var>} -> value of environment variable 'var'
• Environment variable lookup with default value fallback: {env:<var>:<defaultValue>} -> value of environment variable 'var' falling back to 'defaultValue' if env var is not set
• Stack/global variable lookup: {var:<var>} -> value variable 'var'
• Current timestamp: {tstamp} -> e.g. '1453151115'
• MD5 sum: {md5:<filename>} -> e.g. 'fdd747e9989440289dcfb476c75b4268'
• Clean: {clean:2.1.7} -> '217' (removes all characters that aren't allowed in stack names
• Switch profile: [profile:<profileName>:...] will switch to a different profile and evaluate the second parameter there. This is useful in cross account setups.

Output and resource lookup allow you to "connect" stacks to each other by wiring the output or resources created in one stack to the input parameters needed in another stack that sits on top of the first one without manually managing the input values.

Example

blueprints:
  - stackname: stack1-db
    template: templates/stack1.template
    [...]
  - stackname: stack2-app
    template: templates/stack2.template
    parameters:
      build: 's3://{output:stack1:bucketName}/{env:BUILD}/build.tar.gz'
      db: '{output:stack1-db:DatabaseRds}'

Variables (global/local, nested into other placeholders)

vars:
  KeyPair: 'mykeypair'
    
blueprints:
  - stackname: mystack
    vars:
      ParentStack: 'MyParentStack'
    parameters:
      KeyPair: '{var:mykeypair}'
      Database: '{output:{var:ParentStack}:DatabaseRds}'
    [...]

Switch Profile Example (in this example an AMI is baked in a different account and shared with this account)

blueprints:
  - stackname: mystack
    parameters:
      BaseAmi: '[profile:myDevAccountProfile:{output:bakestack:BaseAmi}]'

You might end up deploying the same stacks to multiple environments or accounts. Instead of duplicating the blueprints (or using YAML reference) you'll probably want to parameterize your blueprints like this

blueprints:
  - stackname: 'app-{env:Environment}-build'
    template: 'build.template'
    parameters:
      KeyPair: 'MyKeyPair'
    [...]

... and then before deploying (locally or from your CI server) you'd set the env var first and then deploy:

export Environment=prod
bin/stackformation.php blueprint:deploy 'app-{env:Environment}-build'

But in many cases those stacks do have some minor differences in some of the parameters (e.g. different VPCs or KeyNames,...) You could solve it like this with nested placeholders:

blueprints:
  - stackname: 'app-{env:Environment}-build'
    template: 'build.template'
    vars:
      prod-KeyName: MyProdKey
      stage-KeyName: MyStageKey
    parameters:
      KeyPair: '{var:{env:Environment}-KeyName}'

While this is perfectly possible this gets very confusing soon. Plus you'll have to mention every variation of the variable explicitely.

Instead you can use a conditional value:

blueprints:
  - stackname: 'app-{env:Environment}-build'
    template: 'build.template'
    parameters:
      KeyPair: 
        '{env:Environment}==prod': MyProdKey
        '{env:Environment}==stage': MyStageKey
        'default': MyDevKey

StackFormation will evaluate all keys from top to bottom and the first key that evaluates to true will be returned. Allowed conditions:

• 'A==B'
• 'A!=B'
• 'default' (will always evaluate to true. Make sure you put this at the very end since everything after this will be ignored). Placeholders will be resolved before the conditions are evaluated.

When referencing a stack in {output:<stack>:<output>}, {resource:<stack>:<logicalResource>}, or {parameter:<stack>:<logicalResource>} you can use a wildcard to specify a stack. In this case StackFormation looks up all live stacks and finds a stack matching the pattern. If there's no stack or more than a single stack matching the pattern StackFormation will throw an exception. This feature is helpful when you know there's always only a single stack of one type that has a placeholder in it's stackname:

Example: Stackname: deployment-{env:BUILD_NUMBER} In blueprints.yml:

blueprints:
  - stackname: mystack
    parameters:
      Elb: '{output:deployment-*:Elb}'

You can include environment variable in your stackname (which is very handy for automation via Jenkins). In this case your effective stackname (e.g. build-5) will be different from the configured stackname (e.g. build-{env:BUILD_NUMBER})

Example

blueprints:
  - stackname: 'build-{env:BUILD_NUMBER}'
    template: templates/deploy_build.template

Please note that all files paths in the template section of a blueprints.yml are relative to the current blueprints.yml file and all files included via Fn::FileContent/ Fn:FileContentTrimLines or Fn:FileContentMinify are relative to the CloudFormation template file.

Example:

blueprints/
  stack1/
    userdata/
      provisioning.sh
    blueprints.yml
    my.template

blueprints.yml:

blueprints:
  - stackname: test
    template: my.template

my.template

{ [...]
  "Ec2Instance": {
    "Type": "AWS::AutoScaling::LaunchConfiguration",
    "Properties": {
      "UserData": {"Fn::Base64": {"Fn::FileContent": "userdata/provisioning.sh"}}
    }
  }
}

StackFormation allows you to configure more than one template:

blueprints:
  - stackname: iam
    template:
      - iam_role_jenkins.template
      - iam_user_inspector.template
    description: 'IAM users and roles'

The template files cannot have duplicate keys in any of the top level attributes. StackFormation will then merge them into a single CloudFormation template and deploy this one instead. This feature helps you to structure your template logically without having to deploy and manage them separatly. Also with this you can choose which template to include in case you're pulling in a StackFormation module like https://github.com/AOEpeople/cfn-lambdahelper.

You can always inspect the final merged and preprocessed template:

bin/stackformation.php stack:template iam

If you list your templates with attributes instead of a plain list, the attribute keys will be used to prefix every element of that template. This way you can you the same template with different input parameters instead of duplicating resources. This comes in handy for VPC setups.

blueprints:
  - stackname: vpc-subnets
    template:
      ZoneA: az.template
      ZoneB: az.template
    parameters:
      ZoneAVpc: MyVPC
      ZoneAPublicSubnetCidrBlock: '10.0.0.0/24'
      ZoneAPrivateSubnetCidrBlock: '10.0.10.0/24'
      ZoneAAZ: 'eu-west-1a'
      ZoneBVpc: MyVPC
      ZoneBAPublicSubnetCidrBlock: '10.0.1.0/24'
      ZoneBPrivateSubnetCidrBlock: '10.0.11.0/24'
      ZoneBAZ: 'eu-west-1b'
      [...]

If you have a parameter that needs to be passed to all templates you can prefix it with '' (make sure you add quotes around that key since JSON will consider this a reference instead) and StackFormation will replace '' with each prefix used in the template: section.

blueprints:
  - stackname: vpc-subnets
    template:
      ZoneA: az.template
      ZoneB: az.template
    parameters:
      '*Vpc': MyVPC # Will automatically be expanded to 'ZoneAVpc: MyVPC' and 'ZoneBVpc: MyVPC'
      '*Igw': MyInternetGateway
      ZoneAPublicSubnetCidrBlock: '10.0.0.0/24'
      ZoneAPrivateSubnetCidrBlock: '10.0.10.0/24'
      ZoneAAZ: 'eu-west-1a'
      ZoneBVpc: MyVPC
      ZoneBAPublicSubnetCidrBlock: '10.0.1.0/24'
      ZoneBPrivateSubnetCidrBlock: '10.0.11.0/24'
      ZoneBAZ: 'eu-west-1b'
      [...]

You can run shell commands before the CloudFormation is being deployed. The commands will be executed in the directory where the blueprints.yml file lives.

Example:

blueprints:
  - stackname: 'my-lambda-function'
    template: lambda.template
    Capabilities: CAPABILITY_IAM
    before:
    - cd function 
    - npm install aws-sdk
    - zip -r nat_gateway.zip nat_gateway.js node_modules/
    - aws s3 cp nat_gateway.zip s3://mybucket/lambda/nat_gateway.zip

and you can even use placeholders:

blueprints:
  - stackname: 'my-lambda-function'
    template: lambda.template
    Capabilities: CAPABILITY_IAM
    vars:
      bucket: mybucket
      key: 'lambda/nat_gateway.zip'
    parameters:
      # these are the input parameters passed to the cfn template that match the upload location in the custom script below
      S3Bucket: '{var:bucket}'
      S3Key: '{var:key}'
    before:
    - cd function
    - npm install aws-sdk
    - zip -r nat_gateway.zip nat_gateway.js node_modules/
    - aws s3 cp nat_gateway.zip s3://{var:bucket}/{var:key}

Similar to before scripts you can define scripts that are being executed after the stack has been deployed. Please note this only work if you're 'observing' the deploying (no if you deployed with '--no-observe' or if you're stopping the process (e.g. CTRL+C) during the deployment.

The after configuration equals the before configuration with the addition that you have access to the status in the ${STATUS} variable/ (Special status values in addition to the default ones like 'CREATE_COMPLETE',... are 'NO_UPDATES_PERFORMED' and 'STACK_GONE')

Example

blueprints:
  - stackname: 'my-static-website'
    description: 'Static website hosted in S3'
    template: 'website.template'
    after:
      - 'if [[ $STATUS =~ ^(UPDATE|CREATE)_COMPLETE|NO_UPDATES_PERFORMED$ ]] ; then aws s3 sync --delete content/ s3://www-tst.aoeplay.net/; fi'

before or after are being executed in the base directory of the current blueprint (that's the directory the blueprint's blueprint.yml file is located at). But you can switch directories in your script. The ${CWD} variable holds the current working directory (the project root) in case you want to switch to that.

When a profile is being used (even if the profile is loaded via the profiles.yml file) the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables will be set in the script context, so you can safely call the aws cli tool in the same context the blueprint is being deployed.

In addition to that ${BLUEPRINT} will hold the current blueprint's name and ${STACKNAME} the current resulting stack name Also ${STATUS} will hold the last status of the stack that has just been deployed (after scripts only).

You can separate the script lines in an array (that will then be concatenated with \n before executing:

blueprints:
  - stackname: 'my-static-website'
    [...]
    after:
      - 'echo "Line 1"'
      - 'echo "Line 2"'

or you can use the YAML multiline notation:

blueprints:
  - stackname: 'my-static-website'
    [...]
    after: |
      echo "Line 1"
      echo "Line 2"

StackFormation uses the AWS SDK for PHP. You should configure your keys in env vars:

export AWS_ACCESS_KEY_ID=INSERT_YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=INSERT_YOUR_PRIVATE_KEY
export AWS_DEFAULT_REGION=eu-west-1

Before uploading CloudFormation template to the API there's some pre-processing going on: I've introduced a new function "FileContent" that accepts a path to a file. This file will be read, converted into JSON (using Fn::Join). The path is relative to the path of the current CloudFormation template file.

Usage Example:

    [...]
    "UserData": {"Fn::Base64": {"Fn::FileContent":"../scripts/setup.sh"}},
    [...]

These functions are similar to Fn::FileContent but additional they trim whitespace or minify the code. This comes in handy when deploying Lambda function where the content can't be larger than 2048kb if you want to directly embed the source code via CloudFormation (instead of deploying a zip file).

This function is the same as Fn::FileContent expect it will return the resulting JSON without formatting it, which will reduce the file size significantly due to the missing whitespace in the JSON structure (not inside the file content!) This is useful if you're seeing the "...at 'templateBody' failed to satisfy constraint: Member must have length less than or equal to 51200" error message.

Sometime you have a dynamic number of array items. Fn::Split allows you to configure them as a single string and transforms them into an array:

"Aliases": { "Fn::Split": [",", "www.example.com,cdn.example.com"]}

results in:

"Aliases": ["www.example.com","cdn.example.com"]

The scripts (included via Fn::FileContent) may contain references to other CloudFormation resources or parameters. Part of the pre-processing is to convert snippets like {Ref:MagentoWaitConditionHandle} or {Ref:AWS::Region} (note the missing quotes!) into correct JSON snippets and embed them into the Fn::Join array.

Usage Example:

#!/usr/bin/env bash
/usr/local/bin/cfn-signal --exit-code $? '{Ref:WaitConditionHandle}'

will be converted to:

{"Fn::Join": ["", [
"#!\/usr\/bin\/env bash\n",
"\/usr\/local\/bin\/cfn-signal --exit-code $? '", {"Ref": "WaitConditionHandle"}, "'"
]]}

You can include content from a different file into a script. Use this is you have duplicate code that you need to embed into multiple resource's UserData:

Example:

#!/usr/bin/env bash

###INCLUDE:../generic/includes/base.sh

[...]

###JSON###
{ "hello": "world" }
######

You can configure a regular expression in the STACKFORMATION_NAME_FILTER environment variable (e.g. via .env.default) which will filter all your stack lists to the stacks matching this pattern. This is useful if you have a naming convention in place and you don't want to see other team's stacks in your list.

Example:

STACKFORMATION_NAME_FILTER=/^myproject-(a|b)-/

You can add comments to your JSON file. Due to a current bug you can't have double quotes in your comment block.

Example:

{"IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "CidrIp": "1.2.3.4/32"}, /* AOE WI Office */
{"IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "CidrIp": "5.6.7.8/32"}, /* Fabrizio Home Office */

"Port":"..." will automatically expanded to "FromPort": "...", "ToPort": "...". So if you're specifying a single port instead of a range of ports you can reduce the redundancy:

Example:

{"IpProtocol": "tcp", "Port": "80", "CidrIp": "1.2.3.4/32"}, 
/* expands to: */
{"IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "CidrIp": "1.2.3.4/32"},

Tired of concatenating strings with {"Fn::Join": ["", [ manually? Just add the references in a string and StackFormation will expand this for you:

Example:

"Key": "Name", "Value": "magento-{Ref:Environment}-{Ref:Build}-instance"
/* will be replaced with: */
"Key": "Name", "Value": {"Fn::Join": ["", ["magento-", {"Ref":"Environment"}, "-", {"Ref":"Build"}, "-instance"]]}

Use the jq tool to create a simple list of all parameters (almost) ready to paste it in the blueprints.yml

cat my.template | jq '.Parameters | keys' | sed 's/",/: \'\'/g' | sed 's/"//g'