weberc2 / infra Goto Github PK

• Create aws-cognito-user-pool module
  • Module creates app authentication pages Guide
  • Module configures Cognito to send emails properly (requires SES role + validated email -- consider SendGrid for the latter)
• Create README for module
• Create test API Gateway + Lambda to confirm functionality

• tf plan for all targets
• tf fmt -check for all tf files in the repo (targets and modules)

Concretely for #15 I want to be able to define a golanglambda project type which will define an s3publish job that will take a golang project and publish it to an S3 bucket (plus or minus building it). This requires a dependency from golanglambda.s3publish on golang.test and golang.lint.

In the abstract, we want to be able to express a dependency between any two jobs in the same workflow.

The "key file" concept: pattern-match a directory to infer whether or not it was a project and if so, what its type was--e.g., if a directory contained a terraform.tf file I would regard it as a terraform project, if it had a go.mod file then I would infer it to be a golang project, etc.

This was a neat concept, but it doesn't handle dependencies well (NOTE: I don't have dependencies yet, but I intend to add them soon). Specifically I want to be able to associate a particular project to a list of other project dependencies. Since a project is a (path, type) tuple, it doesn't suffice to drop in an optional DEPENDENCIES file that lists dependency projects because in cases where the project path contains multiple projects (i.e., multiple key files corresponding to different project types) it wouldn't be clear which of those dependencies are associated with which project type.

Instead it seems more natural to create an explicit projects.yaml file that explicitly defines what project types are associated with the current directory and eventually mapping those types to their respective lists of dependencies.

For this ticket, it suffices to have a structure with schema {"projects": [{"type": "<type>"}, ...]}.

For each target, we should generate a plan PR workflow and a apply merge workflow from a template.

Candidate Implementation

• Script which generates the plan and apply workflow files for all targets as well as the static workflows
• PR job that applies the script and fails if there are any diffs (including targets that have been deleted but whose workflow files remain)

Motivation

• Lambda + API Gateway is the cheapest way to run services on AWS
• Multiple targets are expected to leverage this pattern
  • Blog analytics service
  • Blog comments service

Module Resources

• Lambda Function
• Lambda Function IAM role
• API Gateway
• API Gateway IAM role

Requirements

• Lambda function source code is directly referenced by Terraform (no need to zip or upload to S3 before running Terraform)
• Lambda function is publicly accessible via the API Gateway
• Lambda function logs are available in CloudWatch
• Module outputs the various resources
• Module documented by README

Note

• See hashicorp/terraform#8344 (comment) for details on referencing source code directly from Terraform. This is from 2016 and so may be outdated.
• See https://github.com/weberc2/monorepo/blob/master/blog/analytics/src/blog_analytics/main.py for an example of creating and stringing-together the required resources.

• S3 bucket for source code artifacts
• Create and demo golang project type
• Create golang-lambda project type template which builds an artifact and publishes it to the lambda source code bucket
• Update workflows script to render the lambda deploy pipeline template for lambda projects

We need a bootstrap target to initialize the resources for managing remote state. The state file for the bootstrap target will live in the bootstrap target directory in the repository, all other targets will use the remote state.

Right now, any job type that wants to declare dependencies on another job type must generate its own needs: block in its YAML template. We should generate this automatically.

Proposal

Rather than a job declaring its full YAML block as a text template, we let them define distinct step strings which are then templated.

golanglambda project type should build a Go project, zip it up, and push it to an S3 bucket.

Presently the golint PR job type is only linting the {{ .Path }} directory, rather than recursive subdirectories.

• Presently because project types must declare the names and types of all instances of their dependencies, there is no way for a terraformtarget type to specify a dependency on a golanglambda type.
• Even if that support existed, the terraformtarget for a lambda needs to take a parameter (the s3 address for the code artifact) and thread that through to its underlying aws_lambda_function resource.

• "targets" directory is organized by environment subdirectories
• Each target's state key is updated with an environment prefix