briefercloud / layerform Goto Github PK

• Allow passing tfvars file in the CLI
• Allow passing passing individual vars
• Support all input modes Terraform does (double check)
• Duplicate names -> Allow terraform to error out (default behaviour)
• Use terraform.tfvars by default (from cwd)

• Links to docs
• Links to discord
• Links to GH
• Available at layerform.dev
• Favicon and title
• SEO + OG

• Support writing a config file in the .layerform directory
• Layerform CLI should read from this file to know where to store configs locally

Example:

currentContext: my-s3-context
contexts:
  my-local-context:
    type: local
    dir: folder-a # this can be relative to this file or absolute

• Support killing sibling layers
• Support killing underlying layers and all the ones on top
• --force

Ability to update a layer after updating and provisioning a terraform file instead of spawning a new thing.

• $ layerform spawn upper_layer
• Spawns all underlying layers recursively using the name default
• If a base layer called default already exists, reuse it
• Support multiple underlying layers (multiple dependencies)

hi, great blog about opensource washing.

I couldn't help but notice that the Venn diagram seems to have a fault.
the three states of software is:
1- is proprietary AND closed source
2- is proprietary AND source is available
3- is fully open source (is open source AND source is available)

so the Venn should look like this:

The current diagram leaves a section which is source available and not proprietary and not open source which means it's undefined.

It's a silly issue but just wanted to point it out 😁

Sometimes there are things that shouldn't leak out to upper layers from base layers(say kubernetes root account token), these things might end up in the state file. And since layerform works by copying around sate files according to the blog .

Is there a way to restrict/filter the information that ends up in shared layerform state backend?

• layerform list contexts -> prints all contexts saved on ~/.layerform/config. If none, print message.
• layerform set context -> uses a context from the list to set as "current"

Bug

If I write a layerform.json like this:

{
  "layers": [
    {
      "name": "foo",
      "files": ["foo.tf"]
    },
    {
      "name"  : "bar",
      "files": ["bar.tf"],
      "dependencies": ["foo", "this_layer_does_not_exist"]
    },
    {
      "name"  : "baz",
      "files": ["baz.tf"],
      "dependencies": ["foo"]
    }
  ]
}

Then run layerform configure on this file, layerform will just panic. See video below:

Solution

We should not assume that all layer listed in dependencies array always exists, we should check if that is the case and fail with a good error message to the user.

This validation can be performed in this method:
https://github.com/ergomake/layerform/blob/a4f6211eb76c602ca5ad8f6429d0afa531499d48/internal/layerfile/layerfile.go#L37

• Print list of which layers will be reused and which will be created
• Show INFO logs of what's happening
• After done, show success message
• If error, show error message
• If LAYERFORM_LOG_LEVEL is debug show underlying layerform logs for DEBUG
• If LAYERFORM_LOG_LEVEL is trace show actual terraform logs too

• Ability to upload layer definitions to Ergomake back-end
• Ability to read layer definitions from Ergomake back-end (for spawn)
• Ability to upload layer states to Ergomake back-end
• Ability to read layer states from Ergomake back-end

After spawning or killing a layer, successfully or not we save that information to the layerform state.

This step is crucial for us to be able to later spawn layers on top of it or kill it.

Currently, we delete all the temporary directories we create in order to perform the spawn/kill process, that means that it is literally impossible for the user to even try to fix the issue by hand.

• When failing to upload, retry 3 times (back-off 10s).
• When all attempts fail, keep file on disk (not temporary folder) and show error message (link to Discord)

• When applying layer, save state in the configured S3 bucket
• When deleting layer, save state in the configured S3 bucket

• layerform list instances
• Support S3 state
• Support local state

Awesome stuff! This seems like a neat approach to bring ownership boundaries into a Terraform codebase 👍

I have some general questions (disclaimer: I've read about Layerform but haven't tried it yet, just skimmed a blog post and the readme.md):

• Why not use "ephemeral"?

  You mention "ephemeral environments" in blog posts - but in layerform docs, it seems to always be multiple "staging" environments - Is there a reason you're steering clear of the term "ephemeral"? What sets Layerform's layers apart from what's typically considered ephemeral environments in your view?

• When the base layer runs, does it also run all the layers above it? Thoughts on How to deal with potential breakages in upper layers?

  Using the example from docs, if the core team updates the EKS cluster, and runs layerform spawn, Will that also run all the layers above it, so they inherit any changes from the cluster apply?

• Are random id's managed in terraform via discipline

  In order to be able to deploy additional "staging" layers, resource names should have random strings in them to prevent clashes. Is this random naming completely hand-managed through dev-discipline? Do you have any thoughts about layerform features that would either tack on random strings or at least detect and warn about names that would potentially collide?

• To further enforce ownership boundaries, Can each layer live in a separate git repo?

  Reasons to do this, reasons to never do this?

• Use ephemeral layers as a means to enable Blue/Green deployments?

  Do you see a path towards, say standing up a "to-be-prod" layer in parallel to a "prod" layer and after completing testing/verification, making the "to-be-prod" the default layer?

Looking forward to hearing your insights!

• Collect number of layers spun-up and date/time
• Collect number of resources applied
• Allow users to opt-out by setting LAYERFORM_TELEMETRY_DISABLED
• Throw them on posthog

We should create PR templates with the following sections:

• Background -> why this matters
• Solution -> brief description of the approach
• How to test it -> self-explanatory and say it's optional

If you have other ideas, lmk.

For some reason we print the errors twice, we print the error, then we print help (i don't think this is necessary), then we print the error once again.

This happens for all commands, not just spawn.

Example with spawn:

> layerform spawn eks                
✓ Instance "U6jvvBcw32X9ika5nm4aMb" of layer "eks" ready
Error: fail to spawn layer: fail to terraform apply: exit status 1

Error: creating Amazon S3 (Simple Storage) Bucket (eks-bucket-pv76pd56): BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
        status code: 409, request id: ****, host id: ****

  with module.eks.aws_s3_bucket.eks_bucket,
  on eks/main.tf line 22, in resource "aws_s3_bucket" "eks_bucket":
  22: resource "aws_s3_bucket" "eks_bucket" {


Usage:
  layerform spawn <layer> [desired_id] [flags]

Flags:
      --base stringToString   a map of underlying layers and their IDs to place the layer on top of (default [])
  -h, --help                  help for spawn
      --var stringArray       a map of variables for the layer's Terraform files. I.e. 'foo=bar,baz=qux'

fail to spawn layer: fail to terraform apply: exit status 1

Error: creating Amazon S3 (Simple Storage) Bucket (eks-bucket-pv76pd56): BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
        status code: 409, request id: ****, host id: ****

  with module.eks.aws_s3_bucket.eks_bucket,
  on eks/main.tf line 22, in resource "aws_s3_bucket" "eks_bucket":
  22: resource "aws_s3_bucket" "eks_bucket" {

• Use format --base="layer1=id1,layer2=id2"
• When a layer is missing but the layer underneath is given a desired ID, the skipped layer gets a random ID

• layerform kill <layer_name> <id>
• Error-out when layers on top exist
• Ask for confirmation before deleting
• Default id is default

When the user configures a context of type local it also has to provide the dir for the state and definitions to be saved to. Currently, if that directory is non existing, layerform just errors out, see video.

Layerform should only accept "url friendly" names for the instances.
To make it simpler, let's just not accept anything that contains "special characters".
In another words, we should only accept alphanumerical characters: [A-Za-z0-9]

For example:

layerform spawn base "lucas' his layer"

This should fail with an error message explaining that the given instance name lucas' his layer is invalid.

• layerform list definitions
• Support S3 state
• Support local state

Reproducing the bug

1. Configure the local example:

layerform configure --file examples/local/layerform.json

2. Spawn an instance of bar

layerform spawn bar bar_instance

3. Update examples/local/layerform.json removing layer bar:

{
  "layers": [
    {
      "name": "foo",
      "files": ["foo.tf"]
    },
    {
      "name"  : "baz",
      "files": ["baz.tf"],
      "dependencies": ["foo"]
    }
  ]
}

4. Run configure again

layerform configure --file examples/local/layerform.json

Well, we just broke layerform, just trying to run layeform list instances will result in a panic, trying to kill bar_instance will also fail because it will not find the definition.

> layerform list instances                                
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x40 pc=0x1054f05c0]

goroutine 1 [running]:
github.com/ergomake/layerform/cmd/cli.computeDepth(0x1058a89a0?, 0x140001c5ae8?, 0x0)
        /Users/lucas/code/ergomake/layerform/cmd/cli/list_instances.go:111 +0x20
github.com/ergomake/layerform/cmd/cli.sortInstancesByDepth.func1(0x140001c58e8?, 0x1)
        /Users/lucas/code/ergomake/layerform/cmd/cli/list_instances.go:125 +0x68
sort.insertionSort_func({0x140001c59a0?, 0x14000141ec0?}, 0x0, 0x3)
        /opt/homebrew/Cellar/go/1.20.5/libexec/src/sort/zsortfunc.go:12 +0xc0
sort.stable_func({0x140001c59a0?, 0x14000141ec0?}, 0x3)
        /opt/homebrew/Cellar/go/1.20.5/libexec/src/sort/zsortfunc.go:343 +0x6c
sort.SliceStable({0x105856e40?, 0x1400012ced0?}, 0x4?)
        /opt/homebrew/Cellar/go/1.20.5/libexec/src/sort/slice.go:38 +0x104
github.com/ergomake/layerform/cmd/cli.sortInstancesByDepth({0x14000141e60, 0x3, 0x4}, 0x140001c5ae8)
        /Users/lucas/code/ergomake/layerform/cmd/cli/list_instances.go:122 +0x78
github.com/ergomake/layerform/cmd/cli.glob..func4(0x105e85c60?, {0x1054f29c3?, 0x0?, 0x0?})
        /Users/lucas/code/ergomake/layerform/cmd/cli/list_instances.go:82 +0x450
github.com/spf13/cobra.(*Command).execute(0x105e85c60, {0x105ec51b8, 0x0, 0x0})
        /Users/lucas/go/pkg/mod/github.com/spf13/[email protected]/command.go:944 +0x5b0
github.com/spf13/cobra.(*Command).ExecuteC(0x105e86220)
        /Users/lucas/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068 +0x35c
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/lucas/go/pkg/mod/github.com/spf13/[email protected]/command.go:992
github.com/ergomake/layerform/cmd/cli.Execute()
        /Users/lucas/code/ergomake/layerform/cmd/cli/root.go:30 +0xfc
main.main()
        /Users/lucas/code/ergomake/layerform/main.go:13 +0x144

Proposed solution

The second configure command should just fail in this situation, telling the user that it can't update configuration because it is trying to remove a definition that has an instance currently running.

With new contributors coming in, we should write a proper CONTRIBUTING.md file to guide new developers and offer then a good experience.

Here are a few things I think we should include:

• How to install the Layerform CLI locally
• How to run the Layerform CLI straight from source go run main.go
• What the examples do and how to run them
• Brief guidelines on how the internals of the project work and where important files/functions are
• Code standards (go fmt etc)

• Configuration files manually written on disk (schema TBD)
• Each layer has a random name
• $ layerform spawn base_layer

• Support writing a config file in the .layerform directory (file name is configurations.yml)
• Support setting current context in the same file
• Now, user manually uploads definitions to this place (currently a json)
• S3 bucket is all ours, we'll override file if it exists.

Example:

currentContext: my-s3-context
contexts:
  my-s3-context:
    type: s3
    bucket: blabla
  my-local-context:
    type: local
    dir: folder-a # this can be relative to this file or absolute

Layerform should only accept "url friendly" names for the definitions.
To make it simpler, let's just not accept anything that contains "special characters".
In another words, we should only accept alphanumerical characters: [A-Za-z0-9]

For example:

Considering the following content for the layerform.json file.

{
  "layers": [
    {
      "name": "invalid name for a layer definition", // this is invalid because it has spaces
      "files": ["foo.tf"]
    },
    {
      "name"  : "invalid!", // this is invalid because it has a special character
      "files": ["bar.tf"]
    }
  ]
}

Running configure passing that file should fail with an error message explaining that those definitions names are invalid.

Should support homebrew.
Should use gorelease and point people to download the binary.

• Show INFO messages when deleting
• Show DEBUG messages when deleting when LAYERFORM_LOG_LEVEL is debug
• Show TRACE messages when LAYERFORM_LOG_LEVEL is trace - these are actual terraform logs for deletion + debug + trace logs

Problem:

When using layerform in a script, or CI environment, I want to be able to kill layer instances without having to answer yes to the "are you sure?" question.

Proposal:

Add a --auto-approve flag to the kill command that makes the cli skip the confirmation when killing a layer instance.

👋 it would be good to cut a release for version management and distribution.