The deployment (sub) module uses a data source of an existing S3 bucket for storing pipeline artifacts. This bucket is (re-used) for all pipelines.

There is no documentation about this bucket and it's name can't be configured so this won't be usable outside our team.

Options:

• create the bucket (one for each service) inside the deployment (sub) module (service specific naming)
• make id of required bucket configurable

I'd prefer option one since it's more coherent IMO and requires less upfront terraforming outside the module.

This module can't be applied using terraform 0.13. It fails with Error: ECR Repository (service-name) not found.

Details:

2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 2020/08/25 09:49:16 [DEBUG] [aws-sdk-go] DEBUG: Response ecr/DescribeRepositories Details:
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: ---[ RESPONSE ]--------------------------------------
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: HTTP/1.1 400 Bad Request
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Connection: close
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Content-Length: 153
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Content-Type: application/x-amz-json-1.1
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Date: Tue, 25 Aug 2020 07:49:16 GMT
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: X-Amzn-Requestid: 99522912-b9c0-44a3-9be2-1fc3b09ef2a7
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: -----------------------------------------------------
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 2020/08/25 09:49:16 [DEBUG] [aws-sdk-go] {"__type":"RepositoryNotFoundException","message":"The repository with name 'service-name' does not exist in the registry with id '12432442342'"}
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 2020/08/25 09:49:16 [DEBUG] [aws-sdk-go] DEBUG: Validate Response ecr/DescribeRepositories failed, attempt 0/25, error RepositoryNotFoundException: The repository with name 'service-name' does not exist in the registry with id '12432442342'
2020/08/25 09:49:16 [ERROR] eval: *terraform.evalReadDataPlan, err: ECR Repository (service-name) not found
2020/08/25 09:49:16 [ERROR] eval: *terraform.EvalSequence, err: ECR Repository (service-name) not found

• example(s)
• documentation

To simplify client code, we could attach an IAM policy to the existing ecs task role in order to allow containers to put log events into firehose.

Module clients could remove the following code and rely on the module:

data "aws_iam_policy_document" "policy" {
  statement {
    actions = ["firehose:PutRecordBatch"]
    // todo limit this service stream
    resources = ["*"]
  }
}

All aws resources get default tags when created with this module.

When I create an ECS service via another github repository where I use terraform-aws-buzzgate the source tag should be of the service repository and not of github.com/stroeer/terraform-aws-buzzgate.

Example
in the github.com/stroeer/polyphase repo i use this module and I create aws resources with tf files, that are in the given repo. So to be able to terraform these resources you would need to know, that the tf files reside in github.com/stroeer/polyphase and not in github.com/stroeer/terraform-aws-buzzgate

Problem Description

The Provisioning phase for the CodeBuild step currently takes several minutes, which slows down deployments of ecs tasks by a lot.

Quick Analysis & Solution Proposal

• Currently we use aws/codebuild/amazonlinux2-x86_64-standard:1.0, which is pretty old.
• The Internet says that older images get dropped on the AWS caches, so we should probably switch to a newer image.
• According to this, aws/codebuild/amazonlinux2-x86_64-standard:3.0 should be the one to to switch to:
https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-available.html

There is an easier way to automatically add tags to all resources, see blog post or aws tf provider docs

With #14 we introduced an IAM role with required permissions to run CodePipeline for services. This role currently contains commented code and it's not finally clear, which permissions are actually needed.

We should provide examples depicting major features of this module. Those examples need to be terraform apply'able by users without errors.

Possible examples:

• complete example with all basic variables set
• with ALB
• with deployment pipeline
• with app mesh and logging
• with NLB

If create_deployment_pipeline is set to false, terraform plan fails with:

Error: Failed getting S3 bucket: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadBucketInput.Bucket.
 Bucket: ""

  on .terraform/modules/client/terraform-aws-ecs-fargate-0.1.1/modules/deployment/s3.tf line 13, in data "aws_s3_bucket" "codepipeline":
  13: data "aws_s3_bucket" "codepipeline" {

Monitoring aws/containers-roadmap#698 in order to send app logs directly to Elasticsearch instead of piping through Kinesis Firehose streams.

We should list and cut all dependencies to our internal datasources and remote states, to make this module more generic to team/organization external users.

Affected resources:

• ssm_ecs_task_execution_role data source
• aws_service_discovery_service namespace_id

We might remove internal dependencies to 3rd party terraform modules like terraform-aws-modules/s3-bucket/aws which cuts interdependencies in terms of required/supported aws provider and terraform versions.

Currently the deployment is part of the main stack, which can be deactivated using a variable.

We might also separate those 2 stacks like in terraform-aws-lambda/tree/master/modules/deploy.

Add a CONTRIBUTING.md describing how to contribute to this project.

With enabled logs sub-module, we should provide the opportunity to provide an existing CloudWatch Log Group for logs of the fluent-bit sidecar or create one (which should be the default).

│ Error: Invalid count argument
│ 
│   on .terraform/modules/services.otel_container_definition/modules/assert/main.tf line 2, in data "external" "assertion":
│    2:   count   = var.condition ? 0 : 1
│ 
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To
│ work around this, use the -target argument to first apply only the resources that the count depends on.
╵
make: *** [tf] Error 1

Workaround: apply using 0.29.0 first.

An initial terraform apply using create_ingress_security_group = true fails with

╷
│ Error: Invalid for_each argument
│ 
│   on ../../main.tf line 2, in data "aws_lb" "public":
│    2:   for_each = var.create_ingress_security_group ? toset([for target in var.target_groups : lookup(target, "load_balancer_arn", "")]) : []
│     ├────────────────
│     │ var.create_ingress_security_group is true
│     │ var.target_groups is tuple with 1 element
│ 
│ The "for_each" set includes values derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.
│ 
│ When working with unknown values in for_each, it's better to use a map value where the keys are defined statically in your configuration and where only the values contain apply-time results.
│ 
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.

As a workaround it's possible to apply with create_ingress_security_group = false and then afterwards setting the variable to true.

Versions:

Terraform v1.9.0
on darwin_arm64
+ provider registry.terraform.io/cloudposse/template v2.2.0
+ provider registry.terraform.io/cloudposse/utils v1.24.0
+ provider registry.terraform.io/hashicorp/aws v5.58.0
+ provider registry.terraform.io/hashicorp/http v3.4.3
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/random v3.6.2

If the a ALB target group needs to be replaced (e.g. aws_alb_target_group.public because the container_port changed), this fails with:

Error deleting Target Group: ResourceInUse: Target group 'arn:aws:elasticloadbalancing:eu-west-1:12345678:targetgroup/my-service/1541e04161e7f3ef' is currently in use by a listener or a rule

see https://github.com/telia-oss/terraform-aws-ecs-fargate/blob/0f3cbc67c057625f1354f28f6ad4efc3763a8978/main.tf#L208 for a possible solution

The deployment (sub) module uses data sources of existing IAM roles for codepipeline and codebuild.

There is no documentation about the necessary permissions for those roles and their names can't be configured so this won't be usable outside our team.

Options:

• create those roles inside the deployment (sub) module (service specific naming)
• document required permissions and make ARNs of the required roles configurable

I'd prefer option one since it's more coherent IMO and requires less upfront terraforming outside the module.

It would be useful if the deployment process supported more than only production as a tag in ECR, or at least allow this tag to be specified.

Providing a container_port used for the load balancer and/or mesh proxy configuration is mandatory at the moment. We might want to support apps w/o an exposed port as well, e.g. apps consuming Kinesis/Dynamodb streams or sqs queues.

If using the log sub-module, an Elasticsearch domain named application-logs is expected. This variable is not yet configurable from the main module.

aws_codebuild_project(https://www.terraform.io/docs/providers/aws/r/codebuild_project.html) defaults to create a CloudWatch log group which is not managed by terraform lifecycle. Thus those log groups are not destroyed.

If those log groups are needed at all, we should manage them in the module (including definition of retention time).

the variables of the deployment (sub) module have no description at the moment.