The fury-eks-installer from sighupio

fury-eks-installer's Issues

IAM policy is missing in the latest aws account

With a new AWS account, a new permission is needed to create a load balancer.
The 12.2.0 introduced the needed patch to the policy: terraform-aws-modules/terraform-aws-eks#902
Should be useful to update asap to this version.
Thanks!

Can't use my kubernetes provider if it's defined inside the module

cat main.tf
/**
 * Copyright (c) 2017-present SIGHUP s.r.l All rights reserved.
 * Use of this source code is governed by a BSD-style
 * license that can be found in the LICENSE file.
 */

terraform {
  experiments = [module_variable_optional_attrs]
  backend "s3" {
    bucket = "furyctl-issue-196"
    key    = "barebone/use1/cluster.json"
    region = "us-east-1"
  }

  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
    }
  }
}

provider "aws" {
  region = "us-east-1"
  default_tags {
    tags = {
      env = "product-day-qa"
      githubIssue = "303"
      k8s = "eks-barebone"
    }
  }
}

provider "kubernetes" {
   host = data.aws_eks_cluster.fury.endpoint
   cluster_ca_certificate = base64decode(data.aws_eks_cluster.fury.certificate_authority[0].data)
   token = data.aws_eks_cluster_auth.fury.token
}

module "fury" {
  source = "~/.furyctl/eks-barebone/vendor/installers/eks/modules/eks"

  cluster_name               = var.cluster_name
  cluster_version            = var.cluster_version
  cluster_log_retention_days = var.cluster_log_retention_days
  network                    = var.network
  subnetworks                = var.subnetworks
  dmz_cidr_range             = var.dmz_cidr_range
  ssh_public_key             = var.ssh_public_key
  node_pools                 = var.node_pools
  node_pools_launch_kind     = var.node_pools_launch_kind
  tags                       = var.tags

  # Specific AWS variables.
  # Enables managing auth using these variables
  eks_map_users    = var.eks_map_users
  eks_map_roles    = var.eks_map_roles
  eks_map_accounts = var.eks_map_accounts
}

cat .terraform/modules/fury/kubernetes.tf
data "aws_eks_cluster" "cluster" {
  name = module.cluster.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.cluster.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  load_config_file       = false
}
terraform apply
module.fury.module.cluster.aws_cloudwatch_log_group.this[0]: Refreshing state... [id=/aws/eks/eks-barebone/cluster]
module.fury.aws_key_pair.nodes: Refreshing state... [id=eks-barebone-20230301083258319400000001]
module.fury.module.cluster.aws_security_group.cluster[0]: Refreshing state... [id=sg-0f0046bd94eebbe6d]
module.fury.aws_security_group.nodes: Refreshing state... [id=sg-0905fc5bc2b95d013]
module.fury.module.cluster.aws_security_group.workers[0]: Refreshing state... [id=sg-0e4e1c5b331e9c995]
module.fury.aws_security_group.node_pool[2]: Refreshing state... [id=sg-096878a7e921d4246]
module.fury.aws_security_group.node_pool[1]: Refreshing state... [id=sg-020ae1b0b10ac4ea5]
module.fury.aws_security_group.node_pool[0]: Refreshing state... [id=sg-0976063af499fbac7]
module.fury.module.cluster.aws_iam_policy.cluster_elb_sl_role_creation[0]: Refreshing state... [id=arn:aws:iam::492816857163:policy/eks-barebone/eks-barebone-elb-sl-role-creation20230301083258320600000002]
module.fury.module.cluster.aws_iam_role.cluster[0]: Refreshing state... [id=eks-barebone20230301083258320800000003]
module.fury.aws_security_group_rule.ssh_from_dmz_to_nodes: Refreshing state... [id=sgrule-2968004512]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_self[0]: Refreshing state... [id=sgrule-2573013755]
module.fury.module.cluster.aws_security_group_rule.workers_egress_internet[0]: Refreshing state... [id=sgrule-2558094647]
module.fury.module.cluster.aws_security_group_rule.cluster_https_worker_ingress[0]: Refreshing state... [id=sgrule-1596022399]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_cluster_https[0]: Refreshing state... [id=sgrule-4230566281]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_cluster[0]: Refreshing state... [id=sgrule-2751700965]
module.fury.module.cluster.aws_security_group_rule.cluster_egress_internet[0]: Refreshing state... [id=sgrule-4246159717]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSVPCResourceControllerPolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-2023030108330150460000000a]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_elb_sl_role_creation[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301493800000008]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301481900000007]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSServicePolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301500700000009]
module.fury.module.cluster.aws_eks_cluster.this[0]: Refreshing state... [id=eks-barebone]
module.fury.module.cluster.aws_iam_role.workers[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEKS_CNI_Policy[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-20230301084516827400000011]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEKSWorkerNodePolicy[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-2023030108451680890000000f]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEC2ContainerRegistryReadOnly[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-20230301084516808900000010]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[0]: Refreshing state... [id=eks-barebone2023030108451615630000000d]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[2]: Refreshing state... [id=eks-barebone2023030108451615640000000e]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[1]: Refreshing state... [id=eks-barebone2023030108451615610000000c]
module.fury.module.cluster.aws_launch_template.workers_launch_template[1]: Refreshing state... [id=lt-059ae5d3ea75f17ba]
module.fury.module.cluster.aws_launch_template.workers_launch_template[0]: Refreshing state... [id=lt-06a755105d5409348]
module.fury.module.cluster.kubernetes_config_map.aws_auth[0]: Refreshing state... [id=kube-system/aws-auth]
module.fury.module.cluster.aws_launch_template.workers_launch_template[2]: Refreshing state... [id=lt-0292a97d1426825e2]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[0]: Refreshing state... [id=eks-barebone-infra20230301084520960500000019]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[2]: Refreshing state... [id=eks-barebone-app20230301084520960500000018]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[1]: Refreshing state... [id=eks-barebone-ingress2023030108452096060000001a]
╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│
│   on main.tf line 8, in terraform:
│    8:   experiments = [module_variable_optional_attrs]
│
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
│
│ (and one more similar warning elsewhere)
╵
╷
│ Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp [::1]:80: connect: connection refused
│
│   with module.fury.module.cluster.kubernetes_config_map.aws_auth[0],
│   on .terraform/modules/fury.cluster/aws_auth.tf line 65, in resource "kubernetes_config_map" "aws_auth":
│   65: resource "kubernetes_config_map" "aws_auth" {
│
╵

Removing provider from module, terraform use the right cluster endpoint.

cat .terraform/modules/fury/kubernetes.tf
data "aws_eks_cluster" "cluster" {
  name = module.cluster.cluster_id
}

❯ terraform apply
module.fury.module.cluster.aws_cloudwatch_log_group.this[0]: Refreshing state... [id=/aws/eks/eks-barebone/cluster]
module.fury.module.cluster.aws_security_group.cluster[0]: Refreshing state... [id=sg-0f0046bd94eebbe6d]
module.fury.aws_security_group.nodes: Refreshing state... [id=sg-0905fc5bc2b95d013]
module.fury.aws_key_pair.nodes: Refreshing state... [id=eks-barebone-20230301083258319400000001]
module.fury.aws_security_group.node_pool[0]: Refreshing state... [id=sg-0976063af499fbac7]
module.fury.module.cluster.aws_security_group.workers[0]: Refreshing state... [id=sg-0e4e1c5b331e9c995]
module.fury.aws_security_group.node_pool[1]: Refreshing state... [id=sg-020ae1b0b10ac4ea5]
module.fury.aws_security_group.node_pool[2]: Refreshing state... [id=sg-096878a7e921d4246]
module.fury.module.cluster.aws_iam_policy.cluster_elb_sl_role_creation[0]: Refreshing state... [id=arn:aws:iam::492816857163:policy/eks-barebone/eks-barebone-elb-sl-role-creation20230301083258320600000002]
module.fury.module.cluster.aws_iam_role.cluster[0]: Refreshing state... [id=eks-barebone20230301083258320800000003]
module.fury.aws_security_group_rule.ssh_from_dmz_to_nodes: Refreshing state... [id=sgrule-2968004512]
module.fury.module.cluster.aws_security_group_rule.cluster_egress_internet[0]: Refreshing state... [id=sgrule-4246159717]
module.fury.module.cluster.aws_security_group_rule.cluster_https_worker_ingress[0]: Refreshing state... [id=sgrule-1596022399]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_self[0]: Refreshing state... [id=sgrule-2573013755]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_cluster_https[0]: Refreshing state... [id=sgrule-4230566281]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_cluster[0]: Refreshing state... [id=sgrule-2751700965]
module.fury.module.cluster.aws_security_group_rule.workers_egress_internet[0]: Refreshing state... [id=sgrule-2558094647]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSServicePolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301500700000009]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSVPCResourceControllerPolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-2023030108330150460000000a]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301481900000007]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_elb_sl_role_creation[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301493800000008]
module.fury.module.cluster.aws_eks_cluster.this[0]: Refreshing state... [id=eks-barebone]
module.fury.module.cluster.aws_iam_role.workers[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEKS_CNI_Policy[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-20230301084516827400000011]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEKSWorkerNodePolicy[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-2023030108451680890000000f]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEC2ContainerRegistryReadOnly[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-20230301084516808900000010]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[0]: Refreshing state... [id=eks-barebone2023030108451615630000000d]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[2]: Refreshing state... [id=eks-barebone2023030108451615640000000e]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[1]: Refreshing state... [id=eks-barebone2023030108451615610000000c]
module.fury.module.cluster.aws_launch_template.workers_launch_template[1]: Refreshing state... [id=lt-059ae5d3ea75f17ba]
module.fury.module.cluster.aws_launch_template.workers_launch_template[2]: Refreshing state... [id=lt-0292a97d1426825e2]
module.fury.module.cluster.aws_launch_template.workers_launch_template[0]: Refreshing state... [id=lt-06a755105d5409348]
module.fury.module.cluster.kubernetes_config_map.aws_auth[0]: Refreshing state... [id=kube-system/aws-auth]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[0]: Refreshing state... [id=eks-barebone-infra20230301084520960500000019]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[2]: Refreshing state... [id=eks-barebone-app20230301084520960500000018]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[1]: Refreshing state... [id=eks-barebone-ingress2023030108452096060000001a]
╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│
│   on main.tf line 8, in terraform:
│    8:   experiments = [module_variable_optional_attrs]
│
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
│
│ (and one more similar warning elsewhere)
╵
╷
│ Error: Get "https://BEF030853566ED2382C53403F9949892.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 10.10.9.152:443: i/o timeout
│
│   with module.fury.module.cluster.kubernetes_config_map.aws_auth[0],
│   on .terraform/modules/fury.cluster/aws_auth.tf line 65, in resource "kubernetes_config_map" "aws_auth":
│   65: resource "kubernetes_config_map" "aws_auth" {
│
╵

Improve documentation

To further improve the module documentation:

Add changelogs of previous releases
Add documentation on how to perform day-2 operations (cluster updates, node-pool updates, etc...)
Explain better the module usage

Write the v1.11 to v2.0 upgrade guide

Please write the full upgrade guide for the installer.
It was initially described here, but later works caused that guide to go out of sync, so we should update the whole thing once it's well tested.

Keep in mind that the guide above was including a lot of furyctl details: we should limit the procedure to the installer, and move the furyctl-specific instructions in the furyctl documentation.

Change CIDRBlock in Node Pools Additional Firewall Rules from string to list

The provider support passing more than one CIDR block to the additional firewall rules of a node pool, but the installer lets you specify only one.

Change CIDRBlock in Node Pools Additional Firewall Rules from string to list:

fury-eks-installer/modules/eks/variables.tf

Line 60 in 5f3741c

cidr_block = string

Here it gets implicitly converted:

fury-eks-installer/modules/eks/ec2.tf

Line 34 in 5f3741c

cidr_blocks = [rule.cidr_block]

This is a breaking change.

Add vpc-and-vpn module example to the repository

Currently, the examples/ folder contains only an example for the eks module.
This assumes that the required networking infrastructure is already present.

It would be useful to show the usage of the vpc-and-vpn module as well and show how to create the networking infrastructure.

Replace hard-coded ami list with a data source in modules/vpn/locals.tf

Please replace the hardcoded list of ubuntu_amis in modules/vpn/locals.tf with a data source, and allow to select (cheaper) arm images for the vpn server.

Add cluster-level variable max_pods as defaults value for node pools

I believe that it would be useful to specify a cluster-level variable max_pods that act as default values for the node pools.

This value can be overwritten by the node_pools if necessary.

module "eks" {
  source          = ....
  cluster_name    = ...
  max_pods        = 100
  node_pools = [
    {
      name          = "infra"
      # ...
      # I will use `max_pods` = 100
    },
    {
      name          = "app"
      max_pods  = 10 # I will not use the default value
    },
  ]
}

I think the same approach can be followed for other variables if necessary.

As an example, each node_pool expects a tags variable that if not specified results in error.
I think this is cumbersome; we need to repeat (possibly) the same tags to all the node pools.
We could also promote the tags variable at the cluster level and have node pools inherit this value.

SpotInstances is marked as optional but installer breaks when not set

Even though the installer marks the node_pools' spot_instance as optional, if the parameter is not set the installer breaks with a Null condition error:

Error: Null condition

  on .terraform/modules/fury/modules/eks/eks.tf line 25, in locals:
  25: %{if lookup(worker, "max_pods", null) != null}--max-pods ${worker.max_pods} %{endif}--node-labels=sighup.io/cluster=${var.cluster_name},sighup.io/node_pool=${worker.name},%{for k, v in worker.labels}${k}=${v},%{endfor}${worker.spot_instance ? "node.kubernetes.io/lifecycle=spot" : ""}
    ├────────────────
    │ worker.spot_instance is null

The condition value is null. Conditions must either be true or false.

Either mark the field as required or optional but with a default (false) or handle the case when is not set.

EKS module installer update 1.18 & 1.19

For previous reference:
#36

upgrade to v18:
upgrade to v19:

https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-18.0.md
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-19.0.md

Update the architecture diagram

The image has some references to AKS, we should fix it. While we are at it, we should also do a graphical overhaul

Node Pool additional firewall rules don't work

When specify additional firewall rules in some node pool, the aws_security_group_rule will not been created.

Test EKS Installer against 1.24 version

AWS has released support for Kubernetes 1.24 in EKS. Test if the installer is capable of creating a 1.24 cluster.

Create issues with the eventual changes that need to be made.

AWS EKS Terraform module is Outdated

KFD EKS Installers is using version 1.16.x of the AWS EKS Terraform module, this version is outdated, latest upstream is already in 1.18.

upgrade to v17: #48

Warning there are several breaking changes, see:
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-17.0.md

Test EKS Installer against 1.25 version

Check if the installer uses Auto Scaling launch configurations

check if in the Terraform module we use under the hood launch templates.
because launch configuration are going to be dismissed december 1

Add support to change container-runtime

AWS EKS supports changing the container runtime from dockershim to containerd by passing an extra boot flag --container-runtime containerd to test containerd before it is the default.

The installer currently does not support setting this flag. See:

fury-eks-installer/modules/eks/eks.tf

Line 23 in 474bdd2

    
           "bootstrap_extra_args" : "%{if lookup(worker, "max_pods", null) != null}--use-max-pods false%{endif}",

References:
https://aws.amazon.com/blogs/containers/amazon-eks-1-21-released/
https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html

Introduce EKS add-ons

Upgrading an EKS cluster does not update core components that come with the cluster such as vpc-cni, coredns and kube-proxy.

In the past, this required to pull the updated Kubernetes resources for such components and applying them to the cluster. Now, AWS introduced EKS add-ons to address this issue thus allowing the upgrade of such components in a straightforward way.

I think we should add this feature (available since EKS v1.18) to improve day-2 operations management.

Ref:

https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html

Todo list:

Remove from furyctl the patch on coredns (in the tf kubernetes project), and move the tolerations/selector using terraform variables
Relax aws provider requirement on all tf modules on core modules
Test the upgrade between 1.25 to 1.26 (the target version for eks addons is 1.26)
- Check storageclasses while using the add-ons EBS csi driver (the ebs one should be default)

edit cloudwatch log retention in cluster.yml

Is there any way to properly set the Cloudwatch log group retention days via ‘cluster.yaml’ file?
It seems that the variable for such feature is not exposed and the Fury module set is as a default of 90 days.

Problem with max_pods

Hi team, when using EKS clusters, makes no sense to set max_pods because the size of the ec2 instance implies the maximum number of ENI that can be attached to a node, see:
https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt

We should use this list, maybe create a map and use the value from the map to set the max pods for the nodes

sighupio / fury-eks-installer Goto Github PK

fury-eks-installer's People

Contributors

Stargazers

Watchers

Forkers

fury-eks-installer's Issues

Recommend Projects

Recommend Topics

Recommend Org