sighupio / fury-eks-installer Goto Github PK
View Code? Open in Web Editor NEWFury Kubernetes Installer - EKS (AWS Kubernetes Managed Service)
Home Page: https://kubernetesfury.com
License: BSD 3-Clause "New" or "Revised" License
Fury Kubernetes Installer - EKS (AWS Kubernetes Managed Service)
Home Page: https://kubernetesfury.com
License: BSD 3-Clause "New" or "Revised" License
With a new AWS account, a new permission is needed to create a load balancer.
The 12.2.0 introduced the needed patch to the policy: terraform-aws-modules/terraform-aws-eks#902
Should be useful to update asap to this version.
Thanks!
cat main.tf
/**
* Copyright (c) 2017-present SIGHUP s.r.l All rights reserved.
* Use of this source code is governed by a BSD-style
* license that can be found in the LICENSE file.
*/
terraform {
experiments = [module_variable_optional_attrs]
backend "s3" {
bucket = "furyctl-issue-196"
key = "barebone/use1/cluster.json"
region = "us-east-1"
}
required_providers {
aws = {
source = "hashicorp/aws"
}
kubernetes = {
source = "hashicorp/kubernetes"
}
}
}
provider "aws" {
region = "us-east-1"
default_tags {
tags = {
env = "product-day-qa"
githubIssue = "303"
k8s = "eks-barebone"
}
}
}
provider "kubernetes" {
host = data.aws_eks_cluster.fury.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.fury.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.fury.token
}
module "fury" {
source = "~/.furyctl/eks-barebone/vendor/installers/eks/modules/eks"
cluster_name = var.cluster_name
cluster_version = var.cluster_version
cluster_log_retention_days = var.cluster_log_retention_days
network = var.network
subnetworks = var.subnetworks
dmz_cidr_range = var.dmz_cidr_range
ssh_public_key = var.ssh_public_key
node_pools = var.node_pools
node_pools_launch_kind = var.node_pools_launch_kind
tags = var.tags
# Specific AWS variables.
# Enables managing auth using these variables
eks_map_users = var.eks_map_users
eks_map_roles = var.eks_map_roles
eks_map_accounts = var.eks_map_accounts
}
cat .terraform/modules/fury/kubernetes.tf
data "aws_eks_cluster" "cluster" {
name = module.cluster.cluster_id
}
data "aws_eks_cluster_auth" "cluster" {
name = module.cluster.cluster_id
}
provider "kubernetes" {
host = data.aws_eks_cluster.cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.cluster.token
load_config_file = false
}
terraform apply
module.fury.module.cluster.aws_cloudwatch_log_group.this[0]: Refreshing state... [id=/aws/eks/eks-barebone/cluster]
module.fury.aws_key_pair.nodes: Refreshing state... [id=eks-barebone-20230301083258319400000001]
module.fury.module.cluster.aws_security_group.cluster[0]: Refreshing state... [id=sg-0f0046bd94eebbe6d]
module.fury.aws_security_group.nodes: Refreshing state... [id=sg-0905fc5bc2b95d013]
module.fury.module.cluster.aws_security_group.workers[0]: Refreshing state... [id=sg-0e4e1c5b331e9c995]
module.fury.aws_security_group.node_pool[2]: Refreshing state... [id=sg-096878a7e921d4246]
module.fury.aws_security_group.node_pool[1]: Refreshing state... [id=sg-020ae1b0b10ac4ea5]
module.fury.aws_security_group.node_pool[0]: Refreshing state... [id=sg-0976063af499fbac7]
module.fury.module.cluster.aws_iam_policy.cluster_elb_sl_role_creation[0]: Refreshing state... [id=arn:aws:iam::492816857163:policy/eks-barebone/eks-barebone-elb-sl-role-creation20230301083258320600000002]
module.fury.module.cluster.aws_iam_role.cluster[0]: Refreshing state... [id=eks-barebone20230301083258320800000003]
module.fury.aws_security_group_rule.ssh_from_dmz_to_nodes: Refreshing state... [id=sgrule-2968004512]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_self[0]: Refreshing state... [id=sgrule-2573013755]
module.fury.module.cluster.aws_security_group_rule.workers_egress_internet[0]: Refreshing state... [id=sgrule-2558094647]
module.fury.module.cluster.aws_security_group_rule.cluster_https_worker_ingress[0]: Refreshing state... [id=sgrule-1596022399]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_cluster_https[0]: Refreshing state... [id=sgrule-4230566281]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_cluster[0]: Refreshing state... [id=sgrule-2751700965]
module.fury.module.cluster.aws_security_group_rule.cluster_egress_internet[0]: Refreshing state... [id=sgrule-4246159717]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSVPCResourceControllerPolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-2023030108330150460000000a]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_elb_sl_role_creation[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301493800000008]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301481900000007]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSServicePolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301500700000009]
module.fury.module.cluster.aws_eks_cluster.this[0]: Refreshing state... [id=eks-barebone]
module.fury.module.cluster.aws_iam_role.workers[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEKS_CNI_Policy[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-20230301084516827400000011]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEKSWorkerNodePolicy[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-2023030108451680890000000f]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEC2ContainerRegistryReadOnly[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-20230301084516808900000010]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[0]: Refreshing state... [id=eks-barebone2023030108451615630000000d]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[2]: Refreshing state... [id=eks-barebone2023030108451615640000000e]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[1]: Refreshing state... [id=eks-barebone2023030108451615610000000c]
module.fury.module.cluster.aws_launch_template.workers_launch_template[1]: Refreshing state... [id=lt-059ae5d3ea75f17ba]
module.fury.module.cluster.aws_launch_template.workers_launch_template[0]: Refreshing state... [id=lt-06a755105d5409348]
module.fury.module.cluster.kubernetes_config_map.aws_auth[0]: Refreshing state... [id=kube-system/aws-auth]
module.fury.module.cluster.aws_launch_template.workers_launch_template[2]: Refreshing state... [id=lt-0292a97d1426825e2]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[0]: Refreshing state... [id=eks-barebone-infra20230301084520960500000019]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[2]: Refreshing state... [id=eks-barebone-app20230301084520960500000018]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[1]: Refreshing state... [id=eks-barebone-ingress2023030108452096060000001a]
╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│
│ on main.tf line 8, in terraform:
│ 8: experiments = [module_variable_optional_attrs]
│
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
│
│ (and one more similar warning elsewhere)
╵
╷
│ Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp [::1]:80: connect: connection refused
│
│ with module.fury.module.cluster.kubernetes_config_map.aws_auth[0],
│ on .terraform/modules/fury.cluster/aws_auth.tf line 65, in resource "kubernetes_config_map" "aws_auth":
│ 65: resource "kubernetes_config_map" "aws_auth" {
│
╵
Removing provider from module, terraform use the right cluster endpoint.
cat .terraform/modules/fury/kubernetes.tf
data "aws_eks_cluster" "cluster" {
name = module.cluster.cluster_id
}
❯ terraform apply
module.fury.module.cluster.aws_cloudwatch_log_group.this[0]: Refreshing state... [id=/aws/eks/eks-barebone/cluster]
module.fury.module.cluster.aws_security_group.cluster[0]: Refreshing state... [id=sg-0f0046bd94eebbe6d]
module.fury.aws_security_group.nodes: Refreshing state... [id=sg-0905fc5bc2b95d013]
module.fury.aws_key_pair.nodes: Refreshing state... [id=eks-barebone-20230301083258319400000001]
module.fury.aws_security_group.node_pool[0]: Refreshing state... [id=sg-0976063af499fbac7]
module.fury.module.cluster.aws_security_group.workers[0]: Refreshing state... [id=sg-0e4e1c5b331e9c995]
module.fury.aws_security_group.node_pool[1]: Refreshing state... [id=sg-020ae1b0b10ac4ea5]
module.fury.aws_security_group.node_pool[2]: Refreshing state... [id=sg-096878a7e921d4246]
module.fury.module.cluster.aws_iam_policy.cluster_elb_sl_role_creation[0]: Refreshing state... [id=arn:aws:iam::492816857163:policy/eks-barebone/eks-barebone-elb-sl-role-creation20230301083258320600000002]
module.fury.module.cluster.aws_iam_role.cluster[0]: Refreshing state... [id=eks-barebone20230301083258320800000003]
module.fury.aws_security_group_rule.ssh_from_dmz_to_nodes: Refreshing state... [id=sgrule-2968004512]
module.fury.module.cluster.aws_security_group_rule.cluster_egress_internet[0]: Refreshing state... [id=sgrule-4246159717]
module.fury.module.cluster.aws_security_group_rule.cluster_https_worker_ingress[0]: Refreshing state... [id=sgrule-1596022399]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_self[0]: Refreshing state... [id=sgrule-2573013755]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_cluster_https[0]: Refreshing state... [id=sgrule-4230566281]
module.fury.module.cluster.aws_security_group_rule.workers_ingress_cluster[0]: Refreshing state... [id=sgrule-2751700965]
module.fury.module.cluster.aws_security_group_rule.workers_egress_internet[0]: Refreshing state... [id=sgrule-2558094647]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSServicePolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301500700000009]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSVPCResourceControllerPolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-2023030108330150460000000a]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301481900000007]
module.fury.module.cluster.aws_iam_role_policy_attachment.cluster_elb_sl_role_creation[0]: Refreshing state... [id=eks-barebone20230301083258320800000003-20230301083301493800000008]
module.fury.module.cluster.aws_eks_cluster.this[0]: Refreshing state... [id=eks-barebone]
module.fury.module.cluster.aws_iam_role.workers[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEKS_CNI_Policy[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-20230301084516827400000011]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEKSWorkerNodePolicy[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-2023030108451680890000000f]
module.fury.module.cluster.aws_iam_role_policy_attachment.workers_AmazonEC2ContainerRegistryReadOnly[0]: Refreshing state... [id=eks-barebone2023030108451346050000000b-20230301084516808900000010]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[0]: Refreshing state... [id=eks-barebone2023030108451615630000000d]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[2]: Refreshing state... [id=eks-barebone2023030108451615640000000e]
module.fury.module.cluster.aws_iam_instance_profile.workers_launch_template[1]: Refreshing state... [id=eks-barebone2023030108451615610000000c]
module.fury.module.cluster.aws_launch_template.workers_launch_template[1]: Refreshing state... [id=lt-059ae5d3ea75f17ba]
module.fury.module.cluster.aws_launch_template.workers_launch_template[2]: Refreshing state... [id=lt-0292a97d1426825e2]
module.fury.module.cluster.aws_launch_template.workers_launch_template[0]: Refreshing state... [id=lt-06a755105d5409348]
module.fury.module.cluster.kubernetes_config_map.aws_auth[0]: Refreshing state... [id=kube-system/aws-auth]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[0]: Refreshing state... [id=eks-barebone-infra20230301084520960500000019]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[2]: Refreshing state... [id=eks-barebone-app20230301084520960500000018]
module.fury.module.cluster.aws_autoscaling_group.workers_launch_template[1]: Refreshing state... [id=eks-barebone-ingress2023030108452096060000001a]
╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│
│ on main.tf line 8, in terraform:
│ 8: experiments = [module_variable_optional_attrs]
│
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
│
│ (and one more similar warning elsewhere)
╵
╷
│ Error: Get "https://BEF030853566ED2382C53403F9949892.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 10.10.9.152:443: i/o timeout
│
│ with module.fury.module.cluster.kubernetes_config_map.aws_auth[0],
│ on .terraform/modules/fury.cluster/aws_auth.tf line 65, in resource "kubernetes_config_map" "aws_auth":
│ 65: resource "kubernetes_config_map" "aws_auth" {
│
╵
To further improve the module documentation:
Please write the full upgrade guide for the installer.
It was initially described here, but later works caused that guide to go out of sync, so we should update the whole thing once it's well tested.
Keep in mind that the guide above was including a lot of furyctl details: we should limit the procedure to the installer, and move the furyctl-specific instructions in the furyctl documentation.
The provider support passing more than one CIDR block to the additional firewall rules of a node pool, but the installer lets you specify only one.
Change CIDRBlock in Node Pools Additional Firewall Rules from string to list:
Here it gets implicitly converted:
fury-eks-installer/modules/eks/ec2.tf
Line 34 in 5f3741c
This is a breaking change.
Currently, the examples/
folder contains only an example for the eks
module.
This assumes that the required networking infrastructure is already present.
It would be useful to show the usage of the vpc-and-vpn
module as well and show how to create the networking infrastructure.
Please replace the hardcoded list of ubuntu_amis in modules/vpn/locals.tf
with a data source, and allow to select (cheaper) arm images for the vpn server.
I believe that it would be useful to specify a cluster-level variable max_pods
that act as default values for the node pools.
This value can be overwritten by the node_pools
if necessary.
module "eks" {
source = ....
cluster_name = ...
max_pods = 100
node_pools = [
{
name = "infra"
# ...
# I will use `max_pods` = 100
},
{
name = "app"
max_pods = 10 # I will not use the default value
},
]
}
I think the same approach can be followed for other variables if necessary.
As an example, each node_pool
expects a tags
variable that if not specified results in error.
I think this is cumbersome; we need to repeat (possibly) the same tags to all the node pools.
We could also promote the tags
variable at the cluster level and have node pools inherit this value.
Even though the installer marks the node_pools' spot_instance
as optional, if the parameter is not set the installer breaks with a Null condition error:
Error: Null condition
on .terraform/modules/fury/modules/eks/eks.tf line 25, in locals:
25: %{if lookup(worker, "max_pods", null) != null}--max-pods ${worker.max_pods} %{endif}--node-labels=sighup.io/cluster=${var.cluster_name},sighup.io/node_pool=${worker.name},%{for k, v in worker.labels}${k}=${v},%{endfor}${worker.spot_instance ? "node.kubernetes.io/lifecycle=spot" : ""}
├────────────────
│ worker.spot_instance is null
The condition value is null. Conditions must either be true or false.
Either mark the field as required or optional but with a default (false
) or handle the case when is not set.
For previous reference:
#36
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-18.0.md
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-19.0.md
The image has some references to AKS, we should fix it. While we are at it, we should also do a graphical overhaul
When specify additional firewall rules in some node pool, the aws_security_group_rule will not been created.
AWS has released support for Kubernetes 1.24 in EKS. Test if the installer is capable of creating a 1.24 cluster.
Create issues with the eventual changes that need to be made.
KFD EKS Installers is using version 1.16.x of the AWS EKS Terraform module, this version is outdated, latest upstream is already in 1.18.
Warning there are several breaking changes, see:
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-17.0.md
check if in the Terraform module we use under the hood launch templates.
because launch configuration are going to be dismissed december 1
AWS EKS supports changing the container runtime from dockershim
to containerd
by passing an extra boot flag --container-runtime containerd
to test containerd before it is the default.
The installer currently does not support setting this flag. See:
fury-eks-installer/modules/eks/eks.tf
Line 23 in 474bdd2
References:
https://aws.amazon.com/blogs/containers/amazon-eks-1-21-released/
https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html
Upgrading an EKS cluster does not update core components that come with the cluster such as vpc-cni, coredns and kube-proxy.
In the past, this required to pull the updated Kubernetes resources for such components and applying them to the cluster. Now, AWS introduced EKS add-ons to address this issue thus allowing the upgrade of such components in a straightforward way.
I think we should add this feature (available since EKS v1.18) to improve day-2 operations management.
Ref:
Todo list:
Is there any way to properly set the Cloudwatch log group retention days via ‘cluster.yaml’ file?
It seems that the variable for such feature is not exposed and the Fury module set is as a default of 90 days.
Hi team, when using EKS clusters, makes no sense to set max_pods because the size of the ec2 instance implies the maximum number of ENI that can be attached to a node, see:
https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt
We should use this list, maybe create a map and use the value from the map to set the max pods for the nodes
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.