A sample project that deploys an EKS Cluster following a set of best practices with options to install additional addons. Easy deployment of the EBS CSI Driver, EFS CSI Driver, FluentBit Centralized Logging using Cloudwatch, Cluster Autoscaler, ALB Ingress Controller, Secrets CSI Driver and Network Policy Engine.

• AWS CDK >= 2.3.0 check Getting Started with AWS CDK to setup your CDK environment. Run cdk --version to check the CLI version.

git clone https://github.com/aws-samples/amazon-eks-using-cdk-typescript.git
# install dependant packages
npm install
# If you have not used cdk before, you may be advised to create cdk resources
cdk bootstrap aws://ACCOUNT_ID/REGION
# check the diff before deployment to understand any changes, on first run all resources will created
cdk diff
# Deploy the stack, you will be prompted for confirmation for creation of IAM and Security Group resources
cdk -c cluster_name=myfirstcluster deploy --all

Go to the EC2 Console and look for an instance Name: EKSBastionHost. Right click on the instance and select connect

Select Session Manager and click the Connect button

In the new window run the following command to configure kubectl with EKS Cluster. Replace CLUSTER_NAME and REGION_NAME with the proper values.

aws eks update-kubeconfig --name CLUSTER_NAME --region REGION_NAME

To destroy the cluster run the command below

cdk destroy -c cluster_name=myfirstcluster --all

cdk deploy -c cluster_name=mycluster --all

cdk deploy -c cluster_name=mycluster EKSStack

cdk deploy -c cluster_name=mycluster -c stack_prefix="dev-" --all

All cdk stacks will contain prefix ex. dev-EKSStack dev-EKSK8sBaseline

Tag existing VPC with proper EKS Tags for ALB/NLB auto discovery

cdk deploy -c use_vpc_id=vpc-1234567 -c cluster_name=mycluster --all

Requires VPC to have private subnets

cdk deploy -c cluster_name=mycluster --all --parameter EKSK8sBaseline:ebsDriverParameter=true --parameter EKSK8sBaseline:albDriverParameter=false

If using stack_prefix context, add prefix to parameters. ex dev-EKSK8sBaseline:ebsDriverParameter=true

Parameters can be configured via Cloudformation console or via cdk deploy --parameter STACK_NAME:PARAMETER_NAME=PARAMETER_VALUE

Context can only be configured at synthesis time via cdk deploy -c CONTEXT_NAME=CONTEXT_VALUE or editing cdk.json file.

*Run aws eks describe-addon-versions --kubernetes-version 1.21 to see full compatible list

Provides an example of EKS best practices in Infrastructure as Code

• Make the EKS Cluster Endpoint private
• IAM Roles for Service Accounts(IRSA) for deployable addons (ex. ALB Ingress Controller)
• Update the aws-node daemonset to use IRSA via EKS AddOns
• Option to enable Audit Logs
• Option to enable network policies using Calico engine
• Use AWS KMS for envelope encryption of kubernetes secrets
• Minimize access to worker node use SSM Manager
• Deploy workers onto private subnets*
• ECR Private Endpoints*
• Option to enable Cluster Autoscaler
• EKS Console Read Only Group called eks-console-dashboard-full-access-group
• EBS Encryption of Worker and Bastion Instances

*Use of vpc created by stack

Helm Chart versions can be configured by editing the cdk.json before deployment. The versions can be listed by using helm commands. Example for Cluster Autoscaler:

helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm search repo autoscaler --versions

Another way to identity Helm Chart versions is looking at that index.yaml of Chart repositories for version information.

Deploys Cluster Autoscaler chart version 9.10.3 with app version 1.21.0 using Helm chart. A full list of chart versions with app versions can be found here

Default configuration below, additional configurations can be edited in k8s-baseline.ts:

       extraArgs: {
          // https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-are-the-parameters-to-ca
          'skip-nodes-with-system-pods': false,
          'skip-nodes-with-local-storage': false,
          'balance-similar-node-groups': true,
          //How long a node should be unneeded before it is eligible for scale down
          'scale-down-unneeded-time' : '300s',
          //How long after scale up that scale down evaluation resumes
          'scale-down-delay-after-add':'300s'
        }

Deploys ALB Ingress Controller chart version 1.2.3 with app version 2.2.1 using Helm. A full list of chart versions with app versions can be found here

Deploys EBS CSI Driver Chart version 2.0.1 with app version 1.1.1 using Helm. A full list of chart versions with app versions can be found here

Needs user to deploy EBS StorageClass for use with pods. Example steps in EBS CSI Driver docs

Deploys EFS CSI Driver chart version 2.1.4 with app version 1.3.2 using Helm. A full list of chart versions with app versions can be found here

Needs user to create EFS Volume and provision a storageclass for use with pods. Example steps in EFS CSI Driver docs

Deploys the Secrets CSI Driver chart version 0.1.0 with app version 0.1.0 using Helm.A full list of chart versions with app versions can be found here

Requires pods to have ServiceAccount with proper permissions to get values from AWS Secrets Manager or AWS Parameter Store. Example steps in AWS Blog post, continue from step 4.

Automated rotation for the driver using the rotation reconciler is turned off by default. Uncomment the below lines in secretsCsiHelmChart k8s-baseline.ts

      values: {
        grpcSupportedProviders: 'aws',
        // alpha feature reconciler feature
        rotationPollInterval: 3600
        enableSecretRotation: true
      },

Deploys Calico Network Policy Engine chart version 0.3.5 with app version 1.15.1 using Helm.

Increases default cpu/memory limits to be

            resources: {
              limits: {
                memory: '256Mi',
                cpu: '500m',
              },
            },

Deploys Container Insights on Amazon EKS Cluster by using the AWS Distro for OpenTelemetry (collectoramazon/aws-otel-collector:v0.11.0). The visualized metrics can be found on Cloudwatch -> Insights -> Container Insights. The logs can be found at /aws/containerinsights/CLUSTER_NAME/performance. List of available metrics that can be collected available here

EKS Pod Metric Example:

Configurations are applied via manifest, edits can be made following the documentation

Deploys FluentBit daemonset using manifests. The configuration is found in the manifest. The FluentBit conf files are stored in ConfigMap k8s definition. Additional configuration like log_retention_days, log_format, role_arn(cross-account) can be configured by editing conf files.

IAM permission to read/write permission to log group: /aws/containerinsights/${CLUSTER_NAME}. Editable in k8s-baseline.ts

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:${REGION}:${ACCOUNT_ID}:log-group:/aws/containerinsights/${CLUSTER_NAME}/*",
                "arn:aws:logs:${REGION}:${ACCOUNT}:log-group:/aws/containerinsights/${CLUSTER_NAME}/*:log-stream:*"
            ],
            "Effect": "Allow"
        }
    ]
}

Include Logs from:

/var/log/containers/*.log

Exclude Logs from:

/var/log/containers/aws-node*
/var/log/containers/kube-proxy*

Output to:

Cloudwatch Log group /aws/containerinsights/${CLUSTER_NAME}/application

Include Logs from:

systemd:docker.service
systemd:kubelet.service
/var/log/containers/aws-node*
/var/log/containers/kube-proxy*

Exclude Logs from:

N/A

Output to:

Cloudwatch Log group /aws/containerinsights/${CLUSTER_NAME}/dataplane

Include Logs from:

/var/log/secure
/var/log/messages
/var/log/dmesg

Exclude Logs from:

N/A

Output to:

Cloudwatch Log group /aws/containerinsights/${CLUSTER_NAME}/host

A list of recommendations to further enhance the environment.

A readonly K8s Group is created called eks-console-dashboard-full-access-group. Adding the below code snippet to the sample will add an existing Role to view eks-console-dashboard-full-access-group RBAC group. See eks-cdk-js-stack.ts for example.

// Add existing IAM Role to Custom Group
this.awsauth.addRoleMapping(
  Role.fromRoleArn(
    this,
    "Role_Admin",
    `arn:aws:iam::${this.account}:role/Admin`
  ),
  {
    groups: ["eks-console-dashboard-full-access-group"],
    username: `arn:aws:iam::${this.account}:role/Admin/{{SessionName}}`,
  }
);

// Add existing IAM User to Custom Group
this.awsauth.addUserMapping(
  User.fromUserArn(
    this,
    "Role_Admin",
    `arn:aws:iam::${this.account}:user/Admin`
  ),
  {
    groups: ["eks-console-dashboard-full-access-group"],
    username: `arn:aws:iam::${this.account}:user/Admin`,
  }
);

The error below can be resolved by adding the iam user/role to eks-console-dashboard-full-access-group k8s group.

The sample deploys a bastion host with outbound to 0.0.0.0 with outbound access on port 443. The IAM role associated with the ec2 instance is registered as Admin role for EKS Cluster. SSM is able to log session data to Cloudwatch Logs or S3, see docs for more details. The end user permissions can be scoped to specific instance ids or tags.

Each pods requires a Service Account with IAM Role to access AWS Secrets Manager or AWS Parameter Store resources.

Sample permissions scoped to Secrets/Parameters with tags ekscluster:myclustername for dynamic access permissions.

const secretsManagerPolicyStatement = new PolicyStatement({
  resources: ["*"],
  actions: ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
  conditions: {
    StringEquals: {
      "secretsmanager:ResourceTag/ekscluster": `${props.eksCluster.clusterName}`,
    },
  },
});
const parameterManagerPolicyStatement = new PolicyStatement({
  resources: ["*"],
  actions: ["ssm:GetParameters"],
  conditions: {
    StringEquals: {
      "ssm:ResourceTag/ekscluster": `${props.eksCluster.clusterName}`,
    },
  },
});
const awsSA = new ServiceAccount(this, "awsSA", {
  cluster: props.eksCluster,
  name: "my-deployment",
  namespace: "default",
});
const secretsManagerPolicy = new Policy(this, "secrets-manager-policy", {
  statements: [parameterManagerPolicyStatement, secretsManagerPolicyStatement],
  policyName: `${props.eksCluster.clusterName}-secrets`,
  roles: [awsSA.role],
});

The sample deploys Managed Node group with IMDSv1 and IMDSv2 enabled with hop count: 2 to ensure broadly levels of compatibility.

Current config:

        // Not all components are IMDSv2 aware. Ex. Fluentbit
        metadataOptions: {
          httpTokens: 'optional',
          httpPutResponseHopLimit: 2,

        },

To enforce IMDSv2 only, change to below code.

        metadataOptions: {
          httpTokens: 'required',
          httpPutResponseHopLimit: 1,
        },

The configuration for the FluentBit is set to create LogGroup and LogStreams with log retention=forever. The full set of options can be found here.

A sample output snippet for cross-account and log retention days (30 days).

    [OUTPUT]
        Name                cloudwatch_logs
        Match               application.*
        region              ${AWS_REGION}
        log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
        log_stream_prefix   ${HOST_NAME}-
        auto_create_group   true
        extra_user_agent    container-insights
        log_retention_days  30
        role_arn            arn:aws:iam::123456789:role/role-name

See CONTRIBUTING for more information.

This library is licensed under the MIT-0 License. See the LICENSE file.