This repository contains resources and configuration scripts for building a custom Amazon EKS AMI with HashiCorp Packer. This is the same configuration that Amazon EKS uses to create the official Amazon EKS-optimized AMI.
You must have Packer installed on your local system. For more information, see Installing Packer in the Packer documentation. You must also have AWS account credentials configured so that Packer can make calls to AWS API operations on your behalf. For more information, see Authentication in the Packer documentation.
Note The default instance type to build this AMI does not qualify for the AWS free tier. You are charged for any instances created when building this AMI.
A Makefile is provided to build the Amazon EKS Worker AMI, but it is just a small wrapper around invoking Packer directly. You can initiate the build process by running the following command in the root of this repository:
make
The Makefile chooses a particular kubelet binary to use per kubernetes version which you can view here. To build an Amazon EKS Worker AMI for a particular Kubernetes version run the following command
make 1.23 ## Build a Amazon EKS Worker AMI for k8s 1.23
To build an Amazon EKS Worker AMI with other versions of Kubernetes that are not listed above run the following AWS Command Line Interface (AWS CLI) commands to obtain values for KUBERNETES_VERSION, KUBERNETES_BUILD_DATE, PLATFORM, ARCH from S3
#List of all avalable Kuberenets Versions:
aws s3 ls s3://amazon-eks
KUBERNETES_VERSION=1.23.9 # Chose a version and set the variable
#List of all builds for the specified Kubernetes Version:
aws s3 ls s3://amazon-eks/$KUBERNETES_VERSION/
KUBERNETES_BUILD_DATE=2022-07-27 # Chose a date and set the variable
#List of all platforms available for the selected Kubernetes Version and build date
aws s3 ls s3://amazon-eks/$KUBERNETES_VERSION/$KUBERNETES_BUILD_DATE/bin/
PLATFORM=linux # Chose a platform and set the variable
#List of all architectures for the selected Kubernetes Version, build date and platform
aws s3 ls s3://amazon-eks/$KUBERNETES_VERSION/$KUBERNETES_BUILD_DATE/bin/linux/
ARCH=amd64 #Chose an architecture and set the variable
Run the following command to build an Amazon EKS Worker AMI based on the chosen parameters in the previous step
make k8s \
kubernetes_version=$KUBERNETES_VERSION \
kubernetes_build_date=$KUBERNETES_BUILD_DATE \
arch=$ARCH
By default, binaries are downloaded from the Amazon EKS public Amazon Simple Storage Service (Amazon S3) bucket amazon-eks in us-west-2. You can instead choose to provide your own version of Kubernetes binaries to be used. To use your own binaries
- Copy the binaries to your own S3 bucket using the AWS CLI. Here is an example that uses Kubelet binary
aws s3 cp kubelet s3://my-custom-bucket/kubernetes_version/kubernetes_build_date/bin/linux/arch/kubelet
Note: Replace my-custom-bucket, amazon-eks, kubernetes_version, kubernetes_build_date, and arch with your values.
Important: You must provide all the binaries listed in the default amazon-eks bucket for a specific kubernetes_version, kubernetes_build_date, and arch combination. These binaries must be accessible through AWS Identity and Access Management (IAM) credentials configured in the Install and configure HashiCorp Packer section.
- Run the following command to start the build process to use your own Kubernetes binaries
make k8s \
binary_bucket_name=my-custom-bucket \
binary_bucket_region=eu-west-1 \
kubernetes_version=1.14.9 \
kubernetes_build_date=2020-01-22
Note: Confirm that the binary_bucket_name, binary_bucket_region, kubernetes_version, and kubernetes_build_date parameters match the path to your binaries in Amazon S3.
The Makefile runs Packer with the eks-worker-al2.json
build specification
template and the amazon-ebs
builder. An instance is launched and the Packer Shell
Provisioner runs the
install-worker.sh
script on the instance to install software and perform other
necessary configuration tasks. Then, Packer creates an AMI from the instance
and terminates the instance after the AMI is created.
If you are just getting started with Amazon EKS, we recommend that you follow our Getting Started chapter in the Amazon EKS User Guide. If you already have a cluster, and you want to launch a node group with your new AMI, see Launching Amazon EKS Worker Nodes in the Amazon EKS User Guide.
By default, the amazon-eks-ami
uses a source_ami_filter that selects the latest hvm AL2 AMI for the given architecture as the base AMI. For more information on what kernel versions are running on published Amazon EKS optimized Linux AMIs, see the public documentation.
When building an AMI, you can set the kernel_version
to 4.14
or 5.4
to customize the kernel version. The upgrade_kernel.sh script contains the logic for updating and upgrading the kernel. For Kubernetes versions 1.18 and below, it uses the 4.14
kernel if not set, and it will install the latest patches. For Kubernetes version 1.19 and above, it uses the 5.4
kernel if not set.
In some cases, customers may want to customize the kubelet configuration on their nodes, and there are two mechanisms to do that with the EKS Optimized AMI.
Set the "--kubelet-extra-args" flag when invoking bootstrap.sh
bootstrap.sh
, the script that bootstraps nodes when using the EKS Optimized AMI, supports a flag called --kubelet-extra-args
that allows you to pass in additional kubelet
configuration. If you invoke the bootstrap script yourself (self-managed nodegroups or EKS managed nodegroups with custom AMIs), you can use that to customize your configuration. For example, you can use something like the following in your userdata:
/etc/eks/bootstrap.sh my-cluster --kubelet-extra-args '--registry-qps=20 --registry-burst=40'
In this case, it will set registryPullQPS
to 20 and registryBurst
to 40 in kubelet
. Some of the flags, like the ones above, are marked as deprecated and you're encouraged to set them in the kubelet
config file (described below), but they continue to work as of 1.23.
Update the kubelet config file
You can update the kubelet
config file directly with new configuration. On EKS Optimized AMIs, the file is stored at /etc/kubernetes/kubelet/kubelet-config.json
. It must be valid JSON. You can use a utility like jq
(or your tool of choice) to edit the config in your user data:
echo "$(jq ".registryPullQPS=20 | .registryBurst=40" /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json
There are a couple of important caveats here:
- If you update the
kubelet
config file afterkubelet
has already started (i.e.bootstrap.sh
already ran), you'll need to restartkubelet
to pick up the latest configuration. - bootstrap.sh does modify a few fields, like
kubeReserved
andevictionHard
, so you'd need to modify the config after the bootstrap script is run and restartkubelet
to overwrite those properties.
View active kubelet config
When kubelet
starts up, it logs all possible flags, including unset flags. The unset flags get logged with default values. These logs do not necessarily reflect the actual active configuration. This has caused confusion in the past when customers have configured the kubelet
config file with one value and notice the default value is logged. Here is an example of the referenced log:
Aug 16 21:53:49 ip-192-168-92-220.us-east-2.compute.internal kubelet[3935]: I0816 21:53:49.202824 3935 flags.go:59] FLAG: --registry-burst="10"
Aug 16 21:53:49 ip-192-168-92-220.us-east-2.compute.internal kubelet[3935]: I0816 21:53:49.202829 3935 flags.go:59] FLAG: --registry-qps="5"
To view the actual kubelet
config on your node, you can use the Kubernetes API to confirm that your configuration has applied.
$ kubectl proxy
$ curl -sSL "http://localhost:8001/api/v1/nodes/ip-192-168-92-220.us-east-2.compute.internal/proxy/configz" | jq
{
"kubeletconfig": {
...
"registryPullQPS": 20,
"registryBurst": 40,
...
}
}
For security issues or concerns, please do not open an issue or pull request on GitHub. Please report any suspected or confirmed security issues to AWS Security https://aws.amazon.com/security/vulnerability-reporting/
This sample code is made available under a modified MIT license. See the LICENSE file.