In this guide, you'll learn about the step-by-step process to create a Kubernetes cluster on AWS using Kops. This project demonstrates a Kubernetes setup with a full-stack application consisting of a ReactJS frontend, NodeJS backend, and MongoDB database. The project also includes a robust monitoring system using Prometheus, Grafana, kube-state-metrics, and AlertManager.
- Frontend: ReactJS (Deployment and Service)
- Backend: NodeJS (Deployment and Service)
- Database: MongoDB (StatefulSet for data persistence and Service)
- Volumes:
- PVC for MongoDB (AWS EBS gp2)
- Monitoring:
- Prometheus
- Grafana
- kube-state-metrics
- Alert Manager
- Security: Ingress network policy for MongoDB to only allow traffic from backend pods
- An AWS account
- Basic knowledge of AWS and Kubernetes
- AWS CLI configured with appropriate permissions
- Domain managed in Route 53
- Prometheus and Grafana
1. Launch an EC2 Instance
Launch an EC2 instance with the t2.micro
instance type. You can either create a new instance or use an existing one. Ensure you have SSH access to the instance.
2. Install Kops
SSH into your EC2 and install Kops. Follow the installation instructions from the official Kops documentation here.
3. Update the System
Update the system packages.
sudo apt update -y
4. Create IAM Role and Attach it to EC2
Create an IAM role with necessary permissions and programmatic access. Attaching AdministratorAccess is recommended for simplicity. Attach the IAM role to your EC2 instance.
5. Create an S3 Bucket
Create an S3 bucket which will be used by Kops to store cluster state information.
6. Create a Hosted Zone in Route53
Create a hosted zone in Route53. If you already have a hosted zone, you can skip this step.
7. Install kubectl
Install kubectl by following the instructions here.
8. Create the Cluster
Run the following command to create your cluster. Update the placeholders with your specific values. Also change the clsuter name that end with your hosted zone name. Example. kubernetes.myhosteszone.com, kubernetes.mydomain.com, etc...
kops create cluster \
--name=k8s-cluster.example.com \
--state=s3://<your-s3-bucket> \
--zones=<your-availability-zone> \
--node-count=1 \
--node-size=t2.medium \
--node-volume-size=15 \
--control-plane-count=1 \
--control-plane-size=t2.medium \
--control-plane-volume-size=15 \
--dns-zone=<your-domain>
9. Update and Apply the Cluster Configuration
The previous command generates a preview of all the resources Kops will create. If the preview looks good, run the following command to create the resources. (Same like terraform plan(8th step) and terraform apply(9th step))
kops update cluster --name=k8s-cluster.example.com--yes --admin --state=s3://<your-s3-bucket>
This process may take 5-10 minutes.
10. Validate the Cluster
Validate that the cluster is ready.
kops validate cluster --name=k8s-cluster.example.com --state=s3://<your-s3-bucket>
![](https://private-user-images.githubusercontent.com/113757574/337239353-adc2d923-f7f0-44f2-a916-7f44a41bedad.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM3OTY5NDUsIm5iZiI6MTcyMzc5NjY0NSwicGF0aCI6Ii8xMTM3NTc1NzQvMzM3MjM5MzUzLWFkYzJkOTIzLWY3ZjAtNDRmMi1hOTE2LTdmNDRhNDFiZWRhZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxNlQwODI0MDVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02NGU3N2FjOTAzOThhMDllMWM0YjllMjIwYWI0NjlkMGRjNWRlYzYwYzI3NjhkODIzYzdhMDMyNTJlYjYwZTNmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.Fvb4RO0TIr6lGKeXDGdiOhzDwD1cr5yjgFCwKSMpmUQ)
Here is an example of the expected output for the command above. In the image, you'll find that both the control plane and node plane are ready and operational, each identified by its unique ID. Additionally, the image displays subnet details where EC2 instances have been deployed, providing comprehensive information about the network configuration and infrastructure setup.
Once the cluster is ready, follow these steps to deploy the application:
-
Clone the repository using the following command:
git clone https://github.com/Divya4242/aws-kops-kubernetes-fullstack-app.git
-
Apply the application deployment, service and statefulset configuration:
kubectl apply -f .
-
Apply the ingress controller and resource configuration:
cd ingress && kubectl apply -f .
![](https://private-user-images.githubusercontent.com/113757574/347336562-c6f35918-1535-4b2d-aad0-f69febe937dd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM3OTY5NDUsIm5iZiI6MTcyMzc5NjY0NSwicGF0aCI6Ii8xMTM3NTc1NzQvMzQ3MzM2NTYyLWM2ZjM1OTE4LTE1MzUtNGIyZC1hYWQwLWY2OWZlYmU5MzdkZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxNlQwODI0MDVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05ZjRiZDcxZjYxYWYyZDgwZmY0OGVhNWI4MDYyZWJlNmE0NGE5Njg5MmQxZGMyZGI4MzljNjg2MTk1Y2IxZTczJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.hwzJAif8TtoEwiu0MM5Vn12K8JLst32s9_UNHF1YMt4)
Here is a snapshot depicting the expected output of the commands mentioned above. The image shows that all necessary pods, as specified in the deployment file, have been successfully created, along with the essential services. To verify the deployment status of the application, simply open your web browser and visit the following URLs: โข <ec2-public-ip>:30006
for accessing the frontend, and โข <ec2-public-ip>:30007
for connecting to the backend.
Until this, we've successfully configured a Kops cluster, deployed Kubernetes resources, and established Kubernetes services.
Add NGINX Ingress Controller:
To expose your services using NGINX Ingress Controller, follow these steps:
-
Download the NGINX Ingress Controller manifest: (Optional)
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.1/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml
-
Go to ingress directory and edit the
nginx-ingress-controller.yaml
file and update the following configurations:- Replace
XXX.XXX.XXX/XX
with your VPC CIDR in use for the Kubernetes cluster underproxy-real-ip-cidr
. - Replace
arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX
with your ACM certificate ID.
- Replace
-
Deploy the NGINX Ingress Controller and NGINX INGRESS Resource:
cd ingress && kubectl apply -f .
-
Verify that the Ingress Controller is deployed successfully by checking the Kubernetes namespace:
kubectl get ns
You should see a namespace named ingress-nginx
as output.
Tip
The NGINX Ingress Controller is the runtime component that actively manages traffic, while the NGINX Ingress Resource File is a configuration artifact that defines the desired behavior for that traffic within the Kubernetes ecosystem. (same like we deploy nginx web server(nginx ingress controller) for serving content to user and reverse proxy and deployment.conf(nginx ingress resource file) in sites availiable that define the servername, listning port, etc.)
Obtain Load Balancer DNS
To find the DNS name of the Load Balancer:
- Run the following command and look for the EXTERNAL-IP of the ingress-nginx-controller service
kubectl get all -n ingress-nginx
![](https://private-user-images.githubusercontent.com/113757574/337536030-466978b1-5e99-488d-bf6f-35ee0468eafb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM3OTY5NDUsIm5iZiI6MTcyMzc5NjY0NSwicGF0aCI6Ii8xMTM3NTc1NzQvMzM3NTM2MDMwLTQ2Njk3OGIxLTVlOTktNDg4ZC1iZjZmLTM1ZWUwNDY4ZWFmYi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxNlQwODI0MDVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lNjJlMjAwZjU5NmUwNmVlMzg4OGVlZjllNzA5NDBiY2IzMzg5MDA2N2FmOWQzYWYzMWJhMGI2Zjc4YWYyNWRlJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.0IHAthP23mJDuHHjguUQT1U3Zy2kGA1rE3eIzjzER4s)
Here is a snapshot displaying the external IP address of the load balancer, identified by its DNS name. Additionally, the image confirms the creation of the ingress-nginx0-controller. Now, your next task involves copying this load balancer DNS name and pasting it into Route 53. Detailed instructions for this process are outlined in the subsequent steps to seamlessly integrate and manage DNS routing for your application's load balancer.
- Alternatively, you can also find the Load Balancer DNS name in the AWS Management Console under EC2 -> Load Balancers.
Configure DNS in Route 53:
Add a CNAME record in Route 53 to point to the Load Balancer DNS name:
-
Go to Route 53 and select your hosted zone.
-
Add a new record:
โข Name: The record name specified in your ingress-resource.yaml file.
โข Type: CNAME
โข Value: The DNS name of the AWS Load Balancer.
Cheers! ๐ You can now access your deployed website.
Open your browser and navigate to your domain or the appropriate service URL. Your application should be live and accessible.
-
Prometheus: Collects metrics from the Kubernetes cluster and applications.
-
Grafana: Visualizes the metrics collected by Prometheus.
-
kube-state-metrics: Exposes cluster-level metrics.
-
Alert Manager: Manages alerts sent by Prometheus. (For this demo, two alerts are configured)
1. Node not ready for more than 5 minutes 2. Node has less than 1 GB memory
cd monitoring && kubectl apply -f .
To check Montoring of the application, open Chrome and navigate to the following URLs:
โข <ec2-public-ip>:30010 # for the prometheus
โข <ec2-public-ip>:30013 # for the grafana
โข <ec2-public-ip>:30011 # for the kube-state-metrics
![](https://private-user-images.githubusercontent.com/113757574/346928446-9cbd9a78-d244-4dac-9d8a-e483bd78d54a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM3OTY5NDUsIm5iZiI6MTcyMzc5NjY0NSwicGF0aCI6Ii8xMTM3NTc1NzQvMzQ2OTI4NDQ2LTljYmQ5YTc4LWQyNDQtNGRhYy05ZDhhLWU0ODNiZDc4ZDU0YS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxNlQwODI0MDVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hZDdlNjM4ZDJlZWY4YjFmZDliMjYyY2I5ZjcyMDEzMzQ3MGQ5MmZkMGNkZGQyMWMwMDZkMDM3MGNiMmNjM2M3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.41vvahLMdaxBmiteiO0TazY-K3-RQChFlqC6T7j9j94)
-
- Ensure all targets in Prometheus are up, as shown in the image. In the targets section, you can see there are two endpoints: one for
kube-state-metrics
which collects Kubernetes data, and another for Prometheus metrics. It is crucial to ensure that both targets are operational to maintain accurate and comprehensive monitoring of your Kubernetes cluster. Ensure all targets in prometheus are up as shown in the image.
- Ensure all targets in Prometheus are up, as shown in the image. In the targets section, you can see there are two endpoints: one for
![](https://private-user-images.githubusercontent.com/113757574/346928953-eae03bdd-990a-4168-8e19-767797168586.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM3OTY5NDUsIm5iZiI6MTcyMzc5NjY0NSwicGF0aCI6Ii8xMTM3NTc1NzQvMzQ2OTI4OTUzLWVhZTAzYmRkLTk5MGEtNDE2OC04ZTE5LTc2Nzc5NzE2ODU4Ni5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxNlQwODI0MDVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xMzdiMmFlZmI1NmQzMTNmM2YzNmIwMjFhNmY1MWUyYjA0MmRiZmJkMjU5MDUxNWJmYTU1MDlhYWFlOWYzMTA5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.tOXHlR4g3lBelAKgwhFrhoN0VODDAoWXYYje_34Rk2A)
-
- Navigate to Grafana at
http://<ec2-public-ip>:30013
and add the Prometheus server URL in the connection settings as depicted in the image. Make sure to include the Prometheus URL with port number30010
. After entering the URL, click the "Save & Test" button at the bottom of the Grafana page. Once the test is successful, you will see a message indicating that the connection is successful. This ensures that Grafana is properly linked to Prometheus for monitoring and data visualization.
- Navigate to Grafana at
![](https://private-user-images.githubusercontent.com/113757574/348457099-23422a22-6eed-4e9e-b779-9bd28dfdc33a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM3OTY5NDUsIm5iZiI6MTcyMzc5NjY0NSwicGF0aCI6Ii8xMTM3NTc1NzQvMzQ4NDU3MDk5LTIzNDIyYTIyLTZlZWQtNGU5ZS1iNzc5LTliZDI4ZGZkYzMzYS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxNlQwODI0MDVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1jNzJhZjBmNWU4MzJjNWU3NTI0MDYzN2Y1Y2RjNWIyNjZjOTEwMmI3ZWVlNjI2YzMwZjhiM2ZjM2UwNjg5YjE5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.oYxbTRrdTA4-Ede2mFchEQQ669Ri2DOSKJo39z06nto)
-
- Once you've successfully added the data source, proceed to Grafana's Dashboards section. Navigate to Import, where you'll enter Dashboard ID
13332
into the designated field. Click 'Load' to import the kube-state-metrics dashboard. This dashboard provides comprehensive visualizations of Kubernetes nodes, pods, deployments, and more, offering valuable insights into your cluster's performance and resource utilization.
- Once you've successfully added the data source, proceed to Grafana's Dashboards section. Navigate to Import, where you'll enter Dashboard ID
Kubernetes Cluster | Kubernetes Nodes |
---|---|
Kubernetes Deployment | Website Photo |
---|---|
To completely remove a Kops-managed Kubernetes cluster, execute the following command:
kops delete cluster k8s-cluster.example.com --state=s3://<your-s3-bucket> --yes
By following these steps, you will have set up a Kubernetes cluster on AWS with Kops, deployed your application, and configured NGINX Ingress for external access via Route 53.
- Get CRI details used in nodes:
kubectl get nodes -o wide
- List pods (controller, scheduler, api-server) running on control-plane:
kubectl get pods -n kube-system
- List the nodes present in the cluster:
kubectl get nodes
- Grab the name of the control-plane node and describe it:
kubectl describe node master-node
- Get the instance group list from the cluster:
kops get ig --state=s3://<your-s3-bucket>
- Edit the node-machine to add the number of instances:
kops edit ig nodes-ap-south-1a --state=s3://<your-s3-bucket>
- In the default namespace:
kubectl get events
- In all namespaces:
kubectl get events --all-namespaces
- Get logs of a pod:
kubectl logs pod-name
- Get the instance group list of Kubernetes Cluster:
kops get ig --state=s3://<your-s3-bucket>
- Add/remove an instance in Kubernetes Cluster:
kops edit ig <instance-name> --state=s3://<your-s3-bucket>
If you encounter an error such as:
Note
error: You must be logged in to the server (the server has asked for the client to provide credentials)
Run the following command to refresh your Kubernetes configuration:
kops export kubecfg --admin --state=s3://<your-s3-bucket>