Comments (12)
GnatorX
commented on May 29, 2024
1
Its a bit more complicated than just adding the label. From what I can tell when we had similar issues, on node creation or pod creation the label must already be on the node object which should be added by the CNI. If the label isn't there and the pod already started creation on the pod, it is possible that the pod will get stuck in ContainerCreating because VPC RC saw a pod land on a "unmanaged node" and its logic is to ignore it until a new event is emit for that pod. This could me unknown amount of time stuck in ContainerCreating.
The label being present on the node prior the pod landing will let VPC resource controller retry until the node is managed. I believe something is wrong with the CNI setup thats why the label isn't present. I am not familiar with EKS setup and I don't work for AWS. It might be worth while to cut a support ticket instead to see.
from amazon-vpc-cni-k8s.
orsenthil
commented on May 29, 2024
1
I sent the BottleRocket logs collected with logdog to the support e-mail address. Having issues running the collector scrip with Bottlerocket.
I notice you have already sent it, we will review the logs.
from amazon-vpc-cni-k8s.
GnatorX
commented on May 29, 2024
Are you using pod security group for these pods? Its interesting to see that there isn't the trunk-attached label on the node and it feels similar to aws/karpenter-provider-aws#1252
from amazon-vpc-cni-k8s.
davidgp1701
commented on May 29, 2024
Are you using pod security group for these pods? Its interesting to see that there isn't the
trunk-attachedlabel on the node and it feels similar to aws/karpenter-provider-aws#1252
Yes, I'm. In the the settings I setup: DISABLE_TCP_EARLY_DEMUX and ENABLE_POD_ENI. I will take a look at the post you mention in detail. Thanks.
from amazon-vpc-cni-k8s.
davidgp1701
commented on May 29, 2024
Ok, I did more testing this problem happens only if you enable Security Groups for Pods adding the environment variables described here: https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html , SGs for pods are needed for our clusters.
@GnatorX the trunk-attached label was not set, I tried to set it, both with true and false values, the problem still persists.
from amazon-vpc-cni-k8s.
orsenthil
commented on May 29, 2024
The node has 45 Pods, it should still be able to use 58 Pods, no CPU, Memory or Disk limitations. And, if no more IP addresses could be assigned to that node, the Pods should end in PENDING status instead.
- Does ipamd.log and plugin.log give any additional error messages besides pod struck on container creation?
CNI versions NOT showing the issue:
v1.16.3-eksbuild.2
Are the reset of the environmental factors same when this issue is observed?
We will try to reproduce this issue to know more details on the behavior with your description and share an update.
Could you gather the logs for bottlerocket and share it with using the instructions given in the troubleshooting guide?
from amazon-vpc-cni-k8s.
wy100101
commented on May 29, 2024
I'm seeing a similar issue here on v1.18.1-eksbuild.1 with ENABLE_POD_ENI set to "true". I'm not actually using pod security groups:
Name: ip-10-201-165-12.ec2.internal
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=c6a.2xlarge
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1b
k8s.io/cloud-provider-aws=d233b588ef1fdb73bec8d62908da3a7f
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-10-201-165-12.ec2.internal
kubernetes.io/os=linux
node.kubernetes.io/instance-type=c6a.2xlarge
olo.com/asg-name=eks-devenv-node-default
olo.com/node-group-name=default
olo.com/node-type=default
topology.ebs.csi.aws.com/zone=us-east-1b
topology.kubernetes.io/region=us-east-1
topology.kubernetes.io/zone=us-east-1b
Annotations: alpha.kubernetes.io/provided-node-ip: 10.201.165.12
csi.volume.kubernetes.io/nodeid:
{"ebs.csi.aws.com":"i-015d8453f004b176e","efs.csi.aws.com":"i-015d8453f004b176e","smb.csi.k8s.io":"ip-10-201-165-12.ec2.internal"}
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 30 Apr 2024 11:46:37 -0400
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: ip-10-201-165-12.ec2.internal
AcquireTime: <unset>
RenewTime: Thu, 02 May 2024 02:03:05 -0400
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 02 May 2024 02:02:21 -0400 Tue, 30 Apr 2024 11:46:37 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 02 May 2024 02:02:21 -0400 Tue, 30 Apr 2024 11:46:37 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 02 May 2024 02:02:21 -0400 Tue, 30 Apr 2024 11:46:37 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 02 May 2024 02:02:21 -0400 Tue, 30 Apr 2024 11:46:58 -0400 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 10.201.165.12
InternalDNS: ip-10-201-165-12.ec2.internal
Hostname: ip-10-201-165-12.ec2.internal
Capacity:
cpu: 8
ephemeral-storage: 50620216Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 16009796Ki
pods: 58
vpc.amazonaws.com/pod-eni: 38
Allocatable:
cpu: 7910m
ephemeral-storage: 45577849165
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 14992964Ki
pods: 58
vpc.amazonaws.com/pod-eni: 38
System Info:
Machine ID: ec2616dd3f2c306558bfd766260713d7
System UUID: ec24155c-3b11-d1f6-a252-f3633d61269e
Boot ID: 34afba82-6fad-49ab-bb09-ff3fbaa11576
Kernel Version: 5.15.0-1055-aws
OS Image: Ubuntu 20.04.6 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.7.2
Kubelet Version: v1.27.6
Kube-Proxy Version: v1.27.6
ProviderID: aws:///us-east-1b/i-015d8453f004b176e
Looking at the cni-driver logs and 1 of 4 enis is a trunk eni that has no IPs to allocate limiting the node to 42 IPs
"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:607","msg":"AssignPodIPv4Address: IP address pool stats: total 42, assigned 42"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.162.211/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.162.211/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.160.139/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.160.139/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.162.207/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.162.207/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.163.134/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.163.134/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.165.111/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.165.111/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.165.202/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.165.202/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.330Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.161.147/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.161.147/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.161.26/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.161.26/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.162.26/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.162.26/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.166.48/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.166.48/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.165.253/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.165.253/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.160.148/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.160.148/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.164.127/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.164.127/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.161.155/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.161.155/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"AssignPodIPv4Address: ENI eni-04a01feee45a761c6 does not have available addresses"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"AssignPodIPv4Address: ENI eni-01dbfeaf18cf82dd7 does not have available addresses"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.165.80/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.165.80/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.164.157/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.164.157/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.163.227/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.163.227/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.166.150/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.166.150/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.164.143/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.164.143/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.162.173/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.162.173/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.166.229/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.166.229/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.167.188/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.167.188/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.162.85/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.162.85/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.161.104/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.161.104/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.166.136/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.166.136/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.162.233/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.162.233/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.161.239/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.161.239/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.164.140/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.164.140/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"AssignPodIPv4Address: ENI eni-0aaff8bf49e7d3734 does not have available addresses"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.164.204/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.164.204/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.161.185/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.161.185/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.164.218/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.164.218/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.161.165/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.161.165/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.166.240/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.166.240/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.164.70/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.164.70/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.162.187/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.162.187/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.164.241/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.164.241/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.160.105/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.160.105/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.163.196/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.163.196/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.163.243/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.163.243/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.163.99/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.163.99/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.163.113/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.163.113/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.201.162.243/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.201.162.243/ffffffff"}
{"level":"debug","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"AssignPodIPv4Address: ENI eni-0a211c5e727746570 does not have available addresses"}
{"level":"error","ts":"2024-05-02T01:54:51.331Z","caller":"datastore/data_store.go:607","msg":"DataStore has no available IP/Prefix addresses"}
from amazon-vpc-cni-k8s.
Dpkbhatt052
commented on May 29, 2024
This is an expected behaviour with security group for pods as IP's to pod that are not using security group for pod would get IP's from the ENI other than trunk ENI.
Basically when you enable security group for pod and if your instance can have maximum of 4 ENI means 56 IPs , vpc resource controller will create a trunk eni out from the 4 ENI. Whenever a pod using security group is deployed , vpc rrsource controller will create a branch eni on the trunk eni and assign IP to it. Normal pods won't get IP from the Trunk ENI. Hence the pods ( that are not using security group for pods will get IPs from the rest of the 3 ENI ) . There are two things here one is pod using security group which is getting IP by vpc resource controller and other is normal pod which is not using security groups who ip allocation is done by ipamd. Now in ipamd you are seeing 42 total IPs because its from remaining 3 ENIs other than trunk ENI which is expected. But max pod contribution would be contributed by both security group for pod and pod not using security group for pod. Hence its expected that 43rd pod will come in container creating as ipamd can max assign 42 IPs to the pods (not using security group for pods). This is an expected behaviour. I believe reducing max pod is the option to get this mitigated.
from amazon-vpc-cni-k8s.
Related Issues (20)
- K3S with AWS VPC CNI breaks Pod communication #9716 HOT 3
- using `amazon-vpc-cni-k8s` outside eks HOT 13
- /run/xtables.lock created as directory when installed with Helm HOT 13
- No additional ENIs are attached after prefix delegation HOT 6
- Configurable log output for the aws-eks-nodeagent in the daemonset HOT 1
- Node created in subnet with low number of IP adresses: failed to assign an IP address to container HOT 2
- Can `AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG` be generalized for multi-homed pods? HOT 5
- Pods stuck in `CrashLoopBackoff` when restarting custom EKS node. HOT 7
- WARM_ENI_TARGET is 1. But worker node ENI 2 (with coredns pods used) HOT 2
- Is it possible to route cluster-ip traffic from EC2 instances (the outside of eks, but same vpc) to EKS HOT 2
- What is the difference between `vX.X.X` and `vX.X.X-eksbuild.x` ? HOT 2
- Upgrading from v1.16.0-eksbuild.1 to v1.17 or v1.18 results in failure to assign IP address to container HOT 9
- RefreshSecurityGroups should only be called on ENIs already checked by the ENI/IP reconciler HOT 8
- Conflicts .data.enable-windows-ipam HOT 2
- Improve VPC CNI memory by reducing number of things it is caching HOT 6
- ip addresses leaking when there are too many ip in cooldown pool HOT 2
- Should node agent be opt-in on vpc CNI HOT 2
- Enhanced subnet discovery should use configurable tags
- make generate-limits script failed due to ENI limit mismatch HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-vpc-cni-k8s.