401 Unauthorized on image pull about amazon-vpc-cni-k8s HOT 8 CLOSED

If anybody else happens to come across this issue, we faced a similar problem. Adding the proper IAM role permissions fixed it, but not until rolling the masters. Kubelet only pulls the authorization token from ECR at initialization, so adding these permissions to your masters' IAM roles won't allow kubelet to get the proper token until the masters have been reinitialized.

from amazon-vpc-cni-k8s.

Upgrading EKS nodecluster to 1.29.0-20240129 fixed the issue

from amazon-vpc-cni-k8s.

@smcquay can you check if your instance role have permission to pull image from ECR registry?
for example,

        {
            "Sid": "kopsK8sECR",
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:BatchGetImage"
            ],
            "Resource": [
                "*"
            ]
        },

from amazon-vpc-cni-k8s.

@liwenwu-amazon I missed that the first time around, but I've added those permissions (and also the L-IPAM perms while I was at it), but I still cannot seem to hit that url successfully.

I'm trying from us-west-1, but I get the same error when I change to point at a registry in that region, but I get the same Not Authorized results.

Any ideas?

from amazon-vpc-cni-k8s.

@smcquay , can I take a look at your instance IAM policy? You can send them to me directly at [email protected].
Also, the CNI plugin docker image is ONLY available at us-west-2 region.

from amazon-vpc-cni-k8s.

Information sent; thanks for looking into this!

from amazon-vpc-cni-k8s.

I was able to build the image myself and push it to my own container registry. I'm going to call this resolved.

from amazon-vpc-cni-k8s.

Ran into the same issue. I was able to get around it by doing the same thing @smcquay did.

from amazon-vpc-cni-k8s.