Entrypoint cannot be found in private repository image about pipeline HOT 12 CLOSED

Thanks @wilstdu for reporting this.
I looked for changes in v0.55 that could have caused this change in behaviour, but nothing stood out for me.
@vdemeester @imjasonh do you have any idea about what might have broken this behaviour?

from pipeline.

Essentially, the pipeline controller pod doesn't have the rights to fetch the images — really the image configuration — not the entrypoint.

@wilstdu what previous version was it working ? 0.54, or even previously ? Also, does it still not work with 0.56 or 0.57 ?

from pipeline.

@vdemeester, I was upgrading Tekton pipelines from v0.44.0 to v0.56.1.
I checked multiple Tekton Pipeline versions in between, and the last working one was 0.54, with 0.55 entry point retrieval no longer works. The only different thing in my setup was tekton-pipelines pod running different version - everything else stayed the same.

It doesn't work with 0.56, nor with 0.57.

from pipeline.

@wilstdu interesting 🤔

So, they way the pipeline controller work (in that part) is that it's taking we are taking the imagePullSecrets from the service account attached to the pipelinerun and the imagePullSecrets from the podTemplate (taskRun.Spec.ServiceAccountName, podTemplate.ImagePullSecrets in code) as well as some amazon (or other cloud) specifics (the "cloud-specific" part comes from go-containerregistry really – and I am not familiar at all of what it does). Nothing in that part of the code changed synced 2022, so either something changed in go-containerregistry, or something else weird is happening 🤔

from pipeline.

I also thought that maybe it was go-containerregistry, but there was no version change between v0.54 and v0.55:

➜ git diff v0.54.0..v0.55.0 -- go.mod | grep containerreg
 	github.com/google/go-containerregistry v0.16.1
 	github.com/google/go-containerregistry/pkg/authn/k8schain v0.0.0-20230625233257-b8504803389b
 	github.com/google/go-containerregistry/pkg/authn/kubernetes v0.0.0-20230516205744-dbecb1de8cfa

from pipeline.

@afrittoli yeah, that's what make me wonder what the hell is happening here 🙃 There is changes in the "indirect" dependencies from aws/ecr.

λ git diff v0.54.0..v0.55.0 -- go.mod | grep aws
-	github.com/sigstore/sigstore/pkg/signature/kms/aws v1.7.5
+	github.com/sigstore/sigstore/pkg/signature/kms/aws v1.7.6
-	github.com/aws/aws-sdk-go-v2/service/kms v1.24.7 // indirect
-	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.17.3 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.3 // indirect
+	github.com/aws/aws-sdk-go-v2/service/kms v1.27.2 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.21.2 // indirect
-	github.com/aws/aws-sdk-go-v2 v1.21.2 // indirect
-	github.com/aws/aws-sdk-go-v2/config v1.19.1 // indirect
-	github.com/aws/aws-sdk-go-v2/credentials v1.13.43 // indirect
-	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.13 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.43 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.37 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/ini v1.3.45 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.23.5 // indirect
+	github.com/aws/aws-sdk-go-v2/config v1.25.11 // indirect
+	github.com/aws/aws-sdk-go-v2/credentials v1.16.9 // indirect
+	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.9 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.8 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.8 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/ini v1.7.1 // indirect
 	github.com/aws/aws-sdk-go-v2/service/ecr v1.18.11 // indirect
 	github.com/aws/aws-sdk-go-v2/service/ecrpublic v1.16.2 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.37 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sso v1.15.2 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sts v1.23.2 // indirect
-	github.com/aws/smithy-go v1.15.0 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.8 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.18.2 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.26.2 // indirect
+	github.com/aws/smithy-go v1.18.1 // indirect
 	github.com/awslabs/amazon-ecr-credential-helper/ecr-login v0.0.0-20230510185313-f5e39e5f34c7 // indirect

But not on github.com/awslabs/amazon-ecr-credential-helper/ecr-login, so I am not sure if it as any impact…

from pipeline.

@wilstdu if you're familiar with the process of building Tekton, you could try building a v0.55 with github.com/sigstore/sigstore/pkg/signature/kms/aws== v1.7.5 - I imagine that's what's pulling in all the new aws packages, and see if that works. If not I could try and build that for you on some public artifact repo.

from pipeline.

@afrittoli if I did no mistakes when building the controller image - result is the same.

from pipeline.

I have the same bug discovered when trying to upgrade to the newest version v0.56.1 with the Operator. Same error message and behaviour.

One thing I managed to get working was, creating a imagePullSecret attaching to a ServiceAccount and using this ServiceAccount in the PipelineRun. This seems to work and the Task started at least. But this is no solution since AWS is resetting the credentials every 12 hours. More of a "POC" if there is a permission problem.
Once I figured that out I double-checked everything permission-wise in my cluster/AWS account, but can not find any problem their. Even trying to provide a ServiceAccount to the PipelineRun with annotation "eks.amazonaws.com/role-arn: " did not work.

Any progress for this topic or something I can help with to get that fixed?

from pipeline.

@afrittoli I investigated the bug a little bit further in the past few days.

I found out updating github.com/sigstore/sigstore/pkg/signature/kms/aws from 1.7.5 to 1.7.6 on version v0.54.0 is introducing the bug. After hunting and digging I found the issue is introduced with the indirect dependency github.com/aws/aws-sdk-go-v2 where they introduced a breaking change with v1.23.0 leading to the bug. They already know about that and also suggest how to fix it. See aws/aws-sdk-go-v2#2370.

All in all every module/package used of the github.com/aws/aws-sdk-go-v2 dependency needs to be updated to a version after the release of v1.23.0. I tried to do that with the newest Tekton Release v0.59.0 build the images deployed to our AWS cluster and everything was working again. To fix it it was enough to pull this into go.mod:

replace (
	github.com/aws/aws-sdk-go-v2/service/ecr => github.com/aws/aws-sdk-go-v2/service/ecr v1.27.3
	github.com/aws/aws-sdk-go-v2/service/ecrpublic => github.com/aws/aws-sdk-go-v2/service/ecrpublic v1.23.3
)

I would open a PR with updated dependencies to get that out ASAP and would try to hunt down where the indirect dependency is coming from to patch that directly there (if possible).

from pipeline.

@afrittoli I tracked down the dependencies.

It seems that the dependency github.com/google/go-containerregistry/pkg/authn/k8schain uses github.com/awslabs/amazon-ecr-credential-helper/ecr-login, which uses github.com/aws/aws-sdk-go-v2 with version v1.18.0 and github.com/aws/aws-sdk-go-v2/service/ecr with version v1.18.10.

github.com/sigstore/sigstore/pkg/signature/kms/aws is also using github.com/aws/aws-sdk-go-v2. But with version v1.26.0.

This leads to the problem that the indirect dependencies of github.com/aws/aws-sdk-go-v2 with version v1.26.0 and github.com/aws/aws-sdk-go-v2/service/ecr with version v1.18.10 does not match as said by the issue aws/aws-sdk-go-v2#2370, because with version v1.23.0 of github.com/aws/aws-sdk-go-v2 they introduced a breaking change.

I would create a PR to fix this with a replace in the go.mod for now in Tekton.
But my question is should I create a PR at github.com/awslabs/amazon-ecr-credential-helper/ecr-login to bump up the version of the github.com/aws/aws-sdk-go-v2 and github.com/aws/aws-sdk-go-v2/service/ecr so we can get rid of the replace or is it fine to leave it with the replace? Because out of the view of the credential-helper everything is fine.

from pipeline.

But my question is should I create a PR at github.com/awslabs/amazon-ecr-credential-helper/ecr-login to bump up the version of the github.com/aws/aws-sdk-go-v2 and github.com/aws/aws-sdk-go-v2/service/ecr so we can get rid of the replace or is it fine to leave it with the replace? Because out of the view of the credential-helper everything is fine.

Ideally yes 👼🏼 🙏🏼

from pipeline.