Comments (8)
I managed to very easily reproduce this issue.
package main
import (
"fmt"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1/daemon"
)
func main() {
err := run()
if err != nil {
panic(err)
}
}
func run() error {
ref, err := name.ParseReference("docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd")
if err != nil {
return err
}
img, err := daemon.Image(ref)
if err != nil {
return err
}
configName, err := img.ConfigName()
if err != nil {
return err
}
fmt.Println("config name", configName)
return nil
}
Running without Containerd snapshotter.
config name sha256:1d34ffeaf190be23d3de5a8de0a436676b758f48f835c3a2d4768b798c15a7f1
Running with Containerd snapshotter.
config name sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
These are the same results that we found when pairing.
Long term this needs to be fixed upstream but in the meantime @AustinAbro321 will add a work around to fix this in the next release.
from zarf.
@RothAndrew #2593 should be the band aid fix to this issue. It's working on my mac, feel free to try it out.
from zarf.
Of note is that if I keep everything else exactly the same, but I go ahead and push the image to my docker registry, everything works fine. But, I really really really don't want to do that
from zarf.
Edit: moved to new issue: #2586
Side-topic: I'd absolutely LOVE a way to specify that images should ONLY be pulled from the local docker daemon. Perhaps something like:
zarf package create --confirm --local-docker-only
with the ability to first pull all images that are referenced in the zarf.yaml in case there are any that are being used that are upstream dependencies
zarf package pull-images
docker build registry.example.com/myimage:1.2.3
zarf package create --confirm --local-docker-only
Why? because release-please controls all my semver versions. So, up in the registry there is definitely v1.2.3 present, but I'm now developing v1.2.4. But, I don't want to have to change versions everywhere, I want release-please to handle that for me. So, locally the version is still specified as v1.2.3 but I don't want the image from the registry, I want the local image that has my changes.
I'm working around it now by running the zarf package create
in a docker container like:
docker network rm no-internet-net || true
docker network create --internal no-internet-net
docker run --platform linux/amd64 --rm -v $(pwd):/work -v /var/run/docker.sock:/var/run/docker.sock -w /work/zarf --network no-internet-net ghcr.io/defenseunicorns/build-harness/build-harness:2.0.24 uds zarf package create --architecture $(scripts/get_arch.sh) --confirm --skip-sbom --no-progress
docker network rm no-internet-net
using the custom no-internet-net
network makes the container run without internet connectivity
Back to the issue at hand: I have tried doing things with just straight zarf, no docker stuff, and it still fails whenever it does the local docker fallback.
from zarf.
Thanks for the detailed issue! If you want to create a separate issue for the local-docker-only flag I think it'd be a good feature to add.
The most interesting thing I'm seeing is from the checksum.txt. File names for image blobs in OCI should be just the sha256sum of that file. It looks to me like the issue is that the correct content is getting placed in the image blob but it is being named incorrectly.
A few questions
- if you run
zarf dev sha256sum /images/blobs/sha256/ad69e88322c92fe909723f882c4c8213d412bbadfef687c7cf5e360adba141b6
do you get the name of the file ora02b607f0d337d98c48e812611a4289e8e10b81e5832685393292d83b059835c
. This will help us verify that the file is really named incorrectly and not an issue of Zarf putting the wrong checksum down. - What type of file is
/images/blobs/sha256/ad69e88322c92fe909723f882c4c8213d412bbadfef687c7cf5e360adba141b6
? It'll be a blob, json manifest, or docker image config file. - Are you able to reproduce with any other images?
- Are you able to reproduce on amd64 hardware?
from zarf.
Troubleshooting notes:
- The issue happens on ARM MacOS, using Docker Desktop with Containerd mode turned on. It does not happen when Containerd mode is off.
- But, I can't turn Containerd mode off, because then other things break, like multi-arch building
from zarf.
After some pairing we have determined that the error is caused by the wrong hash being used for the config layer. As stated before the hash and the file name should match for blob layers. The content of the file is correct and results in the correct hash. The file however has the same name as the hash of the index layer. We managed to reproduce the issue after determining that Docker with Containerd snapshotter was required.
After studying cranes writing logic it becomes clear that it does not hash the content of the config file to determine its hash. Instead it calls the image config name function.
https://github.com/google/go-containerregistry/blob/3764db238e3ebf35a3ea0da696287701214859b9/pkg/v1/layout/write.go#L356-L366
The config name function implementation differs based on the source of the image. Which would explain why it would only occur for local images. For local images the Docker client is used to fetch the config name.
https://github.com/google/go-containerregistry/blob/3764db238e3ebf35a3ea0da696287701214859b9/pkg/v1/daemon/image.go#L177-L181
It turns out that the config name comes from the ID returned in image inspect. What is probably happening is that the ID returned differs when running Docker standalone and Docker with Containerd snapshotter. We will need to produce some example code which shows that this is the actual issue.
from zarf.
Thanks @phillebaba , appreciate the thoroughness.
Any idea what the next step might be?
For now, I'm gonna look into running a local registry:2
combined with the --registry-override
flag, but I have a feeling it's gonna be super janky and I'm gonna hate it.
from zarf.
Related Issues (20)
- Decouple and deduplicate the lint package
- refactor: return a single image for the injector pod to use
- feat: only run the injector in a cluster that has not been previously initialized
- Adoption of Namespaces with `--adopt-existing-resources`
- Use Server Side Apply when creating Kubernetes resources
- Feature request: Zarf dev package templating
- v1 Deprecation `.metadata.aggregateChecksum` -> `.build.aggregateChecksum`
- v1 deprecation, move several metadata fields to `.metadata.annotations`
- v1 Deprecation `.components.x.[charts/manifests].noWait` -> `.components.x.[charts/manifests].x.wait`
- v1 deprecation, `.maxRetries` -> `retries`
- v1 Deprecation, pull `wait` out of actions
- request: Focus prune command to only worry about pruning
- docs(site): edit page button on zarf docs site takes you to wrong path on github
- docs(site): bad link in zarf tutorial - Creating a Zarf Package HOT 1
- Built in wait for data injections to complete before starting pod
- Zarf silently fails to overwrite tagged images in the Zarf Registry HOT 1
- Several functions in Zarf need to be more context aware and return an error after context is cancelled
- Add link checker to docs in CI
- Improved image push progress bar support for long image names
- docs(site): dos-games example dockerfile potentially broken due to archived web link HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zarf.