Comments (12)
Good to hear! I'll close the issue then.
from moby.
Tried the workaround listed in this issue-44579,
Tagging the image does not seems to work after loading it from a tar file.
Image does not show up in docker image ls
from moby.
There were a lot of changes related to the containerd integration since v24. Could you please try reproducing with v26?
from moby.
@vvoland , upgrading docker version to 26 did not solve the issue,
I can see the image in ctr -n moby images ls but it does not show up in docker image ls command.
Here is the docker version I upgraded to.
:~# docker --version
Docker version 26.0.0, build 2ae903e
from moby.
Does it also happen with alpine
image?
If it doesn't, is there anything different about your image?
Could you show the content of the index.json
from the saved archive?
For example:
❱ tar xO index.json < golangci.tar | jq
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.index.v1+json",
"manifests": [
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"digest": "sha256:ed4bd7bb174cdb029a39224da6cc73aa860c0f623d9dbf931aa35e1f63b0ee18",
"size": 1529,
"annotations": {
"io.containerd.image.name": "docker.io/golangci/golangci-lint:v1.50.1",
"org.opencontainers.image.ref.name": "v1.50.1"
},
"platform": {
"architecture": "arm64",
"os": "linux"
}
}
]
}
from moby.
@vvoland ,
Yes it did happened to alpine
image as well!
BUT, the second piece about index.json
has solved the mystery for us!
I totally zoned out and forgot to mention one most important step. which is.
Upgrading the docker to a newer version in between saving the images, switching the storage driver and then loading the images.
We had this older docker version installed as a .deb package (by copying locally).
Saving images from this older version did not had index.json
file even for alpine
image
While trying to upgrade the docker (after saving the image), it was actually upgrading the kernel as well, which we did NOT want it to upgraded for this change.
So we were actually upgrading docker via sudo apt install --only-upgrade docker.io=24.0.5-0ubuntu1~20.04.1
After following this link on how to upgrade/install docker, we actually had to remove docker.io and install other packages listed in this link here.
I have tested the whole process again and it installed the latest version of docker Docker version 26.0.1, build d260a54
This seems to have solved this issue of images not being seen in docker image ls.
from moby.
It appears that the image saved from old version of docker does not have index.json file and so we can not find the digest ID for it.
So above solution of performing following tasks have solved the issue of images are not being seen in docker image ls
Saving images with older version of docker
removing older version of docker
re-installing new version of docker
change it to containerd snapshotter
loading the saved images
However any new tags we try to pull for that loaded image in new version of docker again does not show up in docker image ls
however it can be seen in ctr -n moby images ls
Any help will be appreciated!
from moby.
I managed to reproduce this on v26 with:
$ docker load -i <alpine-non-containerd.tar>
$ docker pull alpine:latest
# docker images will not show alpine
Note: This only happens if the pulled image DiffID is identical (all layers have the same uncompressed digest) layer, so it wouldn't happen for example if the alpine:latest
now points to a new image.
Quick notes on why it happens:
- Old image is exported from graphdrivers which recreates the layer tar file and stores it in uncompressed form
- Pulling the new image downloads the new image, but doesn't fetch the layer content (due to containerd/containerd#8973), because a snapshot for that DiffID already exists.
- Old image is deleted, along with the layer content blob (because it's no longer referenced by the new manifest, as it references the digest of the compressed tar).
- The new image still has a snapshot, but the content blob doesn't exist, so the image doesn't show in
docker images
because it's incomplete.
from moby.
I really appreciate your response and re opening this issue!
As you know we had docker.io installed from a local deb package (version Docker version 20.10.21, build 20.10.21-0ubuntu1~20.04.1
)
Is it safe to upgrade the older package docker.io
to Docker version 24.0.5, build 24.0.5-0ubuntu1~20.04.1
first and then save images? It appears once it is upgraded to 24.0.5 tar file seems to have index.json file along with all blobs.
Then upgrade it to 26.01 and change the storage driver to containerd snapshotter?
We are trying to avoid the docker pull on all the images after the update and switching the storage drivers.
Thank you.
from moby.
Okay so this does not seems to be the save issue from older version of docker.
I have saved all the images from the same version Docker version 26.0.1, build d260a54
,
switched to containerd ,
loaded all the images.
All images shows up.
But when we pull the next tag or latest,
those images does NOT show up.
Is there an interim work around we can try and test? we would really like to migrate to use containerd snapshotter for all our needs.
Thank you.
from moby.
Starting from v24+, docker save
outputs OCI-compatible archives (part of this is the index.json
), but it doesn't actually matter in this case.
The issue happens when re-pulling an image that was loaded from a image tar saved from the graphdriver (non-containerd snapshotter enabled Docker).
You can do what you do now, just remember that if you do docker pull
for the first time, and the pulled image will be exactly the same, you need to delete it first.
# on graphdriver store (for example on 20.10)
$ docker save alpine >alpine.tar
# on a 26.0 with `features: { "containerd-snapshotter": true }`
$ docker load -i /v/alp.tar
Loaded image: alpine:latest
# The image is there
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
alpine latest b3d53b9c3336 2 months ago 16.4MB
# If you want to pull it again, remove it first
$ docker rmi alpine
Untagged: alpine:latest
Deleted: sha256:b3d53b9c333611da7a2888893bf5e6d8884406035837e63facfb5006e24a0fa2
$ docker pull alpine
Using default tag: latest
latest: Pulling from library/alpine
bca4290a9639: Download complete
Digest: sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b
Status: Downloaded newer image for alpine:latest
docker.io/library/alpine:latest
# It will still be there if you delete the previous one before pull
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
alpine latest c5b1261d6d3e 2 months ago 11.8MB
from moby.
Thank you for your help and quick responses in this matter.
Our purpose of saving and loading the images was to save the time it takes to pull and network bandwidth utilization.
However we were able to solve that issue by using the local registry.
For anyone interested, Here is how we did it without having to save/load and without having to pull the images again.
1). Update to latest supported version of docker
2). Create a local registry with bind volume mount.
docker run -d -p 5000:5000 --restart always -v /var/tmp/docker_image_registry:/var/lib/registry --name local_registry registry:2
3). For each image tag and push to local registry.
export IMAGES="image1:release-1 image1:release-2 image2:latest myrepo/image3:0.1.1"
#A). Tag and push the image.
for image in $IMAGES; do
echo $image;
docker tag $image localhost:5000/$image;
docker push localhost:5000/$image;
done
# B) Verify local registry have all the repository
curl -X GET http://localhost:5000/v2/_catalog
4). Stop the local registry and delete all the images using docker rmi -f
docker stop local_registry
for image in $(docker images |grep -v -E "local_registry|REPOSITORY" |awk '{print $1, $2}' |tr " " ":"); do
docker rmi $image
done
# system prune
docker system prune
5). Switch to containerd (features: { "containerd-snapshotter": true }
)
6). Restart docker for change to take affect
systemctl restart docker
7). Re-run local registry
docker run -d -p 5000:5000 --restart always -v /var/tmp/docker_image_registry:/var/lib/registry --name local_registry registry:2
8). Verify all repositories.
curl -X GET http://localhost:5000/v2/_catalog
9). For each image pull from local registry and re-tag
export IMAGES="image1:release-1 image1:release-2 image2:latest myrepo/image3:0.1.1"
# A). Pull and re-tag all images.
for image in $IMAGES; do
echo $image;
docker pull localhost:5000/$image;
docker tag localhost:5000/$image $image;
done
10). Verify all images
docker image ls
- Remove local tags.
export IMAGES="image1:release-1 image1:release-2 image2:latest myrepo/image3:0.1.1"
for image in $IMAGES; do
echo $image
docker rmi -f localhost:5000/$image
done
12). Clean up.
# Remove content of the bind mount directory /var/tmp/docker_image_registry
rm -rf /var/tmp/docker_image_registry`
# Remove local registry
docker stop local_registry
docker container rm local_registry
docker rmi -f registry:2
# System prune:
docker system prune
from moby.
Related Issues (20)
- `docker image ls --filter=reference=docker.io/$MY/$IMG` != `docker image ls --filter=reference=$MY/$IMG`
- Pass proxy settings from host system for Windows containers HOT 1
- Rootless docker cannot start containers bound to different IPs but the same port
- Problem with moby-engine install on raspberry zero w with bullseye.
- Internal network DNS fails to resolve HOT 9
- containerd: classic builder fails to preserve platform information correctly HOT 1
- Cannot create new containers HOT 1
- Running docker containers in existing network namespace (netns) HOT 5
- Failing tests on Apple Chip Mac HOT 2
- [swarm mode] Random published port not accessible upfront using . It needs an update
- New volume mount subpath - does not create sub directory if it doesn't ee
- Add an option to start docker without any containers regardless of their restart policy
- Docker starts containers with restart policy `on-failure` after host reset HOT 1
- failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: “docker-entrypoint.sh”: executable file not found in $PATH: unknown
- [grafana/loki-docker-driver] Docker rootless error mounting "cgroup" to rootfs at "/sys/fs/cgroup"
- A potential goleak in cluster.go HOT 1
- Docker keeps cached manifests and indicies forever somewhere. HOT 6
- Docker compose argument to replace env_file directive, or argument to enable host environment passthrough
- UDP Response Timeout in Bridge Mode Networking HOT 2
- docker cp --parents should exist, analogous to GNU cp --parents
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from moby.