gdraheim / docker-copyedit Goto Github PK
View Code? Open in Web Editor NEWedit docker image metadata (including remove docker volumes)
License: European Union Public License 1.2
edit docker image metadata (including remove docker volumes)
License: European Union Public License 1.2
Running docker-copyedit with a 'set-label name value' leads to an error if the image has no labels. It works fine when an image already has some label(s). The Python stack trace appears thus:
$ docker-copyedit.py from postgres:13.5 into b:latest set-label somename somevalue
File "docker-copyedit.py", line 938, in <module>
edit_image(inp, out, commands)
File "docker-copyedit.py", line 310, in edit_image
changed = edit_datadir(datadir, out_tag, edits)
File "docker-copyedit.py", line 539, in edit_datadir
if target in config[CONFIG][key]:
TypeError: argument of type 'NoneType' is not iterable
An example of an image with no existing labels is postgres:13.5. Here is the 'docker inspect' of this image:
[
{
"Id": "sha256:e01c76bb1351f3fb966f97cf108f1ff586056a517d56c9ec2629ede6be756691",
"RepoTags": [
"postgres:13.5"
],
"RepoDigests": [
"postgres@sha256:2e2e213c0fe4d1229373bebf114f255f68d86fba9e9f30e9bcc08e92cdf7b5b4"
],
"Parent": "",
"Comment": "",
"Created": "2022-01-27T01:11:05.913589796Z",
"Container": "32bd06472b5cfa54b87f8772fe083d5bab2b1144868ac6ce408b757f54bb982a",
"ContainerConfig": {
"Hostname": "32bd06472b5c",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"5432/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/13/bin",
"GOSU_VERSION=1.14",
"LANG=en_US.utf8",
"PG_MAJOR=13",
"PG_VERSION=13.5-1.pgdg110+1",
"PGDATA=/var/lib/postgresql/data"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"postgres\"]"
],
"Image": "sha256:8004cdfe9dee6a2263d8cc24d9f67d6c41dd0fc090c1be01b98418d9d2497bce",
"Volumes": {
"/var/lib/postgresql/data": {}
},
"WorkingDir": "",
"Entrypoint": [
"docker-entrypoint.sh"
],
"OnBuild": null,
"Labels": {},
"StopSignal": "SIGINT"
},
"DockerVersion": "20.10.7",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"5432/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/13/bin",
"GOSU_VERSION=1.14",
"LANG=en_US.utf8",
"PG_MAJOR=13",
"PG_VERSION=13.5-1.pgdg110+1",
"PGDATA=/var/lib/postgresql/data"
],
"Cmd": [
"postgres"
],
"Image": "sha256:8004cdfe9dee6a2263d8cc24d9f67d6c41dd0fc090c1be01b98418d9d2497bce",
"Volumes": {
"/var/lib/postgresql/data": {}
},
"WorkingDir": "",
"Entrypoint": [
"docker-entrypoint.sh"
],
"OnBuild": null,
"Labels": null,
"StopSignal": "SIGINT"
},
"Architecture": "amd64",
"Os": "linux",
"Size": 371198583,
"VirtualSize": 371198583,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/4b5c59b2df623b68662dff9dd759e8b1e5e15ba1065d4cd72737e709d44e38c8/diff:/var/lib/docker/overlay2/79cdf68ee1096872e5a416d505793aa391f2a1ee0b0d25bc406f84954bb3ac94/diff:/var/lib/docker/overlay2/445a42c7637f1b00fd12a9249f55171bdd31ffcbb9de169467030f9cb6073001/diff:/var/lib/docker/overlay2/a1fcb9929fb699e647fc716627cc4bfd64413af74b9af3442c9c8d08784d1bd0/diff:/var/lib/docker/overlay2/d140bf2c2f284959d56fec2afb7b3ef2acfa0ed4ce35188208d775e2212b66c1/diff:/var/lib/docker/overlay2/754602e81673683379f5ca51ac26d3a110ade09ded64671d611e9f77b7468198/diff:/var/lib/docker/overlay2/bd0b0f158d1bae5922aadbf7ce49123e5ea0db06f4d09f36a9bed5a27514d7e1/diff:/var/lib/docker/overlay2/186b4af03de1f2231bfc00d2a5026d5e0ea2741f2f7f3b86726dcbfbe4aac4e8/diff:/var/lib/docker/overlay2/1a582292220a3e41a400d932828548029d48416c4d3b1673a2b9fe253978c515/diff:/var/lib/docker/overlay2/b37a11f200770cec046366d4a5edf2d921c3873234f3f540dd2e94d5a0886fb9/diff:/var/lib/docker/overlay2/dbf7b34785d5b0ac0936370bb6d2a00a886c083e745df51a6b185efe68b9bc75/diff:/var/lib/docker/overlay2/39cf89b7f028e5f5aab843dc56def8f5817dc182f416ad4d59df4a09865e134f/diff",
"MergedDir": "/var/lib/docker/overlay2/a23616e078d0e485b807e0affe1b0e35dc315a1b80ad4439572a02b9dab8dd6c/merged",
"UpperDir": "/var/lib/docker/overlay2/a23616e078d0e485b807e0affe1b0e35dc315a1b80ad4439572a02b9dab8dd6c/diff",
"WorkDir": "/var/lib/docker/overlay2/a23616e078d0e485b807e0affe1b0e35dc315a1b80ad4439572a02b9dab8dd6c/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:7d0ebbe3f5d26c1b5ec4d5dbb6fe3205d7061f9735080b0162d550530328abd6",
"sha256:504d3c78f1942134301a91d83b6d2102b2c398eab2c81a76f62b9eb33f8c000b",
"sha256:8c1f84338867a554126fe7100341ff8174fe71e13518389803e4e1050aa8f200",
"sha256:10e60ad30e5f437ff02dd7273545deeeabb0d07595dae73cc0f1ea2d38a2956f",
"sha256:96d824739a641ee2e8f36bcd8ff576c7ba72029184aaa7c8ec5c418f5c60802f",
"sha256:a4b9517dfa8c2c05e3cd3423f52535a7e551ac7e4c026541742c054b19b1c5e4",
"sha256:f2e4fedb957df6c5380f46b9b08e2554b1936a553acbc0c77579faf53724feda",
"sha256:c1cf0fe2cfa15adc5603ebd8d0047d12afe9b82a634784c8b3967be55ffc33f2",
"sha256:b7035179c078b0703b5d5f322123535179489aa8b0eb820e317862a6448ff240",
"sha256:6fc0879b8f3a07b74f6de0a76c22175ec89d0dcc46827b3f433b736e977a66b7",
"sha256:954be07f25afe8092888629a0603fc6fc773c093bd5b8d9f20e6bfdeff1b8a06",
"sha256:ae0a327d2532a9199bb5d3f7a198315f0ea092eb9c9a7719a0bec76b22da307e",
"sha256:d69fa3581c263ec07ff28eb3edd558c4fb20e4bbad8a039eec0c005cc3152819"
]
},
"Metadata": {
"LastTagTime": "0001-01-01T00:00:00Z"
}
}
]
Perhaps the $.Config.Labels:null is the culprit.
It would be nice if you dockerized your application.
We could use it in ci/cd processes.
THis is only a question.
I wanted to know the significance of this
https://github.com/gdraheim/docker-copyedit/blob/master/docker-copyedit.py#L713-L715
And if adding to the history is helping with properly formatting the tarball for load
ing the image?
Thanks
When trying to remove a certain port of an image, I only receive this error in the last step of the process:
DEBUG:edit:changed 1 layer metadata
ERROR:edit:CMD docker load -i load.tmp/ready.tar
ERROR:edit:EXIT 1
ERROR:edit:STDOUT
ERROR:edit:STDERR invalid tag "jannikz/gluster-prod"
Traceback (most recent call last):
File "./docker-copyedit.py", line 706, in <module>
edit_image(inp, out, commands)
File "./docker-copyedit.py", line 252, in edit_image
sh(cmd.format(**locals()))
File "./docker-copyedit.py", line 43, in sh
raise Exception("shell command failed")
Exception: shell command failed
No matter what tag I'm entering .. did you ever face a problem like this?
./docker-copyedit.py \
FROM jannikz/gluster-prod:latest INTO jannikz/gluster-prod -vvv REMOVE PORT 111
DEBUG:edit:image parsing = jannikz/gluster-prod:latest
DEBUG:edit:.registry = None
DEBUG:edit:.image = jannikz/gluster-prod
DEBUG:edit:.tag = :latest
DEBUG:edit:image parsing = jannikz/gluster-prod
DEBUG:edit:.registry = None
DEBUG:edit:.image = jannikz/gluster-prod
DEBUG:edit:.tag = None
DEBUG:edit:total 1484224
__version__ = "1.2.1402"
Docker version 18.06.1-ce, build e68fc7a
Hi,
Probably, I did something wrong, but it was from README:
in readme:
./docker-copyedit.py \
FROM image1 INTO image2 REMOVE ALL VOLUMES
but it doesn't work:
[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql:5.6 INTO mysql-novol:5.6 REMOVE ALL VOLUMES
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql:5.6 INTO mysql-novol REMOVE ALL VOLUMES
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql INTO mysql-novol REMOVE ALL VOLUMES
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:CMD docker tag mysql mysql-novol
ERROR:edit:EXIT 1
ERROR:edit:STDOUT
ERROR:edit:STDERR Error response from daemon: No such image: mysql:latest
Traceback (most recent call last):
File "./docker-copyedit.py", line 538, in <module>
edit_image(inp, out, commands)
File "./docker-copyedit.py", line 91, in edit_image
sh(cmd.format(**locals()))
File "./docker-copyedit.py", line 41, in sh
raise Exception("shell command failed")
Exception: shell command failed
Exit code: 1
[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$
But this works:
[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql:5.6 INTO mysql-novol:5.6 -vv REMOVE VOLUME /var/lib/mysql
INFO:edit:written new load.tmp/data/19706e2768c04c8301865815ea821c173e3715f40f470a17a057b667db610d54.json
INFO:edit:removed old load.tmp/data/1f47fade220d0ddedb916156cf4d7121a020afff03c19b4638ac494c5d1eca63.json
Alpine has recently got rid of Python2, which is needed for this (awesome) script.
While this is workaround-able for now I was wondering if there was ever any plans to support Python 3?
Hi Guido,
I wanted to remove the /database volume from the ibmcom/db2 image.
But when I process it with your script using:
docker-copyedit.py FROM ibmcom/db2:latest INTO db2:1.0 REMOVE ALL VOLUMES
I get the error that the manifest file already exists:
Traceback (most recent call last): File "docker-copyedit.py", line 929, in <module> edit_image(inp, out, commands) File "docker-copyedit.py", line 301, in edit_image changed = edit_datadir(datadir, out_tag, edits) File "docker-copyedit.py", line 723, in edit_datadir os.rename(manifest_filename + ".tmp", manifest_filename) FileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: 'load.tmp\\data\\manifest.json.tmp' -> 'load.tmp\\data\\manifest.json'
I tried to fix it but I'am a novice to python.
Thanks Mario
Hi, I've spotted that the main python file has a number of places where indent is broken. Can you please re-tab (re-space) it?
I've tried
/docker-copyedit.py FROM nginx:alpine INTO nginx:noexpose REMOVE PORT 80
/docker-copyedit.py FROM nginx:alpine INTO nginx:noexpose remove all ports
Both create a new container, but without the EXPOSE removed
docker inspect nginx:noexpose | jq '.[].ContainerConfig.ExposedPorts'
{
"80/tcp": {}
}
Using moby buildkit and buildx bake I can build arm64 images on my amd64 machine. Is there a way to use your script to alter an arm64 image but doing so on a amd64 machine? I assume it would, like moby buildkit, have to do so in a dockerized arm64 container using qemu.
Hi, thanks for this handy util!
The most recent commit changed the permission bits on the copyedit script. Its not difficult to get around - but it means the docs dont work out of the box
docker-copyedit doesn't allow us to change the architecture "variant" metadata.
For example,
$ docker inspect arm32v7/ubuntu | jq -r .[].Variant
v7
$ ./docker-copyedit.py FROM arm32v7/ubuntu into foo set variant foo
ERROR:edit:unknown edit command starting with set variant
the tar file generated from "docker save" is optionally keeped by the control of KEEPDIR , which I understand mostly for debugging purpose.
but even KEEPDIR =0, the tar file is always saved on disk, only deleted afterwards
I am a bit impatient and did following patch , pipe-lining with tee , to save disk IO, save time
KEEPDIR = 0
251,256c251,254
< if KEEPDIR >= 2:
< cmd = "docker save {inp} |tee {inputfile}|tar x -f - -C {datadir}"
< sh(cmd.format(**locals()))
< else:
< cmd = "docker save {inp} |tar x -f - -C {datadir}"
< sh(cmd.format(**locals()))
cmd = "docker save {inp} -o {inputfile}" sh(cmd.format(**locals())) cmd = "tar xf {inputfile} -C {datadir}" sh(cmd.format(**locals()))
if KEEPDIR >= 2: logg.warning("keeping %s", inputfile) else: if os.path.exists(inputfile): os.remove(inputfile)
Not sure if I'm doing something wrong or is this supported on a mac?
❯ ./docker-copyedit.py FROM golang:latest INTO golang2 -vv -T /tmp set label "userid" "0:0"
WARNING:edit:output image is local (None)
INFO:edit:new /tmp/data from docker save
WARNING:edit:done new label 'userid' 0:0
INFO:edit:written new /tmp/data/8c5b2ffbe44aa2888737f1696530a0558c4e6c2fbf913059167772fd3ea5fede.json
INFO:edit:removed old /tmp/data/blobs/sha256/98ef6bdf40ba9a262cb121fff28759f9431fe888655704c97a65aae643defd77
ERROR:edit:CMD docker load -i /tmp/ready.tar
ERROR:edit:EXIT 1
ERROR:edit:STDOUT b''
ERROR:edit:STDERR b'lsetxattr com.apple.provenance /oci-layout: operation not supported\n'
Traceback (most recent call last):
File "/Users/jeffsaremi/src/docker-copyedit/./docker-copyedit.py", line 976, in <module>
sys.exit(run(*args))
File "/Users/jeffsaremi/src/docker-copyedit/./docker-copyedit.py", line 905, in run
return edit_image(inp, out, commands)
File "/Users/jeffsaremi/src/docker-copyedit/./docker-copyedit.py", line 331, in edit_image
sh(cmd.format(**locals()))
File "/Users/jeffsaremi/src/docker-copyedit/./docker-copyedit.py", line 73, in sh
raise ShellException("shell command failed", result)
__main__.ShellException: shell command failed
Hi Guido,
I wanted to remove the /xyz volume from the image1.
But when I process it with your script using:
python docker-copyedit.py from image1 into image2 -vv remove volume /xyz
and start the container i get the message
pulling image2 ...
ERROR: manifest for image2 not found: manifest unknown: manifest unknown
What can I do?
Thanks Uwe
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.