gdraheim / docker-copyedit Goto Github PK

View Code? Open in Web Editor NEW

189.0 189.0 36.0 159 KB

edit docker image metadata (including remove docker volumes)

License: European Union Public License 1.2

Python 97.36% Makefile 2.50% Dockerfile 0.14%

docker-copyedit's People

Contributors

Stargazers

Watchers

docker-copyedit's Issues

Manifest file already exists

Hi Guido,

I wanted to remove the /database volume from the ibmcom/db2 image.
But when I process it with your script using:
docker-copyedit.py FROM ibmcom/db2:latest INTO db2:1.0 REMOVE ALL VOLUMES
I get the error that the manifest file already exists:
Traceback (most recent call last): File "docker-copyedit.py", line 929, in <module> edit_image(inp, out, commands) File "docker-copyedit.py", line 301, in edit_image changed = edit_datadir(datadir, out_tag, edits) File "docker-copyedit.py", line 723, in edit_datadir os.rename(manifest_filename + ".tmp", manifest_filename) FileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: 'load.tmp\\data\\manifest.json.tmp' -> 'load.tmp\\data\\manifest.json'

I tried to fix it but I'am a novice to python.

Thanks Mario

Python 3 support?

Alpine has recently got rid of Python2, which is needed for this (awesome) script.

While this is workaround-able for now I was wondering if there was ever any plans to support Python 3?

Removing ports doesn't work

I've tried

/docker-copyedit.py FROM nginx:alpine INTO nginx:noexpose REMOVE PORT 80
/docker-copyedit.py FROM nginx:alpine INTO nginx:noexpose remove all ports
Both create a new container, but without the EXPOSE removed

docker inspect nginx:noexpose | jq '.[].ContainerConfig.ExposedPorts'
{
  "80/tcp": {}
}

Copyedit script no longer executable

Hi, thanks for this handy util!

The most recent commit changed the permission bits on the copyedit script. Its not difficult to get around - but it means the docs dont work out of the box

can't set architecture variant

docker-copyedit doesn't allow us to change the architecture "variant" metadata.

For example,

$ docker inspect arm32v7/ubuntu | jq -r .[].Variant
v7

$ ./docker-copyedit.py FROM arm32v7/ubuntu into foo set variant foo
ERROR:edit:unknown edit command starting with set variant

remove all volumes doesn't work

Hi,

Probably, I did something wrong, but it was from README:
in readme:

./docker-copyedit.py \
 FROM image1 INTO image2 REMOVE ALL VOLUMES

but it doesn't work:

[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql:5.6 INTO mysql-novol:5.6 REMOVE ALL VOLUMES
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql:5.6 INTO mysql-novol REMOVE ALL VOLUMES
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql INTO mysql-novol REMOVE ALL VOLUMES
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:all is equivalent to '*' pattern for 'volumes' or 'ports'
ERROR:edit:CMD docker tag mysql mysql-novol
ERROR:edit:EXIT 1
ERROR:edit:STDOUT
ERROR:edit:STDERR Error response from daemon: No such image: mysql:latest

Traceback (most recent call last):
  File "./docker-copyedit.py", line 538, in <module>
    edit_image(inp, out, commands)
  File "./docker-copyedit.py", line 91, in edit_image
    sh(cmd.format(**locals()))
  File "./docker-copyedit.py", line 41, in sh
    raise Exception("shell command failed")
Exception: shell command failed
Exit code: 1
[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$

But this works:

[qa@flowqe3:~/up/flow-demo-uat/docker-copyedit]$ ./docker-copyedit.py FROM mysql:5.6 INTO mysql-novol:5.6 -vv REMOVE VOLUME /var/lib/mysql
INFO:edit:written new load.tmp/data/19706e2768c04c8301865815ea821c173e3715f40f470a17a057b667db610d54.json
INFO:edit:removed old load.tmp/data/1f47fade220d0ddedb916156cf4d7121a020afff03c19b4638ac494c5d1eca63.json

Possible to alter arm64 images on amd64 machine?

Using moby buildkit and buildx bake I can build arm64 images on my amd64 machine. Is there a way to use your script to alter an arm64 image but doing so on a amd64 machine? I assume it would, like moby buildkit, have to do so in a dockerized arm64 container using qemu.

Manifest not found

Hi Guido,

I wanted to remove the /xyz volume from the image1.
But when I process it with your script using:
python docker-copyedit.py from image1 into image2 -vv remove volume /xyz

and start the container i get the message

pulling image2 ...
ERROR: manifest for image2 not found: manifest unknown: manifest unknown

What can I do?

Thanks Uwe

Tabs/spaces broken

Hi, I've spotted that the main python file has a number of places where indent is broken. Can you please re-tab (re-space) it?

set-label causes TypeError on images with no existing labels

Running docker-copyedit with a 'set-label name value' leads to an error if the image has no labels. It works fine when an image already has some label(s). The Python stack trace appears thus:

$ docker-copyedit.py from postgres:13.5 into b:latest set-label somename somevalue
  File "docker-copyedit.py", line 938, in <module>
    edit_image(inp, out, commands)
  File "docker-copyedit.py", line 310, in edit_image
    changed = edit_datadir(datadir, out_tag, edits)
  File "docker-copyedit.py", line 539, in edit_datadir
    if target in config[CONFIG][key]:
TypeError: argument of type 'NoneType' is not iterable

An example of an image with no existing labels is postgres:13.5. Here is the 'docker inspect' of this image:

[
    {
        "Id": "sha256:e01c76bb1351f3fb966f97cf108f1ff586056a517d56c9ec2629ede6be756691",
        "RepoTags": [
            "postgres:13.5"
        ],
        "RepoDigests": [
            "postgres@sha256:2e2e213c0fe4d1229373bebf114f255f68d86fba9e9f30e9bcc08e92cdf7b5b4"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2022-01-27T01:11:05.913589796Z",
        "Container": "32bd06472b5cfa54b87f8772fe083d5bab2b1144868ac6ce408b757f54bb982a",
        "ContainerConfig": {
            "Hostname": "32bd06472b5c",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "5432/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/13/bin",
                "GOSU_VERSION=1.14",
                "LANG=en_US.utf8",
                "PG_MAJOR=13",
                "PG_VERSION=13.5-1.pgdg110+1",
                "PGDATA=/var/lib/postgresql/data"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "CMD [\"postgres\"]"
            ],
            "Image": "sha256:8004cdfe9dee6a2263d8cc24d9f67d6c41dd0fc090c1be01b98418d9d2497bce",
            "Volumes": {
                "/var/lib/postgresql/data": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {},
            "StopSignal": "SIGINT"
        },
        "DockerVersion": "20.10.7",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "5432/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/13/bin",
                "GOSU_VERSION=1.14",
                "LANG=en_US.utf8",
                "PG_MAJOR=13",
                "PG_VERSION=13.5-1.pgdg110+1",
                "PGDATA=/var/lib/postgresql/data"
            ],
            "Cmd": [
                "postgres"
            ],
            "Image": "sha256:8004cdfe9dee6a2263d8cc24d9f67d6c41dd0fc090c1be01b98418d9d2497bce",
            "Volumes": {
                "/var/lib/postgresql/data": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": null,
            "StopSignal": "SIGINT"
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 371198583,
        "VirtualSize": 371198583,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/4b5c59b2df623b68662dff9dd759e8b1e5e15ba1065d4cd72737e709d44e38c8/diff:/var/lib/docker/overlay2/79cdf68ee1096872e5a416d505793aa391f2a1ee0b0d25bc406f84954bb3ac94/diff:/var/lib/docker/overlay2/445a42c7637f1b00fd12a9249f55171bdd31ffcbb9de169467030f9cb6073001/diff:/var/lib/docker/overlay2/a1fcb9929fb699e647fc716627cc4bfd64413af74b9af3442c9c8d08784d1bd0/diff:/var/lib/docker/overlay2/d140bf2c2f284959d56fec2afb7b3ef2acfa0ed4ce35188208d775e2212b66c1/diff:/var/lib/docker/overlay2/754602e81673683379f5ca51ac26d3a110ade09ded64671d611e9f77b7468198/diff:/var/lib/docker/overlay2/bd0b0f158d1bae5922aadbf7ce49123e5ea0db06f4d09f36a9bed5a27514d7e1/diff:/var/lib/docker/overlay2/186b4af03de1f2231bfc00d2a5026d5e0ea2741f2f7f3b86726dcbfbe4aac4e8/diff:/var/lib/docker/overlay2/1a582292220a3e41a400d932828548029d48416c4d3b1673a2b9fe253978c515/diff:/var/lib/docker/overlay2/b37a11f200770cec046366d4a5edf2d921c3873234f3f540dd2e94d5a0886fb9/diff:/var/lib/docker/overlay2/dbf7b34785d5b0ac0936370bb6d2a00a886c083e745df51a6b185efe68b9bc75/diff:/var/lib/docker/overlay2/39cf89b7f028e5f5aab843dc56def8f5817dc182f416ad4d59df4a09865e134f/diff",
                "MergedDir": "/var/lib/docker/overlay2/a23616e078d0e485b807e0affe1b0e35dc315a1b80ad4439572a02b9dab8dd6c/merged",
                "UpperDir": "/var/lib/docker/overlay2/a23616e078d0e485b807e0affe1b0e35dc315a1b80ad4439572a02b9dab8dd6c/diff",
                "WorkDir": "/var/lib/docker/overlay2/a23616e078d0e485b807e0affe1b0e35dc315a1b80ad4439572a02b9dab8dd6c/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:7d0ebbe3f5d26c1b5ec4d5dbb6fe3205d7061f9735080b0162d550530328abd6",
                "sha256:504d3c78f1942134301a91d83b6d2102b2c398eab2c81a76f62b9eb33f8c000b",
                "sha256:8c1f84338867a554126fe7100341ff8174fe71e13518389803e4e1050aa8f200",
                "sha256:10e60ad30e5f437ff02dd7273545deeeabb0d07595dae73cc0f1ea2d38a2956f",
                "sha256:96d824739a641ee2e8f36bcd8ff576c7ba72029184aaa7c8ec5c418f5c60802f",
                "sha256:a4b9517dfa8c2c05e3cd3423f52535a7e551ac7e4c026541742c054b19b1c5e4",
                "sha256:f2e4fedb957df6c5380f46b9b08e2554b1936a553acbc0c77579faf53724feda",
                "sha256:c1cf0fe2cfa15adc5603ebd8d0047d12afe9b82a634784c8b3967be55ffc33f2",
                "sha256:b7035179c078b0703b5d5f322123535179489aa8b0eb820e317862a6448ff240",
                "sha256:6fc0879b8f3a07b74f6de0a76c22175ec89d0dcc46827b3f433b736e977a66b7",
                "sha256:954be07f25afe8092888629a0603fc6fc773c093bd5b8d9f20e6bfdeff1b8a06",
                "sha256:ae0a327d2532a9199bb5d3f7a198315f0ea092eb9c9a7719a0bec76b22da307e",
                "sha256:d69fa3581c263ec07ff28eb3edd558c4fb20e4bbad8a039eec0c005cc3152819"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

Perhaps the $.Config.Labels:null is the culprit.

dockerize you application

It would be nice if you dockerized your application.
We could use it in ci/cd processes.

speed up docker save process

the tar file generated from "docker save" is optionally keeped by the control of KEEPDIR , which I understand mostly for debugging purpose.
but even KEEPDIR =0, the tar file is always saved on disk, only deleted afterwards
I am a bit impatient and did following patch , pipe-lining with tee , to save disk IO, save time

23c23
< KEEPDIR = 1

KEEPDIR = 0
251,256c251,254
< if KEEPDIR >= 2:
< cmd = "docker save {inp} |tee {inputfile}|tar x -f - -C {datadir}"
< sh(cmd.format(**locals()))
< else:
< cmd = "docker save {inp} |tar x -f - -C {datadir}"
< sh(cmd.format(**locals()))

    cmd = "docker save {inp} -o {inputfile}"
    sh(cmd.format(**locals()))
    cmd = "tar xf {inputfile} -C {datadir}"
    sh(cmd.format(**locals()))

280,284c278,282
< # if KEEPDIR >= 2:
< # logg.warning("keeping %s", inputfile)
< # else:
< # if os.path.exists(inputfile):
< # os.remove(inputfile)

    if KEEPDIR >= 2:
        logg.warning("keeping %s", inputfile)
    else:
        if os.path.exists(inputfile):
            os.remove(inputfile)

STDERR: invalid tag

When trying to remove a certain port of an image, I only receive this error in the last step of the process:

DEBUG:edit:changed 1 layer metadata
ERROR:edit:CMD docker load -i load.tmp/ready.tar
ERROR:edit:EXIT 1
ERROR:edit:STDOUT
ERROR:edit:STDERR invalid tag "jannikz/gluster-prod"

Traceback (most recent call last):
  File "./docker-copyedit.py", line 706, in <module>
    edit_image(inp, out, commands)
  File "./docker-copyedit.py", line 252, in edit_image
    sh(cmd.format(**locals()))
  File "./docker-copyedit.py", line 43, in sh
    raise Exception("shell command failed")
Exception: shell command failed

No matter what tag I'm entering .. did you ever face a problem like this?

./docker-copyedit.py \
FROM jannikz/gluster-prod:latest INTO jannikz/gluster-prod -vvv REMOVE PORT 111
DEBUG:edit:image parsing = jannikz/gluster-prod:latest
DEBUG:edit:.registry = None
DEBUG:edit:.image = jannikz/gluster-prod
DEBUG:edit:.tag = :latest
DEBUG:edit:image parsing = jannikz/gluster-prod
DEBUG:edit:.registry = None
DEBUG:edit:.image = jannikz/gluster-prod
DEBUG:edit:.tag = None
DEBUG:edit:total 1484224

__version__ = "1.2.1402"
Docker version 18.06.1-ce, build e68fc7a

gdraheim / docker-copyedit Goto Github PK

docker-copyedit's People

Contributors

Stargazers

Watchers

Forkers

docker-copyedit's Issues

23c23 < KEEPDIR = 1

280,284c278,282 < # if KEEPDIR >= 2: < # logg.warning("keeping %s", inputfile) < # else: < # if os.path.exists(inputfile): < # os.remove(inputfile)

Recommend Projects

Recommend Topics

Recommend Org

23c23
< KEEPDIR = 1

280,284c278,282
< # if KEEPDIR >= 2:
< # logg.warning("keeping %s", inputfile)
< # else:
< # if os.path.exists(inputfile):
< # os.remove(inputfile)