fraunhoferfokus / deckschrubber Goto Github PK

Quick Installation failed

# go version
go version go1.16 linux/amd64

# go get github.com/fraunhoferfokus/deckschrubber
go: downloading github.com/fraunhoferfokus/deckschrubber v0.6.0
go: downloading github.com/docker/distribution v2.7.1+incompatible
go: downloading github.com/Sirupsen/logrus v1.8.0
go: downloading golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
go get: github.com/Sirupsen/logrus@none updating to
        github.com/Sirupsen/[email protected]: parsing go.mod:
        module declares its path as: github.com/sirupsen/logrus
                but was required as: github.com/Sirupsen/logrus

go get github.com/fraunhoferfokus/deckschrubber
package context: unrecognized import path "context" (import path does not begin with hostname)
package github.com/distribution/distribution/v3/reference: cannot find package "github.com/distribution/distribution/v3/reference" in any of:
/usr/lib/go-1.6/src/github.com/distribution/distribution/v3/reference (from $GOROOT)
/root/deckschrubber/src/github.com/distribution/distribution/v3/reference (from $GOPATH)
package github.com/distribution/distribution/v3/uuid: cannot find package "github.com/distribution/distribution/v3/uuid" in any of:
/usr/lib/go-1.6/src/github.com/distribution/distribution/v3/uuid (from $GOROOT)
/root/deckschrubber/src/github.com/distribution/distribution/v3/uuid (from $GOPATH)
package github.com/distribution/distribution/v3: cannot find package "github.com/distribution/distribution/v3" in any of:
/usr/lib/go-1.6/src/github.com/distribution/distribution/v3 (from $GOROOT)
/root/deckschrubber/src/github.com/distribution/distribution/v3 (from $GOPATH)
package github.com/distribution/distribution/v3/manifest: cannot find package "github.com/distribution/distribution/v3/manifest" in any of:
/usr/lib/go-1.6/src/github.com/distribution/distribution/v3/manifest (from $GOROOT)
/root/deckschrubber/src/github.com/distribution/distribution/v3/manifest (from $GOPATH)
package github.com/distribution/distribution/v3/digestset: cannot find package "github.com/distribution/distribution/v3/digestset" in any of:
/usr/lib/go-1.6/src/github.com/distribution/distribution/v3/digestset (from $GOROOT)
/root/deckschrubber/src/github.com/distribution/distribution/v3/digestset (from $GOPATH)
package github.com/distribution/distribution/v3/registry/api/errcode: cannot find package "github.com/distribution/distribution/v3/registry/api/errcode" in any of:
/usr/lib/go-1.6/src/github.com/distribution/distribution/v3/registry/api/errcode (from $GOROOT)
/root/deckschrubber/src/github.com/distribution/distribution/v3/registry/api/errcode (from $GOPATH)
package github.com/distribution/distribution/v3/registry/api/v2: cannot find package "github.com/distribution/distribution/v3/registry/api/v2" in any of:
/usr/lib/go-1.6/src/github.com/distribution/distribution/v3/registry/api/v2 (from $GOROOT)
/root/deckschrubber/src/github.com/distribution/distribution/v3/registry/api/v2 (from $GOPATH)
package github.com/distribution/distribution/v3/registry/client/auth/challenge: cannot find package "github.com/distribution/distribution/v3/registry/client/auth/challenge" in any of:
/usr/lib/go-1.6/src/github.com/distribution/distribution/v3/registry/client/auth/challenge (from $GOROOT)
/root/deckschrubber/src/github.com/distribution/distribution/v3/registry/client/auth/challenge (from $GOPATH)
package github.com/distribution/distribution/v3/registry/client/transport: cannot find package "github.com/distribution/distribution/v3/registry/client/transport" in any of:
/usr/lib/go-1.6/src/github.com/distribution/distribution/v3/registry/client/transport (from $GOROOT)
/root/deckschrubber/src/github.com/distribution/distribution/v3/registry/client/transport (from $GOPATH)
package github.com/distribution/distribution/v3/registry/storage/cache: cannot find package "github.com/distribution/distribution/v3/registry/storage/cache" in any of:
/usr/lib/go-1.6/src/github.com/distribution/distribution/v3/registry/storage/cache (from $GOROOT)
/root/deckschrubber/src/github.com/distribution/distribution/v3/registry/storage/cache (from $GOPATH)
package github.com/distribution/distribution/v3/registry/storage/cache/memory: cannot find package "github.com/distribution/distribution/v3/registry/storage/cache/memory" in any of:
/usr/lib/go-1.6/src/github.com/distribution/distribution/v3/registry/storage/cache/memory (from $GOROOT)
/root/deckschrubber/src/github.com/distribution/distribution/v3/registry/storage/cache/memory (from $GOPATH)
package math/bits: unrecognized import path "math/bits" (import path does not begin with hostname)

Describe the bug
When having OCI images (f.e. created with Google Jib) in the registry, cleanup fails since the manifests could not be retrieved.
Logoutput:

time="2021-09-11T11:07:34Z" level=error msg="Could not fetch tag!" repo=nbs/frontend tag=a259bf4
time="2021-09-11T11:07:34Z" level=error msg="Error obtaining tag data - skipping this repo" repo=nbs/frontend

To Reproduce
Add OCI images to repository, run deckschrubber.

Expected behavior
Cleanup of OCI images works.

Additional context
To support OCI images at least the http accept header must include

application/vnd.oci.image.manifest.v1+json

The registry logs contain the following hint

127.0.0.1 - - [11/Sep/2021:11:07:34 +0000] "HEAD /v2/nbs/frontend/manifests/a259bf4 HTTP/1.1" 404 122 "" "Go-http-client/1.1"
time="2021-09-11T11:07:34.042602317Z" level=error msg="response completed with error" err.code="manifest unknown" err.message="OCI manifest found, but accept header does not support OCI manifests" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=380904d3-468c-4eec-a0d9-90592a37106f http.request.method=GET http.request.remoteaddr="127.0.0.1:47562" http.request.uri="/v2/nbs/frontend/manifests/a259bf4" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=12.158518ms http.response.status=404 http.response.written=122 vars.name="nbs/frontend" vars.reference=a259bf4 

The image is created with Jib in OCI image format, registry 2.7.1 is used.

Error:

$ docker run -i -t  golang go get github.com/fraunhoferfokus/deckschrubber
# github.com/fraunhoferfokus/deckschrubber
src/github.com/fraunhoferfokus/deckschrubber/main.go:88:30: too many arguments in call to client.NewRegistry
        have ("context".Context, string, nil)
        want (string, http.RoundTripper)
src/github.com/fraunhoferfokus/deckschrubber/main.go:128:36: too many arguments in call to client.NewRepository
        have ("context".Context, reference.Named, string, nil)
        want (reference.Named, string, http.RoundTripper)

Commit with changes in NewRegistry: distribution/distribution@2c58ce1

We sometimes update images in the registry, so a newer image will get an already used tag, the old image will be left without a tag.

REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
reg:443/app   latest    4e2bace08a25   4 seconds ago   214MB
reg:443/app   9.9.49    4e2bace08a25   4 seconds ago   214MB
reg:443/app   9.9.48    2c4a5c62ce6c   42 hours ago    214MB
reg:443/app   9.9.47    9b3ab1154b85   2 days ago      214MB
reg:443/app   <none>    13ba38c89f33   2 days ago      214MB
reg:443/app   9.9.46    d3949a15fd93   4 days ago      214MB
reg:443/app   <none>    beb24bafaac4   4 days ago      214MB

How can I delete all images without any tag? If it's possible, it would be great to add this case to the examples.

Hey, I tried to use the docker image of this tool and I came to know that it is not working. I have tried multiple times but it is not working. It is detecting the images correctly but it is not deleting it.

Thank you for this tool!

I'd like to use it to cleanup old images in our Gitlab-registry. Unfortunately it looks like we're using Bearer-Authentication which is not supported by the current version of deckschrubber. After passing username and password, I get the following exception:
HEADERS: http.Header{"Www-Authenticate":[]string{"Bearer realm=\"https://ourgitlab.com/jwt/auth\",service=\"container_registry\",scope=\"registry:catalog:*\""}, "X-Content-Type-Options":[]string{"nosniff"}, "Server":[]string{"nginx"}, "D ate":[]string{"Tue, 12 Mar 2019 11:53:35 GMT"}, "Content-Type":[]string{"application/json; charset=utf-8"}, "Content-Length":[]string{"145"}, "Connection":[]string{"keep-alive"}, "Docker-Distribution-Api-Version":[]string{"registry/2.0"}} FATA[0002] Error while fetching repositories! (err: unauthorized: authentication required)

I'm trying to delete images from my registry but only get a "Could not delete image!".

My registry is the docker container registry:2
I am using the binary 0.2.0.
What am I missing?

root@registry:~# ./deckschrubber -repo hello-world -latest 0 -debug
INFO[0000] Successfully fetched repositories.            count=5 entries=[akechita arangodb hello-world l5-elixir l5-gauge]
DEBU[0000] Ignore non matching repository (-repo=hello-world)  entry=akechita repo=akechita
DEBU[0000] Ignore non matching repository (-repo=hello-world)  entry=arangodb repo=arangodb
DEBU[0000] Successfully created repository object.       repo=hello-world
DEBU[0000] Fetching tag...                               repo=hello-world tag=latest
DEBU[0000] Fetching manifest...                          repo=hello-world tag=latest
DEBU[0000] Parsing manifest details...                   repo=hello-world tag=latest
DEBU[0000] Fetching blob                                 repo=hello-world tag=latest
DEBU[0000] Ignore non matching repository (-repo=hello-world)  entry=l5-elixir repo=l5-elixir
DEBU[0000] Ignore non matching repository (-repo=hello-world)  entry=l5-gauge repo=l5-gauge
DEBU[0000] Analyzing tags...                             repo=hello-world
INFO[0000] Delete outdated image (-dry=false)            repo=hello-world tag=latest time=2017-01-13 22:50:56.415736637 +0000 UTC
ERRO[0000] Could not delete image!                       repo=hello-world tag=latest

Hi,

The private registry requires username/password login
How do I input the username/password when executing the script ?

$GOPATH/bin/deckschrubber -registry https:/private.registry/repository/docker/ -tag latest -dry
FATA[0000] Error while fetching repositories! (err: unauthorized: access to the requested resource is not authorized)

Thanks

The private registry is running in Ubuntu 17.10. deckschrubber version v 0.3.0. I have tried executing binaries directly and compiling it locally both are giving the same issue. Is there anyone facing this issue?

First of all: Thanks for providing this binary!

Would be nice to have this in a ready to use docker image ;-)

It seems that when I run my registry server with SSO Auth (via Keycloak) then Deckschrubber cannot authenticate. The error I get is:

deckschrubber -registry $REGISTRY_URL -user $REGISTRY_USERNAME -password $REGISTRY_PASSWORD -repos=9999 -latest 3 
FATA[0000] Error while fetching repositories! (err: unauthorized: authentication required)

In my registry server the error is:

^[[1;5Ftime="2021-06-24T11:35:40.647713286Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.11.2 http.request.host=registry.example.com http.request.id=xxxxxx http.request.method=GET http.request.remoteaddr=xxx.xx.xxx.xxx http.request.uri="/v2/_catalog?n=9999" http.request.useragent="Go-http-client/1.1" 
172.20.0.6 - - [24/Jun/2021:11:35:40 +0000] "GET /v2/_catalog?n=9999 HTTP/1.0" 401 145 "" "Go-http-client/1.1"

Is this a known issue and if yes, will this be solved at some point?

best
Martin

Hello,

I'm trying to use this tool but I get a NPD from a basic command:

➜  ~ deckschrubber -registry https://user:[email protected]/ -dry
INFO[0000] Successfully fetched repositories.            count=1 entries=[yolo]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x124cdcf]

goroutine 1 [running]:
github.com/docker/distribution/registry/api/v2.(*URLBuilder).BuildTagsURL(0xc420499080, 0x0, 0x0, 0xc42049d6b0, 0xc4203dd6f8, 0xc4203dd6b0, 0x1010a58)
	/Users/bordel/go/src/github.com/docker/distribution/registry/api/v2/urls.go:134 +0x6f
github.com/docker/distribution/registry/client.(*tags).All(0xc42049d5c0, 0x14375c0, 0x146cc40, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/bordel/go/src/github.com/docker/distribution/registry/client/repository.go:212 +0x92
main.main()
	/Users/bordel/go/src/github.com/fraunhoferfokus/deckschrubber/main.go:109 +0x97d

Do you have any idea?

I have tried to use deckschrubber for clean my docker registry, but I receive a message for every image:
ERRO[0087] Could not fetch manifest! repo=my-private-ui tag=0.1.0-dev-3107-SNAPSHOT
ERRO[0087] Could not fetch manifest! repo=my-private-ui tag=0.1.0-dev-2472-SNAPSHOT
ERRO[0087] Could not fetch manifest! repo=my-private-ui tag=0.1.0-dev-3140-SNAPSHOT
ERRO[0087] Could not fetch manifest! repo=my-private-ui tag=0.1.0-dev-3172-SNAPSHOT
ERRO[0087] Could not fetch manifest! repo=my-private-ui tag=0.1.0-dev-3302-SNAPSHOT
INFO[0087] Ignore 1 latest matching images (-latest=1) repo=my-private-ui tag=2.0.3497-feat-246-SNAPSHOT time=2017-07-25 12:16:35.976773554 +0000 UTC

I'm trying to use deckschrubber with dockerhub, but always get the error message below.

Version: I'm using the latest master.

$  deckschrubber -day 30 -user myusername -password mypassword -registry https://registry.hub.docker.com/r/myorg/myregistry
FATA[0000] Error while fetching repositories! (err: invalid character '<' looking for beginning of value)

Notably, I get the same error even when don't provide a username & password, and even when I provide the wrong username and password!

(This feels like a very short issue-report, but I can't think of any more detail I can add!)

Please add option to disable SSL certificate validation

I just unintentionally deleted a couple of images from my registry by running deckschrubber without any parameters.

It'd be safer to just print the help when deckschrubber is called. And require some parameter like -run or -repo=N to really cleanup the registry

I have one image with three tags

docker/moodle:latest
docker/moodle:12376
docker/moodle:3.2-stable

The command

deckschrubber -registry "${REGISTRY}" -tag '^\d+$' -repo docker/moodle -latest 1 -repos 999

deletes the Image. Is it possible to delete an image only if there is only one tag?

Hi,
How am i able to use basic auth? can't seem to find it in the docs.

Thanks :)

What do I have to understand under the following messages?

$ deckschrubber -registry "${REGISTRY}" -latest 1 -repo itbh/hopon-fahrplan -repos 999 -dry
...
ERRO[0000] Could not fetch tag!                          repo=itbh/hopon-fahrplan tag=14910
ERRO[0000] Could not fetch tag!                          repo=itbh/hopon-fahrplan tag=14915
ERRO[0000] Could not fetch tag!                          repo=itbh/hopon-fahrplan tag=14913
ERRO[0000] Could not fetch tag!                          repo=itbh/hopon-fahrplan tag=14916
ERRO[0000] Could not fetch tag!                          repo=itbh/hopon-fahrplan tag=14912
ERRO[0000] Could not fetch tag!                          repo=itbh/hopon-fahrplan tag=14911

Is it possible to use longer wait times ? Seems that it fails when there are two many of them (thousands).

$GOPATH/bin/deckschrubber -dry -registry https://test-registry.example.com:5000 -repo 'test/test-redis' -day 1 -repos 100 -debug

DEBU[0005] Successfully created repository object.       repo=test/test-redis
FATA[0035] Couldn't fetch tags! (err: received unexpected HTTP status: 504 Gateway Time-out)  repo=test/test-redis

I think there is a paging issue. It only gets the top 5 repositories in alphabetical order from the registry. I have over 20 repositories.