I'm using docker-mirror to manage a very large number of docker images. If I run this on a new machine, each image needs to be pulled and re-tagged regardless of whether or not that image tag already exists in the target mirror repo. It would be great if there was a config option to change this behavior and have docker-mirror check the target repo first to validate whether the proposed tag exists first. This way it would only pull the image from the source repo when the tag doesn't exist in the target. This saves time and disk space, and avoids the issue of dockerhub pull rate-limiting.

I'm not sure whether SHA validation would be necessary or not, but it could be achieved if the arch was also added to the config, an env var, or otherwise read from the system.

I’d like to have the ability to use docker-mirror to support mirroring of non-Docker repo images from Quay container registry and Google Container Registry (GCR) to be published in public ECR repos.

An example YAML file would look as follows:

target:
  registry: ACCOUNT_ID.dkr.REGION.amazonaws.com
  prefix: "hub/"
repositories:
  - name: quay.io/opentelemetry/opentelemetry-operator
      - "v*"

  - name: gcr.io/kubebuilder/kube-rbac-proxy
      - max_tags: 10

We are working on an enhancement which we plan to contribute to the project.

cc: @Saber-W

There are a couple of related issues open which will hopefully also get addressed by our upcoming PR.

Related Issues:
#75
#70

Hey Folks,

I'm currently unable to pick up a specified Redis image of 5.0.3, I've tried with 5.0.3-stretch, etc. However it doesn't seem to load the tag, however, when I had switched it to a wildcard on the version, and set it to the past 14 days. It seems to scrape all the relevant tags and upload them just fine to the destination.

Could ya'll take a look?

config.yaml

---
target:
  registry: $account_id.dkr.ecr.us-west-2.amazonaws.com
  prefix: "hub/"

repositories:
  - name: redis
    match_tag:
      - "5.0.3"

What I did.

$ docker-mirror
INFO[0000] Creating Docker client
INFO[0000] Connected to Docker daemon: docker-desktop @ 20.10.0
INFO[0000] Creating AWS client
INFO[0000] Loading list of ECR repositories
INFO[0000] Done loading ECR repositories
INFO[0000] Repository mirror completed                   full_repo=redis num_tags=0 repo=redis
INFO[0000] Done

Wildcard Config

---
target:
  registry: $account_id.dkr.ecr.us-west-2.amazonaws.com
  prefix: "hub/"

repositories:
  - name: redis
    match_tag:
      - "5*"
    max_tag_age: 14d

Wildcard uploads successfully.

INFO[0000] Creating Docker client
INFO[0000] Connected to Docker daemon: docker-desktop @ 20.10.0
INFO[0000] Creating AWS client
INFO[0000] Loading list of ECR repositories
INFO[0000] Done loading ECR repositories
INFO[0000] Start mirror tag                              full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12
INFO[0000] Starting docker pull                          full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12
INFO[0001] Completed docker pull in 980.730612ms         full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12
INFO[0001] Starting docker tag                           full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12
INFO[0001] Completed docker tag in 7.22202ms             full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12
INFO[0001] Starting docker push                          full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12

Other Debugging Items

OS: macOS Catalina 10.15.6
golang version:go1.15.5 darwin/amd64
Authentication Method: aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin $account_id.dkr.ecr.us-west-2.amazonaws.com

After building with go build using Go 1.17 on macOS, docker-mirror panics:

$ go version
go version go1.17.3 darwin/amd64
$ go build
$ ./docker-mirror --help
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x7fff20412c9e]

runtime stack:
runtime: unexpected return pc for runtime.sigpanic called from 0x7fff20412c9e
stack: frame={sp:0x7ffeefbff368, fp:0x7ffeefbff3b8} stack=[0x7ffeefb80408,0x7ffeefbff470)
0x00007ffeefbff268:  0x01007ffeefbff288  0x0000000000000004
0x00007ffeefbff278:  0x000000000000001f  0x00007fff20412c9e
0x00007ffeefbff288:  0x0b01dfacedebac1e  0x0000000000000001
0x00007ffeefbff298:  0x0000000004035fd1 <runtime.throw+0x0000000000000071>  0x00007ffeefbff338
0x00007ffeefbff2a8:  0x0000000004423a1c  0x00007ffeefbff2f0
0x00007ffeefbff2b8:  0x0000000004036288 <runtime.fatalthrow.func1+0x0000000000000048>  0x0000000004719840
0x00007ffeefbff2c8:  0x0000000000000001  0x0000000000000001
0x00007ffeefbff2d8:  0x00007ffeefbff338  0x0000000004035fd1 <runtime.throw+0x0000000000000071>
0x00007ffeefbff2e8:  0x0000000004719840  0x00007ffeefbff328
0x00007ffeefbff2f8:  0x0000000004036210 <runtime.fatalthrow+0x0000000000000050>  0x00007ffeefbff308
0x00007ffeefbff308:  0x0000000004036240 <runtime.fatalthrow.func1+0x0000000000000000>  0x0000000004719840
0x00007ffeefbff318:  0x0000000004035fd1 <runtime.throw+0x0000000000000071>  0x00007ffeefbff338
0x00007ffeefbff328:  0x00007ffeefbff358  0x0000000004035fd1 <runtime.throw+0x0000000000000071>
0x00007ffeefbff338:  0x00007ffeefbff340  0x0000000004036000 <runtime.throw.func1+0x0000000000000000>
0x00007ffeefbff348:  0x000000000442aa54  0x000000000000002a
0x00007ffeefbff358:  0x00007ffeefbff3a8  0x000000000404b776 <runtime.sigpanic+0x0000000000000396>
0x00007ffeefbff368: <0x000000000442aa54  0x0000000004719840
0x00007ffeefbff378:  0x00007ffeefbff3e8  0x00000000040292a6 <runtime.(*mheap).allocSpan+0x0000000000000546>
0x00007ffeefbff388:  0x000000c00020c000  0x0000000000002000
0x00007ffeefbff398:  0x0000000000000008  0x0000000004719840
0x00007ffeefbff3a8:  0x00007ffeefbff3f0 !0x00007fff20412c9e
0x00007ffeefbff3b8: >0x00007ffeefbff3f0  0x00000000046bc000
0x00007ffeefbff3c8:  0x0000000000000648  0x000000000427fa05 <golang.org/x/sys/unix.libc_ioctl_trampoline+0x0000000000000005>

I can include more of the traceback if you want but it doesn't seem super useful.

Seems like updating the x/sys package will address this issue. From golang/go#46763

docker-mirror does not support ghcr.io , please add it.
Also it would be nice to add new container registry via additional option

Our developers recently started to receive Apple Sillicon (M1) laptops.

We would like to be able to mirror both amd64 and arm version of public images.

I can take a stab at a PR, if you can point out where in the code we would add such a functionality.

Using dind and docker-mirror within container i can push manually to my private registry using the helper. (using .docker/conf.json)

Not seem to work when calling the app because it rely to .dockercfg

Add a flag to clean up (docker system prune) after the run

I'm considering using this tool in our CI system.

But it not have specific versions tags in dockerhub is would hard to pin to a specific version and it might break easily down the road for us.

Also it a small security concern since a tool that would have access to AWS credentials, it would prefer having more visibility into which version we are using. i.e. just using latest we can't verify the checksums.

Wasn't sure if I could upload to a public registry since the docs only mention private registries, but got it working like so

Login:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

config.yaml

target:
  registry: public.ecr.aws

  # Default or custom alias
  prefix: "g6hzp2z3/"

repositories:
  - name: alpine # repository needs to be manually created
    match_tag:
    - "3.10"

Let me know if you want to split them into two issues.

I can make the PR.

In order to have clean history I propose un-checking the merge button in the settings.

With that done the default prompt when merging pull requests results in a squash making it cleaner and reducing the noise and churn within the PR itself.

Dependabot couldn't find a Gopkg.toml for this project.

Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /Gopkg.toml.

If this isn't a Go project, or if it is a library, you may wish to disable updates for it from within Dependabot.

View the update logs.

Hi!

Your tool seems to be the solution for us to mirror distinct Images from DockerHub, but what I don't understand is if your tool can be used with a corporate private registry instead of ECR? Is ECR mandatory?

Regards,
Holger

Dependabot couldn't find a Gopkg.toml for this project.

Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /Gopkg.toml.

If this isn't a Go project, or if it is a library, you may wish to disable updates for it from within Dependabot.

View the update logs.

The Docker Hub and Docker Registry APIs appear to be different. Different JSON is returned when querying the two most similar URLs:

https://registry.hub.docker.com/v2/repositories/jippi/hashi-ui/tags/?page_size=2048
http://my.docker.registry.com/v2/namespace/repo/tags/list?n=2048

It looks like mirror.go is expecting 'count' etc, that is not present in the Registry JSON. I'm sure there are other differences but I did not pursue much further once I realized the APIs were different.

Scrubbed example of Registry JSON:
{"name":"namespace/reponame","tags":["0.1.0-10","0.1.0-11","0.1.0-13","0.1.0-14","0.1.0-15","0.1.0-16","0.1.0-17","0.1.0-18","0.1.0-19","0.1.0-2","0.1.0-20","0.1.0-21","0.1.0-22","0.1.0-23","0.1.0-24","0.1.0-25","0.1.0-26","0.1.0-27","0.1.0-28","0.1.0-29","0.1.0-3","0.1.0-4","0.1.0-6","0.1.0-7","0.1.0-8","0.1.0-9","latest-ci","latest-qa","latest"]}

I was attempting to mirror quay.io/coreos/kube-state-metrics but it failed with:

time="2021-07-09T04:11:44Z" level=warning msg="Get https://registry.hub.docker.com/v2/repositories/quay.io/coreos/kube-state-metrics/tags/?page_size=2048 [registry.hub.docker.com] failed with 404,
retrying" full_repo=quay.io/coreos/kube-state-metrics

It appears that line 342 of mirror.go just hardcodes registry.hub.docker.com in the url to mirror from.

Would it be possible to check if the start of the location was a domain and mirror from there instead of docker.com?