Code Monkey home page Code Monkey logo

mindthegap's Introduction

mindthegap

GitHub

mindthegap provides utilities to manage air-gapped image bundles, both creating image bundles and seeding images from a bundle into an existing OCI registry.

Usage

Image bundles

Creating an image bundle

mindthegap create image-bundle --images-file <path/to/images.yaml> \
  --platform <platform> [--platform <platform> ...] \
  [--output-file <path/to/output.tar>]

See the example images.yaml for the structure of the images config file. You can also provide the images file in a simple file with an image per line, e.g.

nginx:1.21.5
test.registry2.io/test-image6:atag

Note that images from Docker Hub must be prefixed with docker.io and those "official" images must have the library namespace specified.

Platform can be specified multiple times. Supported platforms:

linux/amd64
linux/arm64
windows/amd64
windows/arm64

All images in the images config file must support all the requested platforms.

The output file will be a tarball that can be seeded into a registry, or that can be untarred and used as the storage directory for an OCI registry served via registry:2.

Pushing an image bundle

This command is deprecated - see Pushing a bundle

mindthegap push image-bundle --image-bundle <path/to/images.tar> \
  --to-registry <registry.address> \
  [--to-registry-insecure-skip-tls-verify]

All images in the image bundle tar file will be pushed to the target OCI registry.

Serving an image bundle

This command is deprecated - see Serving a bundle

mindthegap serve image-bundle --image-bundle <path/to/images.tar> \
  [--listen-address <listen.address>] \
  [--listen-port <listen.port>]

Start an OCI registry serving the contents of the image bundle. Note that the OCI registry will be in read-only mode to reflect the source of the data being a static tarball so pushes to this registry will fail.

Importing an image bundle into containerd

mindthegap import image-bundle --image-bundle <path/to/images.tar> \
  [--containerd-namespace <containerd.namespace]

Import the images from the image bundle into containerd in the specified namespace. If --containerd-namespace is not specified, images will be imported into k8s.io namespace. This command requires ctr to be in the PATH.

Helm chart bundles

Creating a Helm chart bundle

mindthegap create helm-bundle --helm-charts-file <path/to/helm-charts.yaml> \
  [--output-file <path/to/output.tar>]

See the example helm-charts.yaml for the structure of the Helm charts config file.

The output file will be a tarball that can be seeded into a registry, or that can be untarred and used as the storage directory for an OCI registry served via registry:2.

Pushing a Helm chart bundle

This command is deprecated - see Pushing a bundle

mindthegap push helm-bundle --image-bundle <path/to/helm-charts.tar> \
  --to-registry <registry.address> \
  [--to-registry-insecure-skip-tls-verify]

All Helm charts in the bundle tar file will be pushed to the target OCI registry.

Serving a Helm chart bundle

This command is deprecated - see Serving a bundle

mindthegap serve helm-bundle --helm-bundle <path/to/helm-charts.tar> \
  [--listen-address <addr>] \
  [--list-port <port>] \
  [--tls-cert-file <path/to/cert/file> --tls-private-key-file <path/to/key/file>]

Start an OCI registry serving the contents of the image bundle. Note that the OCI registry will be in read-only mode to reflect the source of the data being a static tarball so pushes to this registry will fail.

Pushing a bundle (supports both image or Helm chart)

mindthegap push bundle --bundle <path/to/bundle.tar> \
  --to-registry <registry.address> \
  [--to-registry-insecure-skip-tls-verify]

All images in an image bundle tar file, or Helm charts in a chart bundle, will be pushed to the target OCI registry.

Serving a bundle (supports both image or Helm chart)

mindthegap serve bundle --bundle <path/to/bundle.tar> \
  [--listen-address <listen.address>] \
  [--listen-port <listen.port>]

Start an OCI registry serving the contents of the image bundle or Helm charts bundle. Note that the OCI registry will be in read-only mode to reflect the source of the data being a static tarball so pushes to this registry will fail.

How does it work?

mindthegap starts up an OCI registry and then uses crane as a library to copy the specified images for all specified platforms into the running registry. The resulting registry storage is then tarred up, resulting in a tarball of the specified images.

The resulting tarball can be loaded into a running OCI registry, or be used as the initial storage for running your own registry via Docker or in a Kubernetes cluster.

Contributing

This project uses https://www.jetpack.io/devbox/ to create a reproducible build environment. If you do not have devbox configured, then the following instructions should work for you. For further details, see https://www.jetpack.io/devbox/docs/installing_devbox/.

Integrate with direnv for automatic shell integration

Install direnv: https://direnv.net/docs/installation.html#from-system-packages.

Hook direnv into your shell if you haven't already: https://direnv.net/docs/hook.html.

Building the CLI

mindthegap uses task for running build tasks. task will be automatically available when the devbox environment is correctly set up.

Build the CLI using task build:snapshot that will output binary into ./dist/mindthegap_$(GOOS)_$(GOARCH)/mindthegap.

mindthegap's People

Contributors

dependabot[bot] avatar dkoshkin avatar floridoo avatar github-actions[bot] avatar jimmidyson avatar mesosphere-actions-pr-bot[bot] avatar mhrabovcin avatar tillt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mindthegap's Issues

import command panics if invalid bundle is provided

#6 1.326 2023-09-21 13:54:26 INF  • Creating temporary directory...
#6 1.326  ✓ Creating temporary directory
#6 1.326 2023-09-21 13:54:26 INF  • Unarchiving image bundle "/tmp/mindthegap.tar"...
#6 1.343  ✓ Unarchiving image bundle "/tmp/mindthegap.tar"
#6 1.343 2023-09-21 13:54:26 INF  • Starting temporary Docker registry...
#6 1.348  ✓ Starting temporary Docker registry
#6 1.354 panic: runtime error: invalid memory address or nil pointer dereference
#6 1.354 [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1e4f244]
#6 1.354
#6 1.354 goroutine 1 [running]:
#6 1.354 github.com/mesosphere/mindthegap/cmd/mindthegap/importcmd/imagebundle.NewCommand.func1(0xc000f59200?, {0xc000642d40?, 0x2?, 0x2?})
#6 1.354        github.com/mesosphere/mindthegap/cmd/mindthegap/importcmd/imagebundle/image_bundle.go:88 +0x6e4
#6 1.354 github.com/spf13/cobra.(*Command).execute(0xc000f59200, {0xc000642d20, 0x2, 0x2})
#6 1.354        github.com/spf13/[email protected]/command.go:940 +0x862
#6 1.354 github.com/spf13/cobra.(*Command).ExecuteC(0xc000005500)
#6 1.354        github.com/spf13/[email protected]/command.go:1068 +0x3bd
#6 1.354 github.com/spf13/cobra.(*Command).Execute(...)
#6 1.354        github.com/spf13/[email protected]/command.go:992
#6 1.354 github.com/mesosphere/mindthegap/cmd/mindthegap/root.Execute()
#6 1.354        github.com/mesosphere/mindthegap/cmd/mindthegap/root/root.go:56 +0x55

https://github.com/mesosphere/mindthegap/blob/main/cmd/mindthegap/importcmd/imagebundle/image_bundle.go#L88

Building on Darwin does not work

Most likely Linux version is always getting included.

➜  mindthegap git:(70d15c8) ✗ ./dist/mindthegap_darwin_amd64/mindthegap create image-bundle --images-file images-test.yaml
 ✓ Parsing image bundle config
 ✓ Creating temporary directory
 ✓ Starting temporary Docker registry
I1229 14:43:08.963382   83641 skopeo.go:214]
Error: error logging in to registry: failed to check if already logged in to docker.io: fork/exec /var/folders/cf/02pp6nwj3sg15xdml78ytqfc0000gq/T/skopeo-2734574594/skopeo: exec format error

Provide a URL for `--images-file`

It would be awesome if I could run mindthegap like this:

mindthegap create image-bundle --images-file https://raw.githubusercontent.com/mesosphere/mindthegap/main/images-example.yaml

That way I wouldn't have to download the file first.

Might be a good use case for go-getter.

Failed to pull some images Add the way to make a bundle images from rendered helm chart

Hi Thanks for this awsome tool ! I want to make bundles of image contained in a gitlab helm chart for now i use

helm plugin install https://github.com/nikhilsbhat/helm-images
helm images get gitlab gitlab/gitlab --set global.hosts.domain=example.com \ 
                     --set global.hosts.externalIP=10.10.10.10 \
                     --set [email protected] 

And it give me this

quay.io/jetstack/cert-manager-cainjector:v1.5.4
quay.io/jetstack/cert-manager-controller:v1.5.4
quay.io/jetstack/cert-manager-webhook:v1.5.4
registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v15.10.1
registry.gitlab.com/gitlab-org/build/cng/gitlab-exporter:12.1.1
registry.gitlab.com/gitlab-org/build/cng/certificates:v15.11.0
registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest
registry.gitlab.com/gitlab-org/build/cng/gitlab-shell:v14.18.0
registry.gitlab.com/gitlab-org/build/cng/certificates:v15.11.0
registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest
registry.gitlab.com/gitlab-org/build/cng/gitlab-kas:v15.11.0
registry.gitlab.com/gitlab-org/build/cng/certificates:v15.11.0
registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ee:v15.11.0
registry.gitlab.com/gitlab-org/build/cng/certificates:v15.11.0
registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest
registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ee:v15.11.0
registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ee:v15.11.0
registry.gitlab.com/gitlab-org/build/cng/certificates:v15.11.0
registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest
registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee:v15.11.0
registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ee:v15.11.0
registry.gitlab.com/gitlab-org/build/cng/certificates:v15.11.0
registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest
registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee:v15.11.0
minio/minio:RELEASE.2017-12-28T01-21-00Z
registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest
registry.gitlab.com/gitlab-org/cloud-native/mirror/images/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8
jimmidyson/configmap-reload:v0.5.0
quay.io/prometheus/prometheus:v2.38.0
registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.71.0-gitlab
registry.gitlab.com/gitlab-org/build/cng/certificates:v15.11.0
registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest
registry.gitlab.com/gitlab-org/build/cng/gitaly:v15.11.0
registry.gitlab.com/gitlab-org/build/cng/certificates:v15.11.0
registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest
docker.io/bitnami/postgresql:12.7.0
docker.io/bitnami/postgres-exporter:0.8.0-debian-10-r99
docker.io/bitnami/redis:6.0.9-debian-10-r0
docker.io/bitnami/redis-exporter:1.12.1-debian-10-r11
registry.gitlab.com/gitlab-org/build/cng/kubectl:v15.11.0
registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ee:v15.11.0
registry.gitlab.com/gitlab-org/build/cng/certificates:v15.11.0
registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest
minio/mc:RELEASE.2018-07-13T00-53-22Z
registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee:v15.11.0
quay.io/jetstack/cert-manager-ctl:v1.5.4
registry.gitlab.com/gitlab-org/build/cng/kubectl:v15.11.0
registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest

and then i load in

mindthegap create image-bundle --images-file gitlabimages.yaml --output-file gitlabimagesbundle.tar

but it failed with this error

failed to read image descriptor for "docker.io/bitnami/postgres-exporter:0.8.0-debian-10-r99" from registry: error getting credentials - err: exec: "docker-credential-pass": executable file not found in $PATH, out: ``

Fedora 38
podman-docker-4.5.0-1.fc38.noarch
podman-gvproxy-4.5.0-1.fc38.x86_64
podman-plugins-4.5.0-1.fc38.x86_64
podman-4.5.0-1.fc38.x86_64
podman-compose-1.0.6-1.fc38.noarch
podman-docker-4.5.0-1.fc38.noarch

Registry flag should accept protocol

The push image-bundle command's flag --to-registry should accept http or https as a prefix.

Currently the command just fails with an unclear message:

mindthegap push image-bundle --image-bundle ./bundle.tar --to-registry https://myregistry.jbeber.com:5000 --to-registry-ca-cert-file=registry/ca.crt
 ✓ Creating temporary directory
 ✓ Unarchiving image bundle "./bundle.tar" 
 ✓ Parsing image bundle config
 ✓ Starting temporary Docker registry
---skopeo stdout---:

---skopeo stderr---:
time="2022-11-01T21:09:20Z" level=fatal msg="not logged into myregistry.jbeber.com:5000"

error logging in to target registry: failed to check if already logged in to https://myregistry.jbeber.com:5000: exit status 1

with --to-registry myregistry.jbeber.com:5000 it works.

docker 25: MANIFEST_BLOB_UNKNOWN: blob unknown to registry

I'm using create image-bundle command and getting this error:

 ✓ Parsing image bundle config
 ✓ Creating temporary directory
 ✓ Starting temporary Docker registry
 ✗ Pulling requested images [                                    0/13] (time elapsed 09s) 
PUT http://127.0.0.1:33571/v2/XXX/manifests/sha256:9a4a8f7234957bc76c3a8b062070e429a1b8a23c36c24e5d6a26e9b1d91bc410: multiple errors returned:
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:064b40dd2dfb098da5b4d43876da5d481047121e1ed560129383b74d30b5fe99;
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:6626c9b5b2de04d53cccd1235e03fcb18f812284b61ce453859d282ce8d19cd0;
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:7b92e5eddc133237670f4c75659800f8bd0c3384a713318c79d0531f91b696f2;
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:5ace0b795b80b845f07ef9bb3cc42b3fcab0ec334dd92b68c7d9d710c5986ed4;
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:0ca3d0f7e3e1281d6a3d1ce1a872b6f0761d70842e6e57a02ba6806e9f5557e6;
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:b70bbbe162477aeb68d5c19a3d7d4da389c537b04d11eeabbe35b651314b8a89;
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:b5f71c00b65306805e43337cc17e101eaf23e050afe4f3a0c6647f87f48f6fe6;
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:b5f71c00b65306805e43337cc17e101eaf23e050afe4f3a0c6647f87f48f6fe6;
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:20bfa07587c5c05b57c64bce2a61ca831ae4641161f5c717ceb233579847a76b;
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:255c33047a850954ffbd0de23449674d8dce49b748e2af90afd1b8a1f1824a0a;
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:fb97d22de48e2b66fbc1a59d651f7cb9e169c9f4a2008320a78012b3fe31f7b4;
MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:ee1d465b6ef5a03fccc178453de8d3b619fbe53635fb9e7c9f8893b98c5e0af9

mindthegap version: v1.13.0
OS: ubuntu 22.04
Docker:

Client: Docker Engine - Community
 Version:           25.0.5
 API version:       1.44
 Go version:        go1.21.8
 Git commit:        5dc9bcc
 Built:             Tue Mar 19 15:05:10 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.5
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       e63daec
  Built:            Tue Mar 19 15:05:10 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Downgraded my docker to this and it worked:

Client: Docker Engine - Community
 Version:           24.0.9
 API version:       1.43
 Go version:        go1.20.13
 Git commit:        2936816
 Built:             Thu Feb  1 00:48:39 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.9
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.13
  Git commit:       fca702d
  Built:            Thu Feb  1 00:48:39 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

copyright comment skopeo/default-policy.json corrupts the built binary

When I run the mindthegap binary with the example json, I got:

{19:50}~/workspace/mindthegap:main ✗ ➭ ./mindthegap create image-bundle --images-file images-example.yaml --platform linux/amd64  --output-file example.tar                                    :main|…1
 ✓ Checking if output file already exists
 ✓ Parsing image bundle config
 ✓ Creating temporary directory
 ✓ Starting temporary Docker registry
⢄⡱ Copying k8s.gcr.io/kube-state-metrics/kube-state-metrics:v1.9.8 (platforms: [linux/amd64]) I0112 19:51:02.270676   45587 image_bundle.go:167] time="2022-01-12T19:51:02-05:00" level=fatal msg="Error loading trust policy: invalid policy in \"/var/folders/yl/yxb6ytk578l91924d5v_4grr0000gq/T/skopeo-2970473831/policy.json\": invalid character '/' looking for beginning of value"
 ✗ Copying k8s.gcr.io/kube-state-metrics/kube-state-metrics:v1.9.8 (platforms: [linux/amd64])

After removing the 2 comment lines of skopeo/default-policy.json, the command above can succeed.

If removing the comment lines are not feasible we should do some clean up before the json is fed into build process.

Error: could not find platform

[centos@ip-172-31-29-127 ~]$ cat images-test.yaml
---
docker.io:
    images:
        mesosphere/dkp-diagnostics-node-collector:
            - v0.3.3
[centos@ip-172-31-29-127 ~]$ ./mindthegap create image-bundle --images-file /home/centos/images-test.yaml
 ✓ Parsing image bundle config
 ✓ Creating temporary directory
 ✓ Starting temporary Docker registry
 ✗ Copying docker.io/mesosphere/dkp-diagnostics-node-collector:v0.3.3 (platforms: [linux/amd64])
Error: could not find platform linux/amd64 for image docker.io/mesosphere/dkp-diagnostics-node-collector:v0.3.3

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.