Would it be possible to add buildx build "--load switch" support to the GitHub Action?

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Feature Request - Buildx build "--load switch" support about docker_buildx HOT 7 CLOSED

ilteoood commented on August 26, 2024

Feature Request - Buildx build "--load switch" support

from docker_buildx.

Comments (7)

ilteoood commented on August 26, 2024

Hi @rhastie,
as I can see from buildx documentation, the --load parameter is a shorthand for --output=type=docker

But that output type isn't valid for multi-arch images

So, I think that isn't possibile.
Btw, I would like to understand your needs: what you want to do, caching layers from x86 build, isn't possible. Imagine the cache layers as a stack where at the bottom you have the base image. When you modify a layer, everything that is above it has to rebuild.

When you do a x86 build, you use a base image that is compatible with x86 architecture, but when you do an ARM build, you use a base image that is compatible with ARM architecture: so, with different base images, you can't use any caching method because you have always to rebuild everything.

from docker_buildx.

rhastie commented on August 26, 2024

Thanks for the prompt response...

I'm trying to build for 3 platforms - ARMv7, ARM64 and x86 but I also want to include some CI testing... Basically I want to follow this pattern:

Run buildx build for x86 platform only and load to the docker image library using "--load"
Start x86 image on the linux host
Complete CI tests using just x86 image
If it fails testing, terminate
If its passing testing re-run Buildx build for x86, ARM64, ARMv7 platforms using previous buildx cache (this should still be on the host). This should mean the x86 image is re-built almost immediately.
Complete other platforms buildx build, combine into single buildx manifest and push to Docker Hub

As the GitHub Actions Linux host is x86 the idea is to run the CI tests using the x86 image and if it fails to fail fast. If the tests pass, then I can finish the buildx builds for the other platforms and combine with my previous x86 build from the cache.

I hope this clears things up a bit...Buildx should maintain a cache (that's separate to Docker) between runs - At least that's my experience of it.

Thanks

from docker_buildx.

ilteoood commented on August 26, 2024

All the images that you build, are saved in the local docker registry.
What you can do is:

build the x86 only image
run your tests
build the arm only image
create the dedicated manifest
push all

from docker_buildx.

rhastie commented on August 26, 2024

Matteo,

Do you mean do this outside of buildx using the normal docker processes?

When I run this using buildx without the --load option I get the following output and the image is built but not loaded into docker cache as an image - Please see the WARN at the top of the output

rhastie@ubuntu:~/nmos/build-nmos-cpp$ docker buildx build --platform linux/amd64 -t rhastie/nmos-cpp:test --build-arg makemt=4 .
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load 
[+] Building 21.6s (25/25) FINISHED                                                                                                                                             
 => [internal] load .dockerignore                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                            0.0s
 => [internal] load build definition from Dockerfile                                                                                                                       0.0s
 => => transferring dockerfile: 32B                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/ubuntu:bionic                                                                                                          21.6s
 => [internal] load build context                                                                                                                                          0.0s
 => => transferring context: 432B                                                                                                                                          0.0s
 => [stage1-build  1/16] FROM docker.io/library/ubuntu:bionic@sha256:3235326357dfb65f1781dbc4df3b834546d8bf914e82cce58e6e6b676e23ce8f                                      0.0s
 => => resolve docker.io/library/ubuntu:bionic@sha256:3235326357dfb65f1781dbc4df3b834546d8bf914e82cce58e6e6b676e23ce8f                                                     0.0s
 => CACHED [stage1-build  2/16] RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install -y --no-install-recommends     g++ build-essential     ope  0.0s
 => CACHED [stage1-build  3/16] RUN cd /home/ && wget --no-check-certificate https://cmake.org/files/v3.17/cmake-3.17.2.tar.gz &&     tar xvf cmake-3.17.2.tar.gz && rm c  0.0s
 => CACHED [stage1-build  4/16] RUN cd /home/ && git config --global http.sslVerify false &&     git clone --branch release/1.24 https://github.com/conan-io/conan.git &&  0.0s
 => CACHED [stage1-build  5/16] RUN cd /home && mkdir certs && git config --global http.sslVerify false &&     git clone https://github.com/AMWA-TV/nmos-testing.git &&    0.0s
 => CACHED [stage1-build  6/16] RUN cd /home/ && curl --output - -s -k https://codeload.github.com/sony/nmos-cpp/tar.gz/6434665c3414d6c785d3300e8b343ba7e03fade4 | tar zx  0.0s
 => CACHED [stage1-build  7/16] RUN cd /home/ && wget --no-check-certificate https://opensource.apple.com/tarballs/mDNSResponder/mDNSResponder-878.260.1.tar.gz &&     ta  0.0s
 => CACHED [stage1-build  8/16] RUN mkdir /home/nmos-cpp/Development/build &&     cd /home/nmos-cpp/Development/build &&     cmake     -G "Unix Makefiles"     -DCMAKE_BU  0.0s
 => CACHED [stage1-build  9/16] RUN cd /home/certs && mkdir run-certs && ./generateCerts registration1 nmos.tv query1.nmos.tv &&     cd /home/certs/certs &&     cp ca.ce  0.0s
 => CACHED [stage1-build 10/16] RUN cd /home/ && mkdir example-conf && mkdir admin                                                                                         0.0s
 => CACHED [stage1-build 11/16] ADD example-conf /home/example-conf                                                                                                        0.0s
 => CACHED [stage1-build 12/16] RUN cd /home/ && git config --global http.sslVerify false &&     git clone https://github.com/sony/nmos-js.git                             0.0s
 => CACHED [stage1-build 13/16] COPY mellanox-logo-horizontal-blue.png nmos-js.patch /home/nmos-js/Development/src/assets/                                                 0.0s
 => CACHED [stage1-build 14/16] RUN cd /home &&     mv /home/nmos-js/Development/src/assets/nmos-js.patch /home &&     patch -p0 <nmos-js.patch &&     rm /home/nmos-js/D  0.0s
 => CACHED [stage1-build 15/16] RUN cd /home/nmos-js/Development &&     yarn install --network-timeout 1000000 &&     yarn build &&     cp -rf /home/nmos-js/Development/  0.0s
 => CACHED [stage1-build 16/16] RUN cd /home/nmos-cpp/Development/build &&     cp nmos-cpp-node nmos-cpp-registry /home &&     cd /home && rm -rf .git conan cmake-3.17.2  0.0s
 => CACHED [stage-1 2/6] COPY --from=stage1-build /home /home                                                                                                              0.0s
 => CACHED [stage-1 3/6] RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install -y --no-install-recommends     openssl make     nano curl jq &&    0.0s
 => CACHED [stage-1 4/6] COPY entrypoint.sh container-config registry-json node-json /home/                                                                                0.0s
 => CACHED [stage-1 5/6] RUN chmod +x /home/entrypoint.sh                                                                                                                  0.0s
 => CACHED [stage-1 6/6] WORKDIR /home/                                                                                                                                    0.0s
rhastie@ubuntu:~/nmos/build-nmos-cpp$ docker images
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
rhastie/webmin-dhcp-dns      easynmos            7a2b56ca6a66        7 days ago          349MB
rhastie/webmin-dhcp-dns      latest              cbdab7c8fb48        7 days ago          348MB
amwa/nmos-testing            latest              0fd83ae803dd        12 days ago         361MB
multiarch/qemu-user-static   latest              e623c77ba49d        2 weeks ago         130MB
moby/buildkit                buildx-stable-1     f2a88cb62c92        3 weeks ago         82.8MB

When I add the --load option it correctly imports the buildx output into the docker cache as an image that can be used.

[+] Building 24.9s (27/27) FINISHED                                                                                                                                             
 => [internal] load .dockerignore                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                            0.0s
 => [internal] load build definition from Dockerfile                                                                                                                       0.0s
 => => transferring dockerfile: 32B                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/ubuntu:bionic                                                                                                          21.6s
 => [internal] load build context                                                                                                                                          0.0s
 => => transferring context: 432B                                                                                                                                          0.0s
 => [stage1-build  1/16] FROM docker.io/library/ubuntu:bionic@sha256:3235326357dfb65f1781dbc4df3b834546d8bf914e82cce58e6e6b676e23ce8f                                      0.0s
 => => resolve docker.io/library/ubuntu:bionic@sha256:3235326357dfb65f1781dbc4df3b834546d8bf914e82cce58e6e6b676e23ce8f                                                     0.0s
 => CACHED [stage1-build  2/16] RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install -y --no-install-recommends     g++ build-essential     ope  0.0s
 => CACHED [stage1-build  3/16] RUN cd /home/ && wget --no-check-certificate https://cmake.org/files/v3.17/cmake-3.17.2.tar.gz &&     tar xvf cmake-3.17.2.tar.gz && rm c  0.0s
 => CACHED [stage1-build  4/16] RUN cd /home/ && git config --global http.sslVerify false &&     git clone --branch release/1.24 https://github.com/conan-io/conan.git &&  0.0s
 => CACHED [stage1-build  5/16] RUN cd /home && mkdir certs && git config --global http.sslVerify false &&     git clone https://github.com/AMWA-TV/nmos-testing.git &&    0.0s
 => CACHED [stage1-build  6/16] RUN cd /home/ && curl --output - -s -k https://codeload.github.com/sony/nmos-cpp/tar.gz/6434665c3414d6c785d3300e8b343ba7e03fade4 | tar zx  0.0s
 => CACHED [stage1-build  7/16] RUN cd /home/ && wget --no-check-certificate https://opensource.apple.com/tarballs/mDNSResponder/mDNSResponder-878.260.1.tar.gz &&     ta  0.0s
 => CACHED [stage1-build  8/16] RUN mkdir /home/nmos-cpp/Development/build &&     cd /home/nmos-cpp/Development/build &&     cmake     -G "Unix Makefiles"     -DCMAKE_BU  0.0s
 => CACHED [stage1-build  9/16] RUN cd /home/certs && mkdir run-certs && ./generateCerts registration1 nmos.tv query1.nmos.tv &&     cd /home/certs/certs &&     cp ca.ce  0.0s
 => CACHED [stage1-build 10/16] RUN cd /home/ && mkdir example-conf && mkdir admin                                                                                         0.0s
 => CACHED [stage1-build 11/16] ADD example-conf /home/example-conf                                                                                                        0.0s
 => CACHED [stage1-build 12/16] RUN cd /home/ && git config --global http.sslVerify false &&     git clone https://github.com/sony/nmos-js.git                             0.0s
 => CACHED [stage1-build 13/16] COPY mellanox-logo-horizontal-blue.png nmos-js.patch /home/nmos-js/Development/src/assets/                                                 0.0s
 => CACHED [stage1-build 14/16] RUN cd /home &&     mv /home/nmos-js/Development/src/assets/nmos-js.patch /home &&     patch -p0 <nmos-js.patch &&     rm /home/nmos-js/D  0.0s
 => CACHED [stage1-build 15/16] RUN cd /home/nmos-js/Development &&     yarn install --network-timeout 1000000 &&     yarn build &&     cp -rf /home/nmos-js/Development/  0.0s
 => CACHED [stage1-build 16/16] RUN cd /home/nmos-cpp/Development/build &&     cp nmos-cpp-node nmos-cpp-registry /home &&     cd /home && rm -rf .git conan cmake-3.17.2  0.0s
 => CACHED [stage-1 2/6] COPY --from=stage1-build /home /home                                                                                                              0.0s
 => CACHED [stage-1 3/6] RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install -y --no-install-recommends     openssl make     nano curl jq &&    0.0s
 => CACHED [stage-1 4/6] COPY entrypoint.sh container-config registry-json node-json /home/                                                                                0.0s
 => CACHED [stage-1 5/6] RUN chmod +x /home/entrypoint.sh                                                                                                                  0.0s
 => CACHED [stage-1 6/6] WORKDIR /home/                                                                                                                                    0.0s
 => exporting to oci image format                                                                                                                                          1.3s
 => => exporting layers                                                                                                                                                    0.0s
 => => exporting manifest sha256:8d3f5091b3fc1f13e21be28b5cf29e3969b8b74fe0fad286d57b09eb9de62295                                                                          0.0s
 => => exporting config sha256:c664a0e68c436003b2aa764f2415b13035e544d4ca71ff55299592d227070794                                                                            0.0s
 => => sending tarball                                                                                                                                                     1.2s
 => importing to docker                                                                                                                                                    2.1s
rhastie@ubuntu:~/nmos/build-nmos-cpp$ docker images
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
rhastie/nmos-cpp             test                c664a0e68c43        3 days ago          125MB
rhastie/webmin-dhcp-dns      easynmos            7a2b56ca6a66        7 days ago          349MB
rhastie/webmin-dhcp-dns      latest              cbdab7c8fb48        7 days ago          348MB
amwa/nmos-testing            latest              0fd83ae803dd        12 days ago         361MB
multiarch/qemu-user-static   latest              e623c77ba49d        2 weeks ago         130MB
moby/buildkit                buildx-stable-1     f2a88cb62c92        3 weeks ago         82.8MB

from docker_buildx.

ilteoood commented on August 26, 2024

I don't know how you configured your buildx builder, but with my action that warning doesn't come up, as you can see here:
https://github.com/ilteoood/docker-surfshark/runs/662237886?check_suite_focus=true

from docker_buildx.

rhastie commented on August 26, 2024

This is what I would expect...

You appear to be publishing the images and manifest directly to Docker hub so you are adding the "--push" to the Docker buildx build command in your Action ... ie. input Publish=TRUE.

The FR was to allow something slightly different and publish to the Docker images on the local machine. ie. use the "--load" switch. This means I can then start the container directly and run the CI tests.

I went through your Action and I don't think this is accomodated for... If you run input publish=FALSE (ie. the default) do you get the same warning?

WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

from docker_buildx.

ilteoood commented on August 26, 2024

Inserted in 1.0.3

from docker_buildx.

Feature Request - Buildx build "--load switch" support about docker_buildx HOT 7 CLOSED

Comments (7)

Related Issues (9)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent