Comments (7)
Hi @rhastie,
as I can see from buildx documentation, the --load parameter is a shorthand for --output=type=docker
But that output type isn't valid for multi-arch images
So, I think that isn't possibile.
Btw, I would like to understand your needs: what you want to do, caching layers from x86 build, isn't possible. Imagine the cache layers as a stack where at the bottom you have the base image. When you modify a layer, everything that is above it has to rebuild.
When you do a x86 build, you use a base image that is compatible with x86 architecture, but when you do an ARM build, you use a base image that is compatible with ARM architecture: so, with different base images, you can't use any caching method because you have always to rebuild everything.
from docker_buildx.
Thanks for the prompt response...
I'm trying to build for 3 platforms - ARMv7, ARM64 and x86 but I also want to include some CI testing... Basically I want to follow this pattern:
- Run buildx build for x86 platform only and load to the docker image library using "--load"
- Start x86 image on the linux host
- Complete CI tests using just x86 image
- If it fails testing, terminate
- If its passing testing re-run Buildx build for x86, ARM64, ARMv7 platforms using previous buildx cache (this should still be on the host). This should mean the x86 image is re-built almost immediately.
- Complete other platforms buildx build, combine into single buildx manifest and push to Docker Hub
As the GitHub Actions Linux host is x86 the idea is to run the CI tests using the x86 image and if it fails to fail fast. If the tests pass, then I can finish the buildx builds for the other platforms and combine with my previous x86 build from the cache.
I hope this clears things up a bit...Buildx should maintain a cache (that's separate to Docker) between runs - At least that's my experience of it.
Thanks
from docker_buildx.
All the images that you build, are saved in the local docker registry.
What you can do is:
- build the x86 only image
- run your tests
- build the arm only image
- create the dedicated manifest
- push all
from docker_buildx.
Matteo,
Do you mean do this outside of buildx using the normal docker processes?
When I run this using buildx without the --load
option I get the following output and the image is built but not loaded into docker cache as an image - Please see the WARN at the top of the output
rhastie@ubuntu:~/nmos/build-nmos-cpp$ docker buildx build --platform linux/amd64 -t rhastie/nmos-cpp:test --build-arg makemt=4 .
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 21.6s (25/25) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 32B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:bionic 21.6s
=> [internal] load build context 0.0s
=> => transferring context: 432B 0.0s
=> [stage1-build 1/16] FROM docker.io/library/ubuntu:bionic@sha256:3235326357dfb65f1781dbc4df3b834546d8bf914e82cce58e6e6b676e23ce8f 0.0s
=> => resolve docker.io/library/ubuntu:bionic@sha256:3235326357dfb65f1781dbc4df3b834546d8bf914e82cce58e6e6b676e23ce8f 0.0s
=> CACHED [stage1-build 2/16] RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install -y --no-install-recommends g++ build-essential ope 0.0s
=> CACHED [stage1-build 3/16] RUN cd /home/ && wget --no-check-certificate https://cmake.org/files/v3.17/cmake-3.17.2.tar.gz && tar xvf cmake-3.17.2.tar.gz && rm c 0.0s
=> CACHED [stage1-build 4/16] RUN cd /home/ && git config --global http.sslVerify false && git clone --branch release/1.24 https://github.com/conan-io/conan.git && 0.0s
=> CACHED [stage1-build 5/16] RUN cd /home && mkdir certs && git config --global http.sslVerify false && git clone https://github.com/AMWA-TV/nmos-testing.git && 0.0s
=> CACHED [stage1-build 6/16] RUN cd /home/ && curl --output - -s -k https://codeload.github.com/sony/nmos-cpp/tar.gz/6434665c3414d6c785d3300e8b343ba7e03fade4 | tar zx 0.0s
=> CACHED [stage1-build 7/16] RUN cd /home/ && wget --no-check-certificate https://opensource.apple.com/tarballs/mDNSResponder/mDNSResponder-878.260.1.tar.gz && ta 0.0s
=> CACHED [stage1-build 8/16] RUN mkdir /home/nmos-cpp/Development/build && cd /home/nmos-cpp/Development/build && cmake -G "Unix Makefiles" -DCMAKE_BU 0.0s
=> CACHED [stage1-build 9/16] RUN cd /home/certs && mkdir run-certs && ./generateCerts registration1 nmos.tv query1.nmos.tv && cd /home/certs/certs && cp ca.ce 0.0s
=> CACHED [stage1-build 10/16] RUN cd /home/ && mkdir example-conf && mkdir admin 0.0s
=> CACHED [stage1-build 11/16] ADD example-conf /home/example-conf 0.0s
=> CACHED [stage1-build 12/16] RUN cd /home/ && git config --global http.sslVerify false && git clone https://github.com/sony/nmos-js.git 0.0s
=> CACHED [stage1-build 13/16] COPY mellanox-logo-horizontal-blue.png nmos-js.patch /home/nmos-js/Development/src/assets/ 0.0s
=> CACHED [stage1-build 14/16] RUN cd /home && mv /home/nmos-js/Development/src/assets/nmos-js.patch /home && patch -p0 <nmos-js.patch && rm /home/nmos-js/D 0.0s
=> CACHED [stage1-build 15/16] RUN cd /home/nmos-js/Development && yarn install --network-timeout 1000000 && yarn build && cp -rf /home/nmos-js/Development/ 0.0s
=> CACHED [stage1-build 16/16] RUN cd /home/nmos-cpp/Development/build && cp nmos-cpp-node nmos-cpp-registry /home && cd /home && rm -rf .git conan cmake-3.17.2 0.0s
=> CACHED [stage-1 2/6] COPY --from=stage1-build /home /home 0.0s
=> CACHED [stage-1 3/6] RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install -y --no-install-recommends openssl make nano curl jq && 0.0s
=> CACHED [stage-1 4/6] COPY entrypoint.sh container-config registry-json node-json /home/ 0.0s
=> CACHED [stage-1 5/6] RUN chmod +x /home/entrypoint.sh 0.0s
=> CACHED [stage-1 6/6] WORKDIR /home/ 0.0s
rhastie@ubuntu:~/nmos/build-nmos-cpp$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
rhastie/webmin-dhcp-dns easynmos 7a2b56ca6a66 7 days ago 349MB
rhastie/webmin-dhcp-dns latest cbdab7c8fb48 7 days ago 348MB
amwa/nmos-testing latest 0fd83ae803dd 12 days ago 361MB
multiarch/qemu-user-static latest e623c77ba49d 2 weeks ago 130MB
moby/buildkit buildx-stable-1 f2a88cb62c92 3 weeks ago 82.8MB
When I add the --load
option it correctly imports the buildx output into the docker cache as an image that can be used.
[+] Building 24.9s (27/27) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 32B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:bionic 21.6s
=> [internal] load build context 0.0s
=> => transferring context: 432B 0.0s
=> [stage1-build 1/16] FROM docker.io/library/ubuntu:bionic@sha256:3235326357dfb65f1781dbc4df3b834546d8bf914e82cce58e6e6b676e23ce8f 0.0s
=> => resolve docker.io/library/ubuntu:bionic@sha256:3235326357dfb65f1781dbc4df3b834546d8bf914e82cce58e6e6b676e23ce8f 0.0s
=> CACHED [stage1-build 2/16] RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install -y --no-install-recommends g++ build-essential ope 0.0s
=> CACHED [stage1-build 3/16] RUN cd /home/ && wget --no-check-certificate https://cmake.org/files/v3.17/cmake-3.17.2.tar.gz && tar xvf cmake-3.17.2.tar.gz && rm c 0.0s
=> CACHED [stage1-build 4/16] RUN cd /home/ && git config --global http.sslVerify false && git clone --branch release/1.24 https://github.com/conan-io/conan.git && 0.0s
=> CACHED [stage1-build 5/16] RUN cd /home && mkdir certs && git config --global http.sslVerify false && git clone https://github.com/AMWA-TV/nmos-testing.git && 0.0s
=> CACHED [stage1-build 6/16] RUN cd /home/ && curl --output - -s -k https://codeload.github.com/sony/nmos-cpp/tar.gz/6434665c3414d6c785d3300e8b343ba7e03fade4 | tar zx 0.0s
=> CACHED [stage1-build 7/16] RUN cd /home/ && wget --no-check-certificate https://opensource.apple.com/tarballs/mDNSResponder/mDNSResponder-878.260.1.tar.gz && ta 0.0s
=> CACHED [stage1-build 8/16] RUN mkdir /home/nmos-cpp/Development/build && cd /home/nmos-cpp/Development/build && cmake -G "Unix Makefiles" -DCMAKE_BU 0.0s
=> CACHED [stage1-build 9/16] RUN cd /home/certs && mkdir run-certs && ./generateCerts registration1 nmos.tv query1.nmos.tv && cd /home/certs/certs && cp ca.ce 0.0s
=> CACHED [stage1-build 10/16] RUN cd /home/ && mkdir example-conf && mkdir admin 0.0s
=> CACHED [stage1-build 11/16] ADD example-conf /home/example-conf 0.0s
=> CACHED [stage1-build 12/16] RUN cd /home/ && git config --global http.sslVerify false && git clone https://github.com/sony/nmos-js.git 0.0s
=> CACHED [stage1-build 13/16] COPY mellanox-logo-horizontal-blue.png nmos-js.patch /home/nmos-js/Development/src/assets/ 0.0s
=> CACHED [stage1-build 14/16] RUN cd /home && mv /home/nmos-js/Development/src/assets/nmos-js.patch /home && patch -p0 <nmos-js.patch && rm /home/nmos-js/D 0.0s
=> CACHED [stage1-build 15/16] RUN cd /home/nmos-js/Development && yarn install --network-timeout 1000000 && yarn build && cp -rf /home/nmos-js/Development/ 0.0s
=> CACHED [stage1-build 16/16] RUN cd /home/nmos-cpp/Development/build && cp nmos-cpp-node nmos-cpp-registry /home && cd /home && rm -rf .git conan cmake-3.17.2 0.0s
=> CACHED [stage-1 2/6] COPY --from=stage1-build /home /home 0.0s
=> CACHED [stage-1 3/6] RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install -y --no-install-recommends openssl make nano curl jq && 0.0s
=> CACHED [stage-1 4/6] COPY entrypoint.sh container-config registry-json node-json /home/ 0.0s
=> CACHED [stage-1 5/6] RUN chmod +x /home/entrypoint.sh 0.0s
=> CACHED [stage-1 6/6] WORKDIR /home/ 0.0s
=> exporting to oci image format 1.3s
=> => exporting layers 0.0s
=> => exporting manifest sha256:8d3f5091b3fc1f13e21be28b5cf29e3969b8b74fe0fad286d57b09eb9de62295 0.0s
=> => exporting config sha256:c664a0e68c436003b2aa764f2415b13035e544d4ca71ff55299592d227070794 0.0s
=> => sending tarball 1.2s
=> importing to docker 2.1s
rhastie@ubuntu:~/nmos/build-nmos-cpp$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
rhastie/nmos-cpp test c664a0e68c43 3 days ago 125MB
rhastie/webmin-dhcp-dns easynmos 7a2b56ca6a66 7 days ago 349MB
rhastie/webmin-dhcp-dns latest cbdab7c8fb48 7 days ago 348MB
amwa/nmos-testing latest 0fd83ae803dd 12 days ago 361MB
multiarch/qemu-user-static latest e623c77ba49d 2 weeks ago 130MB
moby/buildkit buildx-stable-1 f2a88cb62c92 3 weeks ago 82.8MB
from docker_buildx.
I don't know how you configured your buildx builder, but with my action that warning doesn't come up, as you can see here:
https://github.com/ilteoood/docker-surfshark/runs/662237886?check_suite_focus=true
from docker_buildx.
This is what I would expect...
You appear to be publishing the images and manifest directly to Docker hub so you are adding the "--push" to the Docker buildx build command in your Action ... ie. input Publish=TRUE.
The FR was to allow something slightly different and publish to the Docker images on the local machine. ie. use the "--load" switch. This means I can then start the container directly and run the CI tests.
I went through your Action and I don't think this is accomodated for... If you run input publish=FALSE (ie. the default) do you get the same warning?
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
from docker_buildx.
Inserted in 1.0.3
from docker_buildx.
Related Issues (9)
- Add --target support HOT 1
- Getting warning with new "load" option HOT 5
- Dockerfile true none false HOT 6
- Could you add build context or workdir to allow building images from subdirs HOT 1
- Error: Command failed: docker_buildx/scripts/dockerhub_login.sh HOT 1
- How do you make the build manually? HOT 2
- Build Step withholds logs until it finishes HOT 2
- Error failed to solve: rpc error: code = Unknown desc = server message: insufficient_scope: authorization failed HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker_buildx.