Comments (14)
uname -m
works
from docker-images.
Given your working experiment Martin, would you be interested in submitting a PR?
Sure! I was waiting for confirmation that this is what is preferred!
from docker-images.
Could this include the arch string in the miniforge download URL?
docker-images/miniforge3/Dockerfile
Line 25 in 8ae623a
https://github.com/conda-forge/miniforge/releases
-
Miniforge3-${MINIFORGE_VERSION}-Linux-x86_64.sh
-
Miniforge3-${MINIFORGE_VERSION}-Linux-aarch64.sh
-
Miniforge3-${MINIFORGE_VERSION}-Linux-ppe64le.sh
-
Miniforge3-${MINIFORGE_VERSION}-Linux-${DEB_TARGET_GNU_CPU}.sh
-
Miniforge3-${MINIFORGE_VERSION}-Linux-${TARGET_CPU_ARCH}.sh
How to make the Dockerfile pass the arch through to be templated into the miniforge release URL?
- In docker experimental mode, the
--platform
flag works- it sets
DEB_TARGET_GNU_CPU=aarch64
- it sets
- In lieu of waiting for things to emerge from experimental mode in the moby project, specifying the CPU arch as a build
ARG
orENV
variable (TARGET_CPU_ARCH=x86_64 | aarch64 | ppe64le
may be the most portable way to support multiple architectures (and any container builder) with the miniforge docker container.
...
- If multi-OS is also a goal, the ansible molecule dockerfile shows one way to execute different package commands depending upon the existence of various package managers:
https://github.com/ansible-community/molecule/blob/master/molecule/data/Dockerfile.j2#L17-L22
from docker-images.
That seems like a question to raise on the miniforge issue tracker.
from docker-images.
FWIU, there's nothing that needs to change in miniforge to make this happen.
The question is how to specify the TARGET_CPU_ARCH for builds of the miniforge Dockerfile that's in this project (conda-forge/docker-images).
docker-images/miniforge3/Dockerfile
Line 25 in 8ae623a
Multi-arch images (including miniforge images) could be created with docker build --build-arg
.
This would default to x86_64:
ARG TARGET_CPU_ARCH="x86_64"
RUN wget --no-hsts --quiet "https://github.com/conda-forge/miniforge/releases/download/${MINIFORGE_VERSION}/Miniforge3-${MINIFORGE_VERSION}-Linux-${TARGET_CPU_ARCH}.sh" -O /tmp/miniforge.sh && \
We could specify a different arch by specifying a different value for TARGET_CPU_ARCH:
docker build --build-arg TARGET_CPU_ARG="aarch64" .
- https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg
- https://vsupalov.com/docker-arg-env-variable-guide/
The new way to accomplish this (that only works with buildkit/docker AFAIU?) is with the --platform
argument; which results in there being a number of env variables set at build time:
export DOCKER_BUILDKIT=1
docker build --platform linux/amd64
docker build --platform linux/arm64
docker build --platform=linux/amd64,linux/arm64,linux/arm/v7 .
docker build --platform=darwin .
RUN wget --no-hsts --quiet "https://github.com/conda-forge/miniforge/releases/download/${MINIFORGE_VERSION}/Miniforge3-${MINIFORGE_VERSION}-Linux-${TARGETARCH}.sh" -O /tmp/miniforge.sh && \
- https://github.com/docker/buildx/#building-multi-platform-images
- https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope
- moby/moby#32487
- https://github.com/moby/buildkit/pull/499/files#diff-3e5af3861d799be5e17a086b5243b0fcR49
- http://sujayopillai.com/2019/01/02/docker-platform-args.html has a Dockerfile that
echo
s each of the platform args but not their values:
- http://sujayopillai.com/2019/01/02/docker-platform-args.html has a Dockerfile that
- https://github.com/moby/buildkit/pull/499/files#diff-3e5af3861d799be5e17a086b5243b0fcR49
It looks like TARGETARCH is not quite what we're looking for in order to get x86_64 | aarch64 | ppe64le
for the miniforge filename; so we may need a conditional on TARGETARCH or one of the other variables: moby/buildkit#499 (comment)
from docker-images.
I'm assuming that uname -m
is something like $BUILDARCH
; which isn't what we want when cross-compiling?
from docker-images.
Is there still an interest to merge linux-anvil-cos7-x86_64
, linux-anvil-aarch64
and linux-anvil-ppc64le
into a multiarch image ?
Currently this is a blocker for bioconda/bioconda-containers#63 to provide multiarch image too.
I'd be happy to help here if there is an interest (and chance to be merged)!
from docker-images.
Here is PR in my fork that builds a multi-arch image for linux-anvil-cos7
, i.e. CentOS 7 for AMD64, ARM64 and PPC64LE - https://github.com/martin-g/docker-images/pull/1/files
Here is the registry with the image: https://github.com/martin-g/docker-images/pkgs/container/condaforge%2Flinux-anvil-cos7
And a second PR that uses a matrix to build multi-arch images for CentOS 7 and AlmaLinux 8 - https://github.com/martin-g/docker-images/pull/2/files
Registry: https://github.com/martin-g?tab=packages&tab=packages&q=condaforge
from docker-images.
Thanks for your interest Martin! 🙏
Think what we would want to do is create a GHA step after the existing Docker builds that pulled the different architecture builds into multiarch images that also get pushed. IOW we could use docker manifest
to combine them
That way users can still retrieve images under the existing names, we still are able to parallelize/separate arch builds, and the end result maps nicely on to our existing images
Am curious to hear your thoughts on this approach 🙂
from docker-images.
I've never done this before but I will be glad to learn something new! Let me investigate!
from docker-images.
Worked like a charm:
PR: https://github.com/martin-g/docker-images/pull/3/files
Docker registry: https://hub.docker.com/repository/docker/mtgrigorov/linux-anvil-cos7-manifest/general
I did it in a separate workflow file because I didn't want to wait for the whole build of all images in ci.yaml, but the build job must be in the ci.yaml and should declare a dependency on the build
job.
from docker-images.
Currently this is a blocker for bioconda/bioconda-containers#63 to provide multiarch image too.
No, this is not a blocker.
But it would make downstream uses easier.
There might be interest in keep arch-specific named-images, see #242 (comment) (cc @xhochy).
The clean way to do this is to provide, e.g., linux-anvil-cos7-x86_64
, linux-anvil-cos7-aarch64
, linux-anvil-cos7-ppc64le
and collect them in a manifest linux-anvil-cos7
(meaning, we'd reuse the name for of the current x86 image for the manifest, but that shouldn't be a problem).
from docker-images.
Yep Marcel think we are proposing the same thing 🤝, which it appears Martin has done 👍
Given your working experiment Martin, would you be interested in submitting a PR? 🙂
from docker-images.
You can see an almost finished WIP at https://github.com/martin-g/docker-images/pull/4/files
At the moment it uses my accounts for DockerHub and Quay.io to fully test the jobs and deploys the images and manifests even for non-main branch, again only for testing!
Anyone is welcome to give feedback directly in the fork PR!
Once all is green I will create a PR here!
from docker-images.
Related Issues (20)
- Add linux-anvil-cos7-x86_64-cuda HOT 1
- Update conda version to 4.8.4 HOT 1
- docker image condaforge/linux-anvil-comp7 failing with yum error
- Add mambaforge image
- move to azure HOT 4
- `/usr/local/cuda/bin` is not added to `$PATH` in `linux-anvil-ppc64le-cuda` HOT 13
- Including mamba in base images HOT 13
- ppc64le image builds failing HOT 3
- Use mamba HOT 6
- Instant segfault for linux-anvil-comp7 on WSL2 HOT 5
- centos images & user+mnt namespaces HOT 2
- Move CUDA 11.5 images to CentOS 7 HOT 1
- CUDA 11.0+ PPC & ARM builds failing HOT 6
- can we move all biz to userspace? HOT 5
- Scheduled runs haven't run in a while
- Latest anvil image installs incompatible conda and mamba versions HOT 1
- Updating QEMU HOT 5
- Drop armv7l bits
- Update qemu
- RUN conda install -n python-rest-1 -c conda-forge tiktoken not working complaining that package is not available. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-images.