Comments (3)
Alternatively you could just use the existing DockerHub image with cudnn9? Or is that not valid to build/support?
I wasn't aware of existing issues when I saw the CI failure for a PR I'm involved in, but looked into it here: pytorch/pytorch#125632 (comment)
A quick fix is to just have the matrix for docker generate a versionless cudnn
portion of the tag. Presumably nvidia may be taking that approach going forward, so if the version of cudnn does not strictly need to be 8, you could relax the major version pin with the docker images? (there is no cudnn9
tag with cuda 12.4
images, only previous minor tag versions).
Otherwise, won't you need to build (or republish) all the nvidia images being used from DockerHub? The CI is failing specifically because it's trying to pull an invalid tag for nvidia/cuda
that you request:
--build-arg BASE_IMAGE=nvidia/cuda:12.4.0-cudnn8-devel-ubuntu22.04
So you need to avoid building that in the docker matrix, and separately build/publish your AWS image, or as I've suggested just add the logic to select the appropriate nvidia/cuda:12.4
image: nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04
from builder.
For cuda 12.2 with cudnn Dockerfile:
https://gitlab.com/nvidia/container-images/cuda/-/tree/master/dist/12.2.2/ubuntu2004/devel?ref_type=heads
For cuda 12.4 with no cudnn:
https://gitlab.com/nvidia/container-images/cuda/-/tree/master/dist/12.4.0/ubuntu2004/devel?ref_type=heads
from builder.
For cuda 12.4.1 with cudnn (9):
- Gitlab Dockerfile: https://gitlab.com/nvidia/container-images/cuda/-/blob/master/dist/12.4.1/ubuntu2204/devel/Dockerfile?ref_type=heads
- DockerHub image tag: https://hub.docker.com/layers/nvidia/cuda/12.4.1-cudnn-devel-ubuntu22.04/images/sha256-0a434eff1826693c1e2a669b20062f9995e73ed3456cdb70416d7ba9c1e3d1f5
from builder.
Related Issues (20)
- Update Magma to 2.8.0 for CUDA 12 & Hopper support
- Update self-hosted runners (`linux.12xlarge.ephemeral`) to newer OS
- Create POC for building wheels based on : https://github.com/pypa/cibuildwheel
- aarch64 linux: torch.compile performance is 2x slow with nightly torch wheel compared to the wheel built with 'build_aarch64_wheel.py' script HOT 18
- `numpy v2.0.0rc1` on pytorch/ conda channel problem HOT 6
- Deprecate LEGACY_DOCKER_IMAGE
- Docker Images Validate. Fix arm64 docker builds to not contain cuda versions HOT 1
- OOM error in building ARM wheel
- Triton build is failing from time to time with error on python 3.12 with Exit code 137
- Move binary Docker manywheel and conda build workflows and scripts to pytorch/pytorch repo
- Unable to build docker image from Builder Repo HOT 4
- Are cuda aarch64 build available anywhere? HOT 3
- Move release and analytics workflows scripts to test-infra
- Move magma builds from builder to test-infra
- Move all validation workflows and scripts from builder to test-infra
- [BE] More code-reuse across tests HOT 1
- CentOS Docker failures: mirrorlist.centos.org no longer online HOT 1
- Unable to build Docker image with release/2.4 branch HOT 1
- Packaging for libcufile
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from builder.