This project implements the well known multi GPU Jacobi solver with different multi GPU Programming Models:
single_threaded_copySingle Threaded using cudaMemcpy for inter GPU communicationmulti_threaded_copyMulti Threaded with OpenMP using cudaMemcpy for inter GPU communicationmulti_threaded_copy_overlapMulti Threaded with OpenMP using cudaMemcpy for itner GPU communication with overlapping communicationmulti_threaded_p2pMulti Threaded with OpenMP using GPUDirect P2P mappings for inter GPU communicationmulti_threaded_p2p_optMulti Threaded with OpenMP using GPUDirect P2P mappings for inter GPU communication with delayed norm executionmulti_threaded_umMulti Threaded with OpenMP relying on transparent peer mappings with Unified Memory for inter GPU communicationmpiMulti Process with MPI using CUDA-aware MPI for inter GPU communicationmpi_overlapMulti Process with MPI using CUDA-aware MPI for inter GPU communication with overlapping communicationncclMulti Process with MPI and NCCL using NCCL for inter GPU communicationnccl_overlapMulti Process with MPI and NCCL using NCCL for inter GPU communication with overlapping communicationnccl_graphsMulti Process with MPI and NCCL using NCCL for inter GPU communication with overlapping communication combined with CUDA GraphsnvshmemMulti Process with MPI and NVSHMEM using NVSHMEM for inter GPU communication.
Each variant is a stand alone Makefile project and most variants have been discussed in various GTC Talks, e.g.:
single_threaded_copy,multi_threaded_copy,multi_threaded_copy_overlap,multi_threaded_p2p,multi_threaded_p2p_opt,mpi,mpi_overlapandnvshmemon DGX-1V at GTC Europe 2017 in 23031 - Multi GPU Programming Modelssingle_threaded_copy,multi_threaded_copy,multi_threaded_copy_overlap,multi_threaded_p2p,multi_threaded_p2p_opt,mpi,mpi_overlapandnvshmemon DGX-2 at GTC 2019 in S9139 - Multi GPU Programming Modelsmulti_threaded_copy,multi_threaded_copy_overlap,multi_threaded_p2p,multi_threaded_p2p_opt,mpi,mpi_overlap,nccl,nccl_overlapandnvshmemon DGX A100 at GTC 2021 in A31140 - Multi-GPU Programming Models
Some examples in this repository are the basis for an interactive tutorial: FZJ-JSC/tutorial-multi-gpu.
- CUDA: version 11.0 (9.2 if build with
DISABLE_CUB=1) or later is required by all variants.nccl_graphsrequires NCCL 2.15.1, CUDA 11.7 and CUDA Driver 515.65.01 or newer
- OpenMP capable compiler: Required by the Multi Threaded variants. The examples have been developed and tested with gcc.
- MPI: The
mpiandmpi_overlapvariants require a CUDA-aware1 implementation. For NVSHMEM and NCCL, a non CUDA-aware MPI is sufficient. The examples have been developed and tested with OpenMPI. - NVSHMEM (version 0.4.1 or later): Required by the NVSHMEM variant.
- NCCL (version 2.8 or later): Required by the NCCL variant
Each variant comes with a Makefile and can be built by simply issuing make, e.g.
multi-gpu-programming-models$ cd multi_threaded_copy
multi_threaded_copy$ make
nvcc -DHAVE_CUB -Xcompiler -fopenmp -lineinfo -DUSE_NVTX -lnvToolsExt -gencode arch=compute_70,code=sm_70 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_90,code=compute_90 -std=c++14 jacobi.cu -o jacobi
multi_threaded_copy$ ls jacobi
jacobiAll variant have the following command line options
-niter: How many iterations to carry out (default 1000)-nccheck: How often to check for convergence (default 1)-nx: Size of the domain in x direction (default 16384)-ny: Size of the domain in y direction (default 16384)-csv: Print performance results as -csv-use_hp_streams: Inmpi_overlapuse high priority streams to hide kernel launch latencies of boundary kernels.
The nvshmem variant additionally provides
-use_block_comm: Use block cooperativenvshmemx_float_put_nbi_blockinstead ofnvshmem_float_pfor communication.-norm_overlap: Enable delayed norm execution as also implemented inmulti_threaded_p2p_opt-neighborhood_sync: Use custom neighbor only sync instead ofnvshmemx_barrier_all_on_stream
The provided script bench.sh contains some examples executing all the benchmarks presented in the GTC Talks referenced above.
The code applies the style guide implemented in .clang-format file. clang-format version 7 or later should be used to format the code prior to submitting it. E.g. with
multi-gpu-programming-models$ cd multi_threaded_copy
multi_threaded_copy$ clang-format -style=file -i jacobi.cuFootnotes
-
A check for CUDA-aware support is done at compile and run time (see the OpenMPI FAQ for details). If your CUDA-aware MPI implementation does not support this check, which requires
MPIX_CUDA_AWARE_SUPPORTandMPIX_Query_cuda_support()to be defined inmpi-ext.h, it can be skipped by settingSKIP_CUDA_AWARENESS_CHECK=1. ↩
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent