Project Roadmap about alpa HOT 4 CLOSED

July 18 - July 24

Feature Implementation

• Pipeline (hao, zhuohan)
  • Implement nccl-based send and recv
  • Implement automatic slicing algorithm
• SPMD (lianmin)
  • Benchmark deepspeed
  • Implement Zero Redundancy Optimizer
• Memory Optimization (yonghao)
  • Integrate rematerializaiton and swapping into the GPT benchmark

Performance Goal

• Match 2d parallelism in Megatron-LM
• Match 3d parallelism in Megatron-LM

July 25 - July 31

Feature Implementation

• Pipeline (hao, zhuohan)
  • Implement nccl-based send and recv
  • Implement automatic slicing algorithm
  • Implement advanced scheduling algorithms
• SPMD (lianmin)
  • Let the auto-sharding pass generate reduce-scatter instead of all-reduce
• Memory Optimization (yonghao)
  • Integrate rematerializaiton and swapping into the GPT benchmark

Performance Goal

• Match 2d parallelism in Megatron-LM
• Match 3d parallelism in Megatron-LM
• Match ZeRO optimizer (stage 2) in deepspeed

Aug. 1 - Aug. 15

Feature Implementation

• Test on new models

  • Mixture of Expert
  • wide-resnet
  • Conformer
  • DLRM
  • TransGAN

• Auto-tuner

  • Write a simple auto-tuner to perform search

Performance Goal

• Outperform the data-parallel only on new models

Aug. 15 - Aug. 21

Feature Implementation

• Paper writing

Performance Goal

• Match strong manual baseline on homogeneous models
• Outperform all baselines on heterogeneous models
• Perfect weak scaling

from alpa.

Evaluation Plan

On 5 models, show

• 1. Our unified parallelism vs. intra-operator parallelism only vs. inter-operator parallelism only vs. data-parallel only
• 2. Our algorithm vs. manual approaches
  • Megatron-LM
  • Deepspeed
  • PyTorch data-parallel / Horovod
  • Add a memory-saving only baseline.
• 3. Our algorithm vs. other simple heuristics
• 4. Weak scaling test

from alpa.

More developments needed for paper submission and MVP

• (Zhuohan) Integrate stage discovery #67 into the current Parax;
• (Zhuohan) Integrate layer grouping DP with stage discovery #67
• (Hao, Yonghao) Finish and Integrate mesh slicing with the above;
• (Lianmin) Gradient accumulation support, see #72;
• (Hao) Make Pipeline-only parallelism performance matching that of Megatron.

from alpa.

Close this because this is outdated. Will open a new one

from alpa.