Intro for OpenFederatedLearning

• Federated AI Technology Enabler, Webank Fin tech , https://github.com/OpenFederatedLearning/FATE https://github.com/OpenFederatedLearning/eggroll
• FedML, USC, https://github.com/OpenFederatedLearning/FedML
• Fedlearner, Bytedance, https://github.com/OpenFederatedLearning/fedlearner
• Harmonia, AL Labs Taiwan, https://github.com/OpenFederatedLearning/harmonia
• PaddleFL, Baidu, https://github.com/OpenFederatedLearning/PaddleFL
• PySyft, OpenMined, https://github.com/OpenFederatedLearning/PySyft
• Tensorflow Federated, Google, https://github.com/OpenFederatedLearning/federated
• 9NFL, Jingdong, https://github.com/OpenFederatedLearning/9nfl

Privacy preserving technique allows sharing sensitive personal information while preserving users' privacy. Below are the technologies used in Federated Learning:

• Anonymization (weakest)
• Differential Privacy
• Fully Homomorphic Encryption (FHE)
• Secure Multi-Party Computation (MPC)
• Zero Knowledge Proof (e.g. Blockchain)
• Confidential Computing through TEE (e.g. SGX and TrustZone)

• Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs NSDI'23

• MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism HPCA'23

• TopoOpt: Optimizing the Network Topology for Distributed DNN Training NSDI'23

• Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression ASPLOS'23

• Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Training arxiv

• Zeus: Understanding and Optimizing {GPU} Energy Consumption of {DNN} Training NSDI'23

• ModelKeeper: Accelerating DNN Training via Automated Training Warmup NSDI'23

• HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework VLDB'22

• Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning OSDI'22

• FastMoE: A Fast Mixture-of-Expert Training System arXiv preprint arXiv:2103.13262

• λDNN: Achieving Predictable Distributed DNN Training with Serverless Architectures TC'21

• STRONGHOLD: Fast and Affordable Billion-scale Deep Learning Model Training SC'22

• AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness NeurIPS '22

• Whale: Efficient Giant Model Training over Heterogeneous {GPUs} ATC'22

• Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization OSDI'22

• NASPipe: High Performance and Reproducible Pipeline Parallel Supernet Training via Causal Synchronous Parallelism ASPLOS'22

• Out-of-order backprop: an effective scheduling technique for deep learning Eurosys'22

• Varuna: Scalable, Low-cost Training of Massive Deep Learning Models Eurosys'22

• Megatron-LM SC'21

• Chimera: efficiently training large-scale neural networks with bidirectional pipelines SC'21

• Piper: Multidimensional Planner for DNN Parallelization NeurIPS'21

• Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

• Efficient Sparse Collective Communication and its application to Accelerate Distributed Deep Learning SIGCOMM'21

• PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models ICML'21

• DAPPLE: An Efficient Pipelined Data Parallel Approach for Large Models Training PPOPP'21

• Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads ASPLOS'22

• TeraPipe:Large-Scale Language Modeling with Pipeline Parallelism ICML'21

• PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications OSDI'20

• KungFu OSDI'20

• A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters OSDI'20

• PipeDream: Pipeline Parallelism for DNN Training SOSP'19

• GeePS: Scalable Deep Learning on Distributed GPUs with a GPU-Specialized Parameter Server Eurosys'16

• SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient

• awesome distributed deep learning

• awsome parallelism

• Fast and Efficient Model Serving Using Multi-GPUs with Direct-Host-Access Eurosys'23

• Hidet: Task-Mapping Programming Paradigm for Deep Learning Tensor Programs.

• MPCFormer: fast, performant, and private transformer inference with MPC ICLR'23

• High-throughput Generative Inference of Large Language Model with a Single GPU

• VELTAIR: Towards High-Performance Multi-Tenant Deep Learning Serving via Adaptive Compilation and Scheduling ASPLOS'22

• DVABatch: Diversity-aware Multi-Entry Multi-Exit Batching for Efficient Processing of DNN Services on GPUs ATC'22

• Cocktail: A Multidimensional Optimization for Model Serving in Cloud NSDI'22

• Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing ATC'22

• RIBBON: cost-effective and qos-aware deep learning model inference using a diverse pool of cloud computing instances SC'21

• INFaaS: Automated Model-less Inference Serving ATC'21

• Abacus SC'21

• Serving DNNs like Clockwork: Performance Predictability from the Bottom Up OSDI'20

• Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models

• Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving ATC'19

• Nexus: a GPU cluster engine for accelerating DNN-based video analysis SOSP'19

• Clipper:A low-latency prediction-serving system NSDI'17

• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts MLSYS'23

• Tutel: Adaptive Mixture-of-Experts at Scale MLSYS'23

• FastMoE: A Fast Mixture-of-Expert Training System PPOPP'22

• awesome MoE

• MoE Paper

• AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers

• ElasticFlow: An Elastic Serverless Training Platform for Distributed Deep Learning ASPLOS'23

• Lucid: A Non-Intrusive, Scalable and Interpretable Scheduler for Deep Learning Training Jobs ASPLOS'23

• Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning NSDI'23

• Multi-Resource Interleaving for Deep Learning Training SIGCOMM'22

• Synergy : Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters OSDI'22

• Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning OSDI'21

• Chronus: A Novel Deadline-aware Scheduler for Deep Learning Training Jobs SOCC'21

• Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads OSDI'20

• Tiresias -- A GPU Cluster Manager for Distributed Deep Learning Training without complete job information NSDI'19

• awesome DL scheduler

• Spada: Accelerating Sparse Matrix Multiplication with Adaptive Dataflow ASPLOS'23

• MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant GPU Clusters SOCC'22

• Accpar: Tensor partitioning for heterogeneous deep learning accelerators HPCA'20

• Hidet: Task Mapping Programming Paradigm for Deep Learning Tensor Programs ASPLOS'23

• iGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud TPDS'22

• CheckFreq: Frequent, Fine-Grained DNN Checkpointing FAST'22

• Efficient Quantized Sparse Matrix Operations on Tensor Cores SC'22

• Harmony: Overcoming the hurdles of GPU memory capacity to train massive DNN models on commodity servers VLDB'22

• Pets ATC'22

• PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections OSDI'21

• APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Cores SC'21

• iGUARD SOSP'21

• Fluid: Resource-Aware Hyperparameter Tuning Engine MLSYS'21

• Baechi: Fast Device Placement on Machine Learning Graphs SOCC'20

• Dynamic Parameter Allocation in Parameter Servers VLDB'20

• Data Movement Is All You Need: A Case Study on Optimizing Transformers

• Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platforms OSDI'23

• COGNN SC'22

• TC-GNN: Accelerating Sparse Graph Neural Network Computation Via Dense Tensor Core on GPUs

• GNNAdvisor: An Efficient Runtime System for GNN Acceleration on GPUs OSDI'21

• Marius: Learning Massive Graph Embeddings on a Single Machine OSDI'21

• Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads OSDI'21

• BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling MLSYS'22

• Accelerating Large Scale Real-Time GNN Inference Using Channel Pruning VLDB'21

• Reducing Communication in Graph Neural Network Training SC'20

• awesome GNN

• Fine-tuning giant neural networks on commodity hardware with automatic pipeline model parallelism ATC'21

• Characterizing Variability in Large-Scale, Accelerator-Rich Systems SC'22

• Prediction of the Resource Consumption of Distributed Deep Learning Systems SIGMETRICS'22

• AI-Enabling Workloads on Large-Scale GPU-Accelerated System: Characterization, Opportunities, and Implications HPCA'22