Awesome Federated Learning

This repository maintains a collection of papers, articles, videos, frameworks, etc of federated learing, for the purpose of learning and research.

Note:

Italic contents are the ones worth reading
Last Update: 16 Aug, 2020

Awesome Federated Learning

Introduction

Federated Learning Comic [Google Blog]
Federated Learning: Collaborative Machine Learning without Centralized Training Data [Google Blog]
GDPR, Data Shotrage and AI (AAAI-19) [Video]
Federated Learning: Machine Learning on Decentralized Data (Google I/O'19) [Youtube]
Federated Learning White Paper V1.0 [Paper]
Federated learning: distributed machine learning with data locality and privacy [Blog]

Survey

Federated Machine Learning: Concept and Applications [Paper]
Federated Learning: Challenges, Methods, and Future Directions [Paper]
Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection [Paper]
Federated Learning in Mobile Edge Networks: A Comprehensive Survey [Paper]
Federated Learning for Wireless Communications: Motivation, Opportunities and Challenges [Paper]
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey [Paper]
Advances and Open Problems in Federated Learning [Paper]

Benchmarks

LEAF: A Benchmark for Federated Settings [Paper] [Github] [Recommend]
A Performance Evaluation of Federated Learning Algorithms [Paper]
Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking [Paper]

Model Aggregation

One-Shot Federated Learning [Paper]
Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating [Paper] (NIPS 2019 Workshop)
Bayesian Nonparametric Federated Learning of Neural Networks [Paper] (ICML 2019)
Agnostic Federated Learning [Paper] (ICML 2019)
Federated Learning with Matched Averaging [Paper] (ICLR 2020)
Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications [Paper]

Statistical Heterogeneity

Federated Learning with Non-IID Data [Paper]
The Non-IID Data Quagmire of Decentralized Machine Learning [Paper]
Robust and Communication-Efficient Federated Learning from Non-IID Data [Paper] [IEEE transactions on neural networks and learning systems]
FedMD: Heterogenous Federated Learning via Model Distillation [Paper] (NIPS 2019 Workshop)
First Analysis of Local GD on Heterogeneous Data [Paper]
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning [Paper]
Federated Optimization for Heterogeneous Networks [Paper]
On the Convergence of FedAvg on Non-IID Data [Paper] [OpenReview]
Agnostic Federated Learning [Paper] (ICML 2019)
Local SGD Converges Fast and Communicates Little [Paper]
Improving Federated Learning Personalization via Model Agnostic Meta Learning [Paper] (NIPS 2019 Workshop)
Adaptive Gradient-Based Meta-Learning Methods [Paper] (NIPS 2019 Workshop)
Federated Adversarial Domain Adaptation [Paper] (ICLR 2020)
LoAdaBoost: Loss-Based AdaBoost Federated Machine Learning on Medical Data [Paper]
On Federated Learning of Deep Networks from Non-IID Data: Parameter Divergence and the Effects of Hyperparametric Methods [Paper] [Rejected in ICML 2020]
Overcoming Forgetting in Federated Learning on Non-IID Data [Paper] [NIPS 2019 Workshop]
FedMAX: Activation Entropy Maximization Targeting Effective Non-IID Federated Learning [Video] [NIPS 2019 Workshop]
Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification [Paper] [NIPS 2019 Workshop]
Fair Resource Allocation in Federated Learning [Paper]
Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data [Paper]
Think Locally, Act Globally: Federated Learning with Local and Global Representations [Paper] [NIPS 2019 Workshop]

Meta Learning

Federated Meta-Learning with Fast Convergence and Efficient Communication [Paper]
Federated Meta-Learning for Recommendation [Paper]
Adaptive Gradient-Based Meta-Learning Methods [Paper]

Multi-task Learning

MOCHA: Federated Multi-Task Learning [Paper] [NIPS 2017] [Slides]
Variational Federated Multi-Task Learning [Paper]
Federated Kernelized Multi-Task Learning [Paper]
Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints [Paper] [NIPS 2019 Workshop]

Convergence

A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication [Paper] [NIPS 2018]
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning [Paper]
Federated Optimization for Heterogeneous Networks [Paper]
On the Convergence of FedAvg on Non-IID Data [Paper] [OpenReview]
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent [Paper] [NIPS 2017]
Communication Efficient Decentralized Training with Multiple Local Updates [Paper]
First Analysis of Local GD on Heterogeneous Data [Paper]
MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling [Paper]
Local SGD Converges Fast and Communicates Little [Paper]
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum [Paper]
Adaptive Federated Learning in Resource Constrained Edge Computing Systems [Paper] [IEEE Journal on Selected Areas in Communications, 2019]
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning [Paper] [AAAI 2018]
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization [Paper] [ICML 2019]
Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data [Paper]
Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data [Paper] [NIPS 2019 Workshop]

System

Towards Federated Learning at Scale: System Design [Paper] [Must Read]
Demonstration of Federated Learning in a Resource-Constrained Networked Environment [Paper]
Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection [Paper]
Applied Federated Learning: Improving Google Keyboard Query Suggestions [Paper]
Federated Learning and Differential Privacy: Software tools analysis, the Sherpa.ai FL framework and methodological guidelines for preserving data privacy [Paper] (Startup)

Communication Efficiency

Communication-Efficient Learning of Deep Networks from Decentralized Data [Paper] [Github] [Google] [Must Read]
Two-Stream Federated Learning: Reduce the Communication Costs [Paper] [2018 IEEE VCIP]
Client-Edge-Cloud Hierarchical Federated Learning [Paper]
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization [Paper] [NIPS 2019], Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi.
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training [Paper] [ICLR 2018] Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally
The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication [Paper] Sebastian U Stich and Sai Praneeth Karimireddy, 2019.
A Communication Efficient Collaborative Learning Framework for Distributed Features [Paper] [NIPS 2019 Workshop]
Active Federated Learning [Paper] [NIPS 2019 Workshop]
Communication-Efficient Distributed Optimization in Networks with Gradient Tracking and Variance Reduction [Paper] [NIPS 2019 Workshop]
Gradient Descent with Compressed Iterates [Paper] [NIPS 2019 Workshop]

Compression

Robust and Communication-Efficient Federated Learning from Non-IID Data [Paper], 2019
Expanding the Reach of Federated Learning by Reducing Client Resource Requirements [Paper] Sebastian Caldas, Jakub Konecny, H Brendan McMahan, and Ameet Talwalkar, 2018
Federated Learning: Strategies for Improving Communication Efficiency [Paper] [NIPS2016 Workshop] [Google]
Natural Compression for Distributed Deep Learning [Paper] Samuel Horvath, Chen-Yu Ho, Ludovit Horvath, Atal Narayan Sahu, Marco Canini, and Peter Richtarik, 2019.
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization [Paper], 2019
ATOMO: Communication-efficient Learning via Atomic Sparsification [Paper] [NIPS 2018], H. Wang, S. Sievert, S. Liu, Z. Charles, D. Papailiopoulos, and S. Wright.
vqSGD: Vector Quantized Stochastic Gradient Descent [Paper] Venkata Gandikota, Raj Kumar Maity, and Arya Mazumdar, 2019.
QSGD: Communication-efficient SGD via gradient quantization and encoding [Paper] [NIPS 2017], Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic.
cpSGD: Communication-efficient and differentially-private distributed SGD [Paper]
Federated Optimization: Distributed Machine Learning for On-Device Intelligence [Paper] [Google]
Distributed Mean Estimation with Limited Communication [Paper] [ICML 2017], Ananda Theertha Suresh, Felix X. Yu, Sanjiv Kumar, and H Brendan McMahan.
Randomized Distributed Mean Estimation: Accuracy vs Communication [Paper] Frontiers in Applied Mathematics and Statistics, Jakub Konecny and Peter Richtarik, 2016
Error Feedback Fixes SignSGD and other Gradient Compression Schemes [Paper] [ICML 2019], Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi.
ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning [Paper] [ICML 2017], H. Zhang, J. Li, K. Kara, D. Alistarh, J. Liu, and C. Zhang.

Important-Based Updating

eSGD: Communication Efficient Distributed Deep Learning on the Edge [Paper] [USENIX 2018 Workshop (HotEdge 18)]
CMFL: Mitigating Communication Overhead for Federated Learning [Paper]

Decentralization

Communication Compression for Decentralized Training [Paper] [NIPS 2018], H. Tang, S. Gan, C. Zhang, T. Zhang, and J. Liu.
𝙳𝚎𝚎𝚙𝚂𝚚𝚞𝚎𝚎𝚣𝚎: Decentralization Meets Error-Compensated Compression [Paper] Hanlin Tang, Xiangru Lian, Shuang Qiu, Lei Yuan, Ce Zhang, Tong Zhang, and Ji Liu, 2019

Resource Allocation

Participants Selection

Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge [Paper] (FedCS)
Hybrid-FL for Wireless Networks: Cooperative Learning Mechanism Using Non-IID Data [Paper]
- Ask to upload some data from client to server
Efficient Training Management for Mobile Crowd-Machine Learning: A Deep Reinforcement Learning Approach [Paper]
- Reward function: accumulated data, energy consumption, training accuracy
Fair Resource Allocation in Federated Learning [Paper]
Low-latency Broadband Analog Aggregation For Federated Edge Learning [Paper]
Federated Learning over Wireless Fading Channels [Paper]
Federated Learning via Over-the-Air Computation [Paper]

Adaptive Aggregation

Asynchronous Federated Learning for Geospatial Applications [Paper] [ECML PKDD Workshop 2018]
Asynchronous Federated Optimization [Paper]
Adaptive Federated Learning in Resource Constrained Edge Computing Systems [Paper] [IEEE Journal on Selected Areas in Communications, 2019]

Incentive Mechanism

Incentive Mechanism for Reliable Federated Learning: A Joint Optimization Approach to Combining Reputation and Contract Theory [Paper]
Motivating Workers in Federated Learning: A Stackelberg Game Perspective [Paper]
Incentive Design for Efficient Federated Learning in Mobile Networks: A Contract Theory Approach [2019] [Paper]

Vertical Federated Learning

A Quasi-Newton Method Based Vertical Federated Learning Framework for Logistic Regression [Paper] [NIPS 2019 Workshop]

Adversarial Attacks

Can You Really Backdoor Federated Learning? [Paper]
Model Poisoning Attacks in Federated Learning [Paper] [NIPS workshop 2018]

Data Privacy and Confidentiality

Gradient-Leaks: Understanding and Controlling Deanonymization in Federated Learning [Paper] [NIPS 2019 Workshop]
Quantification of the Leakage in Federated Learning [Paper]

Courses

Applied Cryptography [Udacity]
- Cryptography basics

Differential Privacy

A Brief Introduction to Differential Privacy [Blog]
Deep Learning with Differential Privacy [Paper]
- Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.
Learning Differentially Private Recurrent Language Models [Paper]
Federated Learning with Bayesian Differential Privacy [Paper] [NIPS 2019 Workshop]
Private Federated Learning with Domain Adaptation [Paper] [NIPS 2019 Workshop]
cpSGD: Communication-efficient and differentially-private distributed SGD [Paper]
Federated Learning with Bayesian Differential Privacy [Paper] [NIPS 2019 Workshop]

PATE

Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data [Paper]
- Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian J. Goodfellow, and Kunal Talwar.
- Private Aggregation of Teacher Ensembles (PATE)
Scalable Private Learning with PATE [Paper]
- Extension of PATE

The original PATE paper at ICLR 2017 and recording of the ICLR oral
The ICLR 2018 paper on scaling PATE to large number of classes and imbalanced data.
GitHub code repo for PATE
GitHub code repo for the refined privacy analysis of PATE

Secure Multi-party Computation

Secret Sharing

Simple Introduction to Sharmir's Secret Sharing and Lagrange Interpolation [Youtube]
Secret Sharing, Part 1 [Blog]: Shamir's Secret Sharing & Packed Variant
Secret Sharing, Part 2 [Blog]: Improve efficiency
Secret Sharing, Part 3 [Blog]

SPDZ

Basics of Secure Multiparty Computation [Youtube]: based on Shamir's Secret Sharing
What is SPDZ?
- Part 1: MPC Circuit Evaluation Overview [Blog]
- Part 2: Circuit Evaluation [Blog]
The SPDZ Protocol [Blog]: implementation codes included

Advance (Not Recommended For Beginners)

Multiparty Computation from Somewhat Homomorphic Encryption [Paper]
- SPDZ introduction
Practical Covertly Secure MPC for Dishonest Majority – or: Breaking the SPDZ Limits [Paper]
MASCOT: Faster Malicious Arithmetic Secure Computation with Oblivious Transfer [Paper]
Removing the crypto provider and instead letting the parties generate these triples on their own
Overdrive: Making SPDZ Great Again [Paper]
Safetynets: Verifiable execution of deep neural networks on an untrusted cloud

Build Safe AI Series

Building Safe A.I. [Blog]
- A Tutorial for Encrypted Deep Learning
- Use Homomorphic Encryption (HE)
Private Deep Learning with MPC [Blog]
- A Simple Tutorial from Scratch
- Use Multiparty Compuation (MPC)
Private Image Analysis with MPC [Blog]
- Training CNNs on Sensitive Data
- Use SPDZ as MPC protocol

MPC related Paper

Helen: Maliciously Secure Coopetitive Learning for Linear Models [Paper] (NIPS 2019 Workshop)

Privacy Preserving Machine Learning

Privacy-Preserving Deep Learning [Paper]
Privacy Partition: A Privacy-Preserving Framework for Deep Neural Networks in Edge Networks [Paper]
Practical Secure Aggregation for Privacy-Preserving Machine Learning [Paper] (Google)
- Secure Aggregation: The problem of computing a multiparty sum where no party reveals its update in the clear—even to the aggregator
- Goal: securely computing sums of vectors, which has a constant number of rounds, low communication overhead, robustness to failures, and which requires only one server with limited trust
- Need to have basic knowledge of cryptographic algorithms such as secret sharing, key agreement, etc.
Practical Secure Aggregation for Federated Learning on User-Held Data [Paper] (Google)
- Highly related to Practical Secure Aggregation for Privacy-Preserving Machine Learning
- Proposed 4 protocol one by one with gradual improvement to meet the requirement of secure aggregation propocol.
SecureML: A System for Scalable Privacy-Preserving Machine Learning [Paper]
DeepSecure: Scalable Provably-Secure Deep Learning [Paper]
Chameleon: A Hybrid Secure Computation Framework for Machine Learning Applications [Paper]
Contains several MPC frameworks

Other Learning

Federated Reinforcement Learning [Paper]
Secure Federated Transfer Learning [Paper]

Frameworks

PySyft [Github]
- A Generic Framework for Privacy Preserving Peep Pearning [Paper]
Tensorflow Federated [Web]
FATE [Github]
Baidu PaddleFL [Github]
Nvidia Clara SDK [Web]

Workshops

NIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality 1 [Video]
NIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality 2 [Video]
NIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality 3 [Video]

Applications

Federated Learning Approach for Mobile Packet Classification [Paper]
Federated Learning for Ranking Browser History Suggestions [Paper] [NIPS 2019 Workshop]

Healthcare

HHHFL: Hierarchical Heterogeneous Horizontal Federated Learning for Electroencephalography [Paper] [NIPS 2019 Workshop]
Learn Electronic Health Records by Fully Decentralized Federated Learning [Paper] [NIPS 2019 Workshop]
Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records [Paper] [News]
- MIT CSAI, Harvard Medical School, Tsinghua University
Federated learning of predictive models from federated Electronic Health Records. [Paper]
- Boston University, Massachusetts General Hospital
FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare [Paper]
- Microsoft Research Asia
Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation [Paper]
- Intel
NVIDIA Clara Federated Learning to Deliver AI to Hospitals While Protecting Patient Data [Blog]
- Nvidia
What is Federated Learning [Blog]
- Nvidia
Split learning for health: Distributed deep learning without sharing raw patient data [Paper]
Two-stage Federated Phenotyping and Patient Representation Learning [Paper] [ACL 2019]
Federated Tensor Factorization for Computational Phenotyping https://dl.acm.org/doi/10.1145/3097983.3098118 SIGKDD 2017
FedHealth- A Federated Transfer Learning Framework for Wearable Healthcare [Paper] [ICJAI19 workshop]
Multi-Institutional Deep Learning Modeling Without Sharing Patient Data- A Feasibility Study on Brain Tumor Segmentation [Paper] [MICCAI'18 Workshop]
Federated Patient Hashing [Paper] [AAAI'20]

Natual Language Processing

Google

Federated Learning for Mobile Keyboard Prediction [Paper]
Applied Federated Learning: Improving Google Keyboard Query Suggestions [Paper]
Federated Learning Of Out-Of-Vocabulary Words [Paper]
Federated Learning for Emoji Prediction in a Mobile Keyboard [Paper]

Snips

Federated Learning for Wake Keyword Spotting [Paper] [Blog] [Github]

Computer Vision

Performance Optimization for Federated Person Re-identification via Benchmark Analysis [Paper] [ACMMM20] [Github]
Real-World Image Datasets for Federated Learning [Paper]
- Webank & Extreme Vision
FedVision- An Online Visual Object Detection Platform Powered by Federated Learning [Paper] [IAAI20]
Federated Learning for Vision-and-Language Grounding Problems [Paper] [AAAI20]

Recommendation

Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System [Paper]
- Huawei
Federated Meta-Learning with Fast Convergence and Efficient Communication [Paper]
- Huawei

Industrial

Turbofan POC: Predictive Maintenance of Turbofan Engines using Federated Learning [Github]
Turbofan Tycoon Simulation by Cloudera/FastForwardLabs [Web]
Firefox Search Bar [Blog] [Github] [Github]
- Detail explaination of their implementationn of Federated Learning in production.

Company

Privacy.ai [Website]
OpenMined [Website]
Arkhn [Website]: Healthcare data
Owkin [Website]: Medical research
Snips [Website]: Voice platform
XAIN [Website] [Github]: Automated Invoicing
S20 [Website]: Multiple third party collaboration

tonylv / awesome-federated-learning Goto Github PK

awesome-federated-learning's Introduction