ibm / adaptive-federated-learning Goto Github PK

Code for paper "Adaptive Federated Learning in Resource Constrained Edge Computing Systems"

License: MIT License

Python 100.00%

adaptive-federated-learning's Introduction

Adaptive Federated Learning in Resource Constrained Edge Computing Systems

This repository includes source code for the paper S. Wang, T. Tuor, T. Salonidis, K. K. Leung, C. Makaya, T. He, and K. Chan, "Adaptive federated learning in resource constrained edge computing systems," IEEE Journal on Selected Areas in Communications, vol. 37, no. 6, pp. 1205 – 1221, Jun. 2019.

@article{wang2019adaptive,
  title={Adaptive federated learning in resource constrained edge computing systems},
  author={Wang, Shiqiang and Tuor, Tiffany and Salonidis, Theodoros and Leung, Kin K and Makaya, Christian and He, Ting and Chan, Kevin},
  journal={IEEE Journal on Selected Areas in Communications},
  volume={37},
  number={6},
  pages={1205-1221},
  year={2019}
}

Getting Started

The code runs on Python 3 with Tensorflow version 1 (>=1.13). To install the dependencies, run

pip3 install -r requirements.txt

Then, download the datasets manually and put them into the datasets folder.

For MNIST dataset, download from http://yann.lecun.com/exdb/mnist/ and put the standalone files into datasets/mnist.
For CIFAR-10 dataset, download the "CIFAR-10 binary version (suitable for C programs)" from https://www.cs.toronto.edu/~kriz/cifar.html, extract the standalone *.bin files and put them into datasets/cifar-10-batches-bin.

To test the code:

Run server.py and wait until you see Waiting for incoming connections... in the console output.
Run 5 parallel instances of client.py on the same machine as the server.
You will see console outputs on both the server and clients indicating message exchanges. The code will run for a few minutes before finishing.
After the server and clients finish, run plot_multi_run.py which will plot the result. This figure will look similar to the SVM(SGD) subfigures in Fig. 4 of the paper (but with higher fluctuation).

Code Structure

All configuration options are given in config.py which also explains the different setups that the code can run with.

The results are saved as CSV files in the results folder. The CSV files should be deleted before starting a new round of experiment. Otherwise, the new results will be appended to the existing file.

Currently, the supported datasets are MNIST and CIFAR-10, and the supported models are SVM and CNN. The code can be extended to support other datasets and models too.

Citation

When using this code for scientific publications, please kindly cite the above paper.

Contributors

This code was written by Shiqiang Wang and Tiffany Tuor.

adaptive-federated-learning's People

Contributors

Stargazers

Watchers

Forkers

yf817 lebeausc bailingbird bhaskers-blu-org1 qxwsniff cnetboy pengchaohan jungjunepyo alerfaromeoo enfangcui oodunsi1 zpskt danhlephuoc mufchen ahmedcs cottaging xqzhong sssupertian wallacehop yyuzhong emma-mens vandung85 jasoncai10 zealya bberzhou warriormay abigail807 ankurhcu soo9826 coplanetary sean-xia tzq2doc abdulmoneamali csjs0820 yyds-xtt fbucket djmartingale arj119 jimmy-inl udai-arneja frontop137 sinofeng shubhang-pandey summitstha anandjagadeesh duanhuicit anso98 tqsenchanting changsenxia csharpyds huangshw252 hd818 liusu169 izfree-edu rohnitt ttt-perfect annimoham hlzcj startleback dddduc jmt1995 kancheng ghas-results tongxinxiaohui naya0000 dwngh learnibg chenwenqiao iwantsomememories nvt18624

adaptive-federated-learning's Issues

I have found a problem, can you help me solve it?

I have found a problem, can you help me solve it?
For ：
label = unpack('>B', f_labels.read(1))[0]
struct.error: unpack requires a buffer of 1 bytes

It is getting memory error with the MNIST dataset.

While I am running the server with the MNIST dataset, it is getting memory error. The stacktrace is mentioned below. Please help me fixing this.
Traceback (most recent call last):
File "C:\Users\Rohit\AppData\Local\Programs\Python\Python37\adaptive-federated-learning\server.py", line 25, in
train_image, train_label, test_image, test_label, train_label_orig = get_data(dataset, total_data, dataset_file_path)
File "C:\Users\Rohit\AppData\Local\Programs\Python\Python37\adaptive-federated-learning\data_reader\data_reader.py", line 30, in get_data
train_image, train_label = mnist_extract(start_index_train, total_data_train, True, dataset_file_path)
File "C:\Users\Rohit\AppData\Local\Programs\Python\Python37\adaptive-federated-learning\data_reader\mnist_extractor.py", line 54, in mnist_extract
return mnist_extract_samples(sample_list, is_train, file_path)
File "C:\Users\Rohit\AppData\Local\Programs\Python\Python37\adaptive-federated-learning\data_reader\mnist_extractor.py", line 38, in mnist_extract_samples
x = np.array(list(f_images.read(rows_im * cols_im)))/255.0
MemoryError

Help to change this repository!

Hello there!
I’m writing to you regarding a github repository you have made, which is of interest to me.
The repository I’m talking about is the current one and I’m doing a project which consists of this one”https://github.com/qub-blesson/FedAdapt" but i made some changes to yours like i wanted to run it for 3 virtual machines (windows 10) with 1Gb of Ram, 20Gb of hard disk, and this project is so important to me, but i wanted to only use SVM and CNN with stochastic gradient descent and the dataset is also just MNIST dataset. So now i want to use and add the part where you have done the online cloud offloading.

So would you please please please help me merge your code with the later repository for the offloading part? and also is it possible to do all that in python or jupyter notebook? i really am so desperate please please help a student in need, I know you must be so busy but i’m so mentally down that i can’t function well and the deadline for this project is approaching please help me merge the offloading part of your project with my customized project with the mentioned features above. it’s the exact federated learning with resource-constrained algorithms but it need the offloading part which i cant wrap up my mind around it.

Does running client.py on five servers mean configuring the same Python environment on all five servers?

自适应联邦学习

您好！我在拜读您这篇论文时有些地方感到不解，虽然您的标题是基于资源受限边缘计算系统下的自适应联邦学习，但我感觉你的模型构造并不是基于边缘计算系统。

我理解中的基于边缘计算系统的联邦学习应如上图有边缘聚合和全局聚合，但您的算法2和算法3中只有节点和聚合，这更像是联邦学习的经典模型。能否请您分享一下您当时是怎么思考的？又或者能不能麻烦您指出我理解上的问题？

the issue of cnncifar-10

Traceback (most recent call last):
File "C:\FL\adaptive-federated-learning-master\server.py", line 25, in
train_image, train_label, test_image, test_label, train_label_orig = get_data(dataset, total_data, dataset_file_path)
File "C:\FL\adaptive-federated-learning-master\data_reader\data_reader.py", line 60, in get_data
train_image, train_label = cifar_10_extract(0, total_data_train, True, dataset_file_path)
File "C:\FL\adaptive-federated-learning-master\data_reader\cifar_10_extractor.py", line 62, in cifar_10_extract
return cifar_10_extract_samples(sample_list, is_train, file_path)
File "C:\FL\adaptive-federated-learning-master\data_reader\cifar_10_extractor.py", line 35, in cifar_10_extract_samples
y = get_one_hot_from_label_index(label)
File "C:\FL\adaptive-federated-learning-master\util\utils.py", line 24, in get_one_hot_from_label_index
one_hot[label] = 1
IndexError: index 128 is out of bounds for axis 0 with size 10

Run 5 parallel instances of client.py on the same machine as the server

I don't know how to run 5 parallel instances of client.py on the same machine as the server.
I've tried multithreading and multi-process, but still stuck at this step "Waiting for incoming connections...".
Could you please teach me how to do this?

ibm / adaptive-federated-learning Goto Github PK

adaptive-federated-learning's Introduction

Adaptive Federated Learning in Resource Constrained Edge Computing Systems

Getting Started

Code Structure

Citation

Contributors

adaptive-federated-learning's People

Contributors

Stargazers

Watchers

Forkers

adaptive-federated-learning's Issues

Recommend Projects

Recommend Topics

Recommend Org