What should I do if I only have a GPUNode with 8 graphics card to run the distributed algorithm? about fedml HOT 17 OPEN

@iuserea I just run our code using 4 and 8 compute nodes, it works well. The issue you mentioned happens when you have only 1 compute node but do not change the compute topology. To address this issue, you can try to modify the computing topology at init_training_device() function, in which you can force all workers/clients run in the same GPU device ID (since many clients/workers share the same GPU device, you may also need to make the batch size smaller to fit the memory constraints, which may degrades the accuracy a little bit and leads to a relatively slow training speed). Besides, you should also change the client_number/worker_number at run_xxx.sh

from fedml.

@chaoyanghe Could fedgkt algorithm be running with only 2 clients or even one client?I tried but I failed.

from fedml.

run_fedgkt.sh:

from fedml.

main_fedgkt.py:

from fedml.

CMD for 10 clients:
sh run_FedGKT.sh 8 cifar10 homo 10 10 1 Adam 0.001 1 0 resnet56 fedml_resnet56_homo_cifar10 "./../../../data/cifar10" 64 10

from fedml.

The success flag I found is that b_all_received = True.
When the process failed, it either failed in b_all_received = false or 'b_all_received' variable even didn't appear after all the clients training has finished.

However it's just the surface of the real problem.

from fedml.

When I set client/worker's number to 2,fedgkt algorithm will also create 8 processes which may result in the failure of itself.

from fedml.

Change your sh script: -n is still 9

from fedml.

@chaoyanghe
The -n option is not essential for training of two clients.

The quesion is when training the two clients,the message below didn't appear.
handle_message_receive_feature_and_logits_from_client
add_model. index = 7

from fedml.

Hi @iuserea We have supported GPU mapping, please have a look at this:

from fedml.

Hi @iuserea , Could you please share your configuration of both software and hardware?

from fedml.

I only have two gpus in one server, how can I train fedgdk?

from fedml.

@iuserea Hi, How do you set your mpi_host_file?

from fedml.

from fedml.

from fedml.

@rambo-coder @iuserea @chaoyanghe Did you guys figure out how to run fedgkt on a single machine with multiple gpus? I followed the thread but was not able to make it finish with success (b_all_received=False).

@chaoyanghe Most researchers has a single machine with multiple gpus. It would be nice to have a guide for this especially if the library is designed specifically for researchers.

from fedml.

@rambo-coder @iuserea @chaoyanghe Did you guys figure out how to run fedgkt on a single machine with multiple gpus? I followed the thread but was not able to make it finish with success (b_all_received=False).

@chaoyanghe Most researchers has a single machine with multiple gpus. It would be nice to have a guide for this especially if the library is designed specifically for researchers.

Hello @korawat-tanwisuth Did you run FedGKT on a single machine with multiple gpus?

from fedml.