Using your specified gpus' list by customizing the function "init_training_device"! about fedml HOT 8 CLOSED

I guess it is because you put your progress on the front end. Please use the back end command (nohup).
Let me also double-check your configuration later with our own machine, and get back to you later.

from fedml.

I see. The WORKERs were sampled from CLIENTs，WORKER_NUM <= CLIENT_NUM.

from fedml.

@weiyikang Thank you for sharing your code!

from fedml.

1. What are the differences between parameter CLIENT_NUM and WORKER_NUM？
2. How to use the specified GPUs，such as #6，#7 GPU，because other GPUs are used by other people.
   

from fedml.

1. CLIENT_NUM is to describe how many users are involved in training, while WORKER_NUM means the parallel processes during training. If the client number is super large (e.g., 1 million users), a common practice for scalability is to use uniform sampling to select $WORKER_NUM (e.g. 10) of users to train each round.

from fedml.

2. In the init_training_device (main_fedavg.py), you can see that GPU_NUM_PER_SERVER is used to arrange GPU devices to each worker. Using this function, you can customize your arrangement.

from fedml.

2. You can always customize this function to meet your own physical configuration. In your case:

gpu_num_per_machine = 2

def init_training_device(process_ID, fl_worker_num, gpu_num_per_machine):
    # initialize the mapping from process ID to GPU ID: <process ID, GPU ID>
    if process_ID == 0:
        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        return device
    process_gpu_dict = dict()
    for client_index in range(fl_worker_num):
        gpu_index = client_index % gpu_num_per_machine + 6
        process_gpu_dict[client_index] = gpu_index

    logging.info(process_gpu_dict)
    device = torch.device("cuda:" + str(process_gpu_dict[process_ID - 1]) if torch.cuda.is_available() else "cpu")
    logging.info(device)
    return device

from fedml.

The problem has been solved by customizing the function "init_training_device" :

The result as following:

from fedml.