Multi-machine distribution problem about pipedream HOT 6 OPEN

Not sure I have enough information here to debug.

What model are you running? How did you generate the model if it's not one of the standard models we provide in this repository? What does your configuration file look like?

from pipedream.

I used vgg16

/runtime/image_classification/models/vgg16/gpus=16/hybrid_conf.json
{
"module_to_stage_map": [0, 1, 2, 3, 4, 5, 5],
"stage_to_rank_map": {"0": [0, 1, 2, 3, 4, 5, 6, 7], "1": [8, 9], "2": [10, 11], "3": [12], "4": [13, 14], "5": [15]}
}
~

from pipedream.

Hmm, that hybrid file seems different from what's checked in in this repository.

{
    "module_to_stage_map": [0, 1, 1],
    "stage_to_rank_map": {
        "0": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
        "1": [15]
    },
    "stage_to_depth_map": {
        "0": 1,
        "1": 0
    }
}

from pipedream.

I execute the code in strict accordance with readme.txt:

step1:

         CUDA_VISIBLE_DEVICES=0 python main.py -a vgg16 -b 64 --data_dir ../../data/

step2:

        pythonoptimizer_graph_hierarchical.py 
        -f ../profiler/image_classification/profiles/vgg16/graph.txt 
        -n 8 2  
        --activation_compression_ratio 1 
        -o vgg16_partitioned   
        -b 4294967296  1879048192

step3:

        python convert_graph_to_model.py 
       -f vgg16_partitioned/gpus=16.txt 
       -n VGG16Partitioned 
       -a vgg16 
       -o ../runtime/image_classification/models/vgg16/gpus=16
       --stage_to_num_ranks 0:8,1:2,2:4,3:1,4:1

I checked the generated configuration file again and again
hybrid config file :

{ "module_to_stage_map": [0, 1, 2, 3, 4, 4], "stage_to_rank_map": {"0": [0, 1, 2, 3, 4, 5, 6, 7], "1": [8, 9], "2": [10, 11, 12, 13], "3": [14], "4": [15]} }

I don't know why it's different from yours.But I checked alexnet. gpus = 16. hybrid_conf.json, and there was no "stage_to_depth_map".
{ "module_to_stage_map": [0, 1, 1], "stage_to_rank_map": {"0": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], "1": [15]} }

from pipedream.

Any updates on this? I can't find a "stage_to_depth_map" myself either

from pipedream.

Same question here, I only find "module_to_stage_map" and "stage_to_rank_map" in config files.

from pipedream.