Is it possible to run multiple eval_net.py process with multiple workers? about temporal-segment-networks HOT 9 CLOSED

It is OK to run two eval_net.py with 4 workers each. There might be some other error.

from temporal-segment-networks.

Hi @GBJim
I used to meet this problem too. Try adding CUDA_VISIBLE_DEVICES=0,1,2,3 before the python command, where the numbers indicate the available GPUs , It works fine for me.

from temporal-segment-networks.

This error means you might be requesting more workers than the number of available GPU.

Also, you are running 4 workers using eval_net.py. I don't quite understand the title of this issue.

from temporal-segment-networks.

@yjxiong
Oh, I get it.
I have 4 GPU cores and have an eval_net.py process running with 4 workers already.
When I run the second process of eval_net.py with 4 workers, the total 8 workers exceeds the amount of my GPU cores and promtps the error messages.

from temporal-segment-networks.

@yjxiong
Thanks a lot! I will try it again.

from temporal-segment-networks.

@xlliuplus7
Thanks for the tip!

from temporal-segment-networks.

It is strange. I did not add CUDA_VISIBLE_DEVICES=0,1,2,3 or anything else. I can still run two eval_net with 4 workers perfectly.

I have 4 GPU.

from temporal-segment-networks.

@xlliu7 http://stackoverflow.com/questions/34775522/tensorflow-mutiple-sessions-with-mutiple-gpus
you mean this ->$ CUDA_VISIBLE_DEVICES=0 python my_script.py # Uses GPU 0. Right?
But I tried, and get error: error = cudaSuccess: (10 vs 0) invalid device ordinal.
Do you have any suggestions?

from temporal-segment-networks.

@KnightOfTheMoonlight That's exactly what I mean.
I use a workstation with 4 GPUs + CUDA 7.5 + cuDNN v5. The toolbox was compiled with MPI support.
I found it neccessary to add the "CUDA_VISIBLE_DEVICES" environment to use multiple GPUs. Otherwise I will get the "invalid device ordinal" error. I guess the caffe toolbox fails to detect available GPUs on my workstation. Your problem seems to be opposite to mine.
In fact, you can also mannually specify the dev_id in "eval_net.py". See https://github.com/yjxiong/temporal-segment-networks/blob/master/tools/eval_net.py#L32

from temporal-segment-networks.