[Core|RayOnSpark] finding ports fails when launching Ray on Spark and verbose logs about ray HOT 7 OPEN

@jjyao can you help triage this? Thanks

from ray.

cc @WeichenXu123 can you take a look at this one?

from ray.

@WeichenXu123 gentle ping here.

from ray.

checking

from ray.

ah , you are creating 70 Ray worker nodes , but how many spark worker nodes there ?

we recommend to make one Ray worker node occupies all CPU/GPUs in a spark worker node. i.e. one spark worker node launches at most one Ray worker node. This reduces risks of port conflicts.

we have mechanism to prevent port conflicts.

according to your error message '1000 ports from 35000 to 35999', you must have started at least 16 Ray worker nodes in the same machine .

from ray.

ports > 30000 might be used by other Ray components. Reducing Ray worker nodes number per spark worker node should address the issue. @jjyao Does Ray system service use port range 10000 ~ 20000 ?

from ray.

Thanks for having a look @WeichenXu123 . I will answer your questions below and add some thoughts:

ah , you are creating 70 Ray worker nodes , but how many spark worker nodes there ?

We have 100+ nodes, so it should be fine to have 70 ray workers. But yes I do see Ray often putting more than one worker on the same node. How can I tell Ray not to without telling it to use all the resources of the machine?

we recommend to make one Ray worker node occupies all CPU/GPUs in a spark worker node

I think there might be a few issues with this: the nodes in the cluster might be different so it will be hard. Also, it might take longer for my ray task to launch if it has to wait for a full machine to be available instead of just using whatever is available.
Also, If I somehow configure ray to use all CPU/GPU in the spark worker node that will potentially lead to underutilisation of the cluster right? We have really big nodes, sometimes with more than 150 cores.

from ray.