[core] Raylet should wait for RuntimeEnvAgent to start before receiving tasks. about ray HOT 9 CLOSED

can reproduce - with a slow rt env agent + task pressure

from ray.

cc @hongchaodeng @jjyao

from ray.

@rynewang Hi, is it convenient to say from which Ray version this problem started, We recently wanted to update the Ray version used on our system (currently using 2.9.0), but found some new problems with Ray 2.20.0 as feedback in the github issue (autoscaler instability, RuntimeEnvSetupError, etc.). So I'm looking for a slightly more stable version to upgrade, Thank you!

from ray.

@yx367563 could you elaborate more on the issues you are facing. Have you created github issues for them?

from ray.

P0 label reflects it already -- but the issue is quite severe; it causes intermittent failures of any production job in which workers access a runtime environment.

from ray.

Nice!
"task pressure" certainly describes our use-case
(we schedule many tasks per workload :) )

from ray.

@yx367563 could you elaborate more on the issues you are facing. Have you created github issues for them?

@jjyao @rynewang The problem faced is in this issue (#45311). I mean, can you know which Ray version this problem was introduced from? I will temporarily avoid upgrading to this version of Ray.

from ray.

@yx367563 we don't have an accurate "culprit" commit. You can stay at your current version until this issue is resolved (and to the next version), or do some stress testing with your workload to confirm.

from ray.

PR under review

from ray.