Ray dashboard should not exit when GCS exits or becomes unavailable about ray HOT 4 OPEN

cc: @alexeykudinkin

from ray.

Per discussion, @rynewang has a related task and will address this altogether.

from ray.

We can simply remove the Dashboard check on gcs liveness, or follow the config RAY_gcs_rpc_server_reconnect_timeout_s used by raylet and workers.

However we also need to update PythonGcsClient to do proper retrying. For example, when GCS is down, now PythonGcsClient keeps all pending requests until the GCS is back alive. If there are so many reqs, the client may OOM. The C++ GcsClient, on the other hand, blocks the thread when the amount of pending reqs exceeds RAY_gcs_grpc_max_request_queued_max_bytes. We need to adapt this behavior to PythonGcsClient, by removing the latter and binding the C++ GcsClient to Python side.

This involves (in this order):

• #45817
• #46057
• #46061
• #45289

from ray.

Process:

1. #45817
2. #46186
3. we can resolve this issue here: remove "number of failed checks". Since it now uses GcsClient (GcsRpcClient) it will inherit its "kill self on gcs_rpc_server_reconnect_timeout_s timeout` behavior and wo don't need to do anything on python side.
4. gcs client async to reimplement GcsAioClient

from ray.