[Feature] Add default volumes and volumeMounts for `/tmp/ray` to Ray Pods about kuberay HOT 7 OPEN

cc @andrewsykim Would you mind commenting on this issue? Without your comment, I am unable to assign the issue to you. Thanks!

from kuberay.

+1 I think just automating the volume mounting is easier.

from kuberay.

Actually, one thing to consider here is that users may not want to use emptyDir for the ray logs. They may want to persist them with remote storage or local disk on nodes. We might make that harder to do if we automatically add these volumes and mounts

from kuberay.

Actually, one thing to consider here is that users may not want to use emptyDir for the ray logs. They may want to persist them with remote storage or local disk on nodes. We might make that harder to do if we automatically add these volumes and mounts

This makes sense. I am wondering if is it common for users to use volumes to persist logs. In my experience, most users utilize logging tools like FluentBit to stream logs to remote storage. Since my data points are quite limited, would you mind sharing your experiences with volume/volumeMount usage in Kubernetes (not limited to KubeRay)?

• Case 1: If it's common for users to set up volume/volumeMount themselves, we should avoid automatically injecting volume/volumeMount for them. Instead, we just need to write the docs to suggest users to add volume for /tmp/ray.
• Case 2: If it's rare, we can inject volume/volumeMount for users with a feature gate to enable users to disable the injection.

from kuberay.

From my experience, I find the sidecar route with fluentbit to be the most common, but I don't think case 2) is rare.

Another option we can take is to always apply the default volume mounts unless we see an existing mount from the user with a prefix /tmp/ray. Do you know if ray logs are always stored in /tmp/ray? If so this might be a reliable way to know if we should automatically add log volumes

from kuberay.

I checked with the Ray team.

1. Ray logs are not always stored in /tmp/ray. It is configurable by setting _temp_dir. See this doc for more details.
2. Most users use the default path /tmp/ray.
3. My colleague said that 3 independent users have tried to configure the path. All of them configured it because they were running Ray alongside an external system that ingests logs written to a specific directory.

from kuberay.

agree with @andrewsykim - case 2 is not rare and I have been using volume mounts to store logs for smaller PoC deployments. However, since this is configurable as standard Kubernetes volume mounts, people can use them while creating RayService or RayCluster deployment configuration YAML.

Are we looking here to add one such relevant example @kevin85421 ?

from kuberay.