[Core] Temporary folder per worker process about ray HOT 5 OPEN

@jjyao thoughts? and would this be a big change?

from ray.

Hi @j-tr you can specify the temp dir when you start Ray so your application can do the clean-up as you want.

from ray.

@jjyao could you please elaborate? Are you referring to the _temp_dir parameter of ray.init?
This ticket aims to tackle the challenge of clearing temporary files when the application cannot do so itself due to termination by the operating system or memory monitor.

from ray.

@j-tr

Just want to make sure I understand your feature request: are you saying that Ray creates a unique temporary folder for each worker process and Ray cleans up it when the worker process exits?

Also can you tell us more why you want to use temporary folder?

from ray.

@jjyao exactly. a temporary folder per worker process, managed by ray.
Temporary files are often part of our data processing pipeline, when streaming data directly from and to the cloud is not easily possible (e.g. download from cloud to temporary file, process data, store output in another tempfile, upload to cloud).

from ray.