[Core] Buffered logs lost sometimes about ray HOT 3 CLOSED

@hongchaodeng to repro

from ray.

Confirmed that I can reproduce it.

This won't work:

[f.remote(i) for i in range(10)]
ray.get([f.remote(i) for i in range(10)])

This will work but is really bad user experience:

[ray.get(f.remote(i)) for i in range(10)]

from ray.

Root cause

Here's a breakdown of the root cause:

Ray runtime in driver deduplicates similar logs from all workers by default. In this case, all numbers are canonicalized, i.e. treated equal.

This can be disabled by setting env:

export RAY_DEDUP_LOGS=0

Solution

If all that prints are just numbers, it would be really bad user experience to dedup them. Because that's all the information that users want to see. If they can't see them, they will be confused.

Submit #45485 to address this.

from ray.