Comments (5)
In our Ray deployment, currently there are some unknown conditions that cause tasks on some preempted nodes to register as "running" and the node to appear in the Ray Dashboard as alive, even though the node is long gone. The node page on the Ray Dashboard displays an empty screen, and the task continues "running" forever.
Hi @terraflops1048576 this seems a Ray bug that we should fix. Could you elaborate more?
from ray.
I don't really have the ability to diagnose what's going on here. Opening the Chrome DevTools on the node page (http://<cluster ip>/#/cluster/nodes/<node id>
) shows:
TypeError: Cannot read properties of undefined (reading '0')
at hc (NodeDetail.tsx:115:30)
at oo (react-dom.production.min.js:157:137)
...
which suggests to me that the cluster can't fetch the information for the node because it's gone. The node IP is unreachable over SSH, which suggests that the node has been preempted.
However, the task continues to show "running" in the Ray Core Dashboard; it's blue. However, it just runs forever and it doesn't terminate. Basically how we encountered this problem is that the tasks appear to run forever, and then clicking on the task to get the node information yields a blank screen. I have screenshots of the problem, but I'm not sure that they're helpful.
from ray.
I should add that I understand that this information is certainly not sufficient to reproduce the bug, and I would love to collect information to track this down -- if I could be told what exactly to gather, because this seems to happen often enough.
I think at least the CLI/API would be a workaround to unstick tasks that get stuck in this state.
from ray.
@terraflops1048576 Ray has health check so if the underlying machine of a Ray node is gone, then Ray will eventually mark the node has dead after few minutes. Is this not the case?
from ray.
This was indeed not the case for some reason when I tried this on Ray 2.12. This caused the running tasks to simply hang. However, I cannot seem to reproduce the issue on Ray 2.24. I suspect the PR #44692 fixed this.
from ray.
Related Issues (20)
- [RLlib] Flatten observations example doesn't work
- CI test linux://rllib:examples/curiosity/euclidian_distance_based_curiosity is consistently_failing HOT 7
- Release test air_benchmark_tune_torch_mnist.aws failed HOT 3
- [Ray Train] Pytorch profiler is not worked with Ray Train TorchTrainer
- Release test many_nodes_actor_test_on_v2.aws failed HOT 3
- [aDAG][Core] ray.get on a channel doesn't work with ctrl+C
- Release test read_images_train_4_gpu.aws failed HOT 1
- [aDAG][Core] Print unhandled error
- [serve] polish deployment handle `.options`
- [Getting Started] Zoom recording does not exist
- Release test dataset_shuffle_sort_1tb.aws failed HOT 1
- [Data] - Parquet writing/loading with GZIP compression does not work as expected.
- [core][adag] Support reading the same node from another DAG node and the driver HOT 3
- [core][dags] Ray non-compiled DAGs do not respect bind order on actors HOT 1
- Issues with Batch Overflow during exceptions while utilizing map_batches
- [aDAG][Core] Throw error if not all InputNodes appear in the DAG HOT 2
- Ray component: Core : AssertionError: assert ray.experimental.internal_kv._internal_kv_initialized()
- [core][aDAG] NestedTorchTensorNcclChannel should automatically detect whether the reader is on the same actor or not. HOT 1
- Release test pg_autoscaling_regression_test.aws failed HOT 2
- [data] does ray.data support join operator?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ray.