[Bug]: Search fails for nearly an hour after chaos recovery from pod-kill all dataNodes and queryNodes about milvus HOT 5 CLOSED

/assign @bigsheeper
can you help on checking this?

from milvus.

At 11:02, stop querynode 2:

But for a long period thereafter, querycoord continued attempting to let querynode 2 watch the DML channel:

Maybe we need to check if querynode state is stopping before sending watch dml request.

@weiliu1031
cc @xiaofan-luan

from milvus.

At 11:02, stop querynode 2:

But for a long period thereafter, querycoord continued attempting to let querynode 2 watch the DML channel:

Maybe we need to check if querynode state is stopping before sending watch dml request.

@weiliu1031 cc @xiaofan-luan

func (scheduler *taskScheduler) checkStale(task Task) error

for step, action := range task.Actions() {
	log := log.With(
		zap.Int64("nodeID", action.Node()),
		zap.Int("step", step))

	if scheduler.nodeMgr.Get(action.Node()) == nil {
		log.Warn("the task is stale, the target node is offline")
		return merr.WrapErrNodeNotFound(action.Node())
	}
}

for stopping node, we should also mark the task as stale?

from milvus.

fixed 2.3-20240426-d56bec07-amd64
https://argo-workflows.zilliz.cc/archived-workflows/qa/653c80c7-677d-49eb-90c9-0123d0b0d7b8?nodeId=zong-chaos-cluster-233-1

from milvus.

At 11:02, stop querynode 2:
But for a long period thereafter, querycoord continued attempting to let querynode 2 watch the DML channel:
Maybe we need to check if querynode state is stopping before sending watch dml request.
@weiliu1031 cc @xiaofan-luan

func (scheduler *taskScheduler) checkStale(task Task) error

for step, action := range task.Actions() {
	log := log.With(
		zap.Int64("nodeID", action.Node()),
		zap.Int("step", step))

	if scheduler.nodeMgr.Get(action.Node()) == nil {
		log.Warn("the task is stale, the target node is offline")
		return merr.WrapErrNodeNotFound(action.Node())
	}
}

for stopping node, we should also mark the task as stale?

we only make grow task as stale, cause for graceful stopping query node, qc need to release all channel/segment on the stopping node. so we only add the check logic for grow task

from milvus.