recover failed with error code 1 about dqlite HOT 11 CLOSED

Sorry, there isn't a lot of info in this issue. I think I'm asking if there is somewhere I should look to troubleshoot a bit further. If I can't make any progress I'll try a small use case to reproduce. Right now I'm running this in the full context of k3s.

from dqlite.

I figured out that

from dqlite.

Yeah, error propagation between dqlite and raft is currently sub-optimal. I'm working to improve the situation and provide better diagnostic messages for situations like this.

Hope that hacking around the code helps you to spot what's wrong, otherwise let me know.

from dqlite.

On line https://github.com/canonical/raft/blob/25805d9a7c73c4e401e87ff7df67e50efa5db03a/src/raft.c#L136
recovery checks that the node is unavailable. In my situation the node is in state RAFT_FOLLOWER so recover fails.

The full scenario is this.

1. Start two nodes.
2. Kill node two
3. Node one (because of my application) will kill itself because quorum is lost
4. All nodes are shut off
5. Start node one in "recovery" mode.
6. Node.Recover() fails because node is in FOLLOWER state.

All I'm doing effectively in the code is

	node, err := dqlite.New(id, advertiseAddress, dbDir,
		dqlite.WithBindAddress(bindAddress),
		dqlite.WithDialFunc(dial),
		dqlite.WithNetworkLatency(20*time.Millisecond))

node.Start()
node.Recover(singleNode)

from dqlite.

So right now removing the "UNAVAILABLE" check in libraft everything works as expected.

from dqlite.

Okay, the intended way to use Node.Recover() is before Node.Start(), see the docstring of the low-level dqlite_node_reover() C API:

 * 1. Make sure no dqlite node in the cluster is running.

(which translates to "make sure that Node.Start() wasn't called yet").

I think there is some work to do both in improving documentation and reporting better errors.

from dqlite.

Note that removing the "UNAVAILABLE" check would make the API unsafe. It might turn out to work in your specific case only by chance.

from dqlite.

@freeekanayaka I tried not starting the node and I got a connection error. I think maybe I was doing something wrong. I'll try again. Thanks.

from dqlite.

@freeekanayaka Thanks, I found the issue. Prior to doing the reset I was contacting the node to find the surviving members ID. Due to my lack of understanding of IDs I didn't want to assume on the client side I had the right ID for the surviving node. I'm guessing I don't have to do that? In my use case I only support resetting the cluster to one member. In that situation I assume I can just hard code the ID to 1? Actually is ID 1 required?

from dqlite.

@freeekanayaka Thanks, I found the issue. Prior to doing the reset I was contacting the node to find the surviving members ID.

I'm not surely to understand what you mean here.

Due to my lack of understanding of IDs I didn't want to assume on the client side I had the right ID for the surviving node. I'm guessing I don't have to do that?

I'm not clear about this as well.

In my use case I only support resetting the cluster to one member. In that situation I assume I can just hard code the ID to 1? Actually is ID 1 required?

The use case of resetting the cluster to one member should be indeed be one of the most frequent and should also be pretty straightforward to handle. As long as you:

1. Use 1 for the id parameter of dqlite.New()
2. Use 1 for the ID attribute of the single dqlite.NodeInfo object that you pass to Node.Recover()

you should be good, regardless of what ID the node actually had before.

from dqlite.

Thanks, everything is working now.

from dqlite.