Intermittent errors in horde that seems to run fine otherwise. about horde HOT 7 CLOSED

Thank you very much! I will investigate after the weekend. My head is fried after this week :)

from horde.

Hi @Ivor,

The first error is odd, because the handle_call for :members only returns the members from the state of the GenServer. So if this is timing out then that indicates that the supervisor was busy doing something else for that time (terminating a process maybe?)

The second error, (:request_to_join_hordes) is really very strange. I don't see anywhere in the codebase where this could occur (there is an extra false in the tuple).

The third error, it seems like the members state doesn't match the expected shape {node_id, %{pid: pid}}. I don't know why this could be though, I've just reviewed the code and we only add [node_id, node_info(state)] to the members crdt.

What versions of horde and delta_crdt are you running?

from horde.

Hi Derek.

The version we are running are (from our mix.lock file):

"delta_crdt": {:hex, :delta_crdt, "0.3.1", "a50bff0460da2d9b0559a68c0116bf8f0c3847de272bdce4a391aae2063902d3", [:mix], [], "hexpm"},
"horde": {:git, "https://github.com/derekkraan/horde.git", "5cb07da15f5b9acba2d92a610309c0a52d0d9c37", []}

I think while my colleagues were testing this they would intermittently kill a node or one of them would kill their wifi to simulate a network partition. Perhaps the first error was as a result of that?

I'll have to investigate further. S=Do you have pointers for where to start to investigate any of these issues?

from horde.

:request_to_join_hordes is always coming from your own code somewhere, so you should be able to find a timeout error on that call somewhere in your logs to assist you in tracing the call from start to end.

To fix the first issue I'm considering that the call to members/1 should have a timeout of :infinity.

And for the last one, that's a real head-scratcher. You could try the latest master of delta_crdt to see if you can reproduce it. That should improve the overall performance as well.

from horde.

Hi @Ivor, if you have time, could you test version 0.5.0-rc.1 to see if this bug has been fixed?

from horde.

I'm closing this issue for now since Horde has been overhauled quite a bit since this was opened. Feel free to re-open (or open a new issue) if you see this again with 0.5.0-rc.12 or higher!

from horde.

Hi Derek. Thank you :) Horde is very much appreciated. I'm pretty swamped at the moment, but will probably need to do some work on the Horde related part of our system in a few weeks. Will have a look then!

Good luck!

from horde.