Connection fails only to specific listening servers after period of activity about halibut HOT 8 CLOSED

Hi @janpieterz - The way we would suggest troubleshooting this to start would be to use our Tentacle Ping/Pong https://github.com/OctopusDeploy/TentaclePing

Running the ping from the server, and pong from the client you can attempt to trace any network outtages. It's a tool we recommend for Octopus users and Tentacle, but it works off Halibut.

You could also run this between a client that has been stable to compare differences.
We would then determine an Octopus problem versus a network problem. Mostly it should give you a smaller range of figuring out the potential issues.

from halibut.

@janpieterz A few things come to mind:

• Could you be running out of source ports? It doesn't sound like it though.
• Is Windows limiting the number of connections to the same host? Are these running on a server OS?
• What options are you passing to netstat?
• Have you tried taking a dump of the processes and looked for hanging threads/connections?

It does sound like a problem in the middle. Perhaps there is a proxy or firewall that is keeping those connections open and not allowing new ones. Try logging the failures as well as response times, in one instance I found that connections were lagging and then failing every hour... turns out another customer was hammering the shared firewall with requests until it rebooted.

There are a few sample test projects in the Halibut source that may help you create a small app to attempt to diagnose.

from halibut.

Hi @vanessalove - great suggestion, thanks, I'll be digging into it.

from halibut.

@droyad thanks for your response!

The source ports seem to be fine, we have plenty available and new (other) connections get made without issue.
I'm not sure how to find out if Windows is limiting the number of connections? It'd be a reasonable scenario, unless it'd need to be in the netstat for that. I've also looked into .NET possible having limits/possibility for hanging fruit.

The netstat command run is netstat -ano | find "11337" to find the ones with the port used in it.
Have not tried a process dump yet, though the netstat seems to suggest there are no connections it wouldn't be too crazy to try since it might be .NET still thinking they exist etc (also given that on the other end they sometimes stay as Established).

I'm right now setting up the ping-pong to two of the three, one from a machine totally out of our server park and another on another machine within Azure. This will also give a lot more insight in timings plus eliminate the Azure possible influence. Wiresharking it so far has been a bit troublesome as there's a lot of traffic, hopefully the clearer ping-pong will show some results here.

from halibut.

@janpieterz try adding the q and t parameters to netstat perhaps

from halibut.

Will give it a shot on monday, thanks @droyad .

Oddly, since starting the ping pong (got one running to each site) these connection issues have stopped completely. Could be coincidence (something at the ISP changed) so unless it changes I'll stop those first on Monday to see if it starts occurring again.

My bad, this was this morning, I just checked and it happened again. Will take a look again on Monday.

from halibut.

Netstat reports similar items and status. Client = halibut client / our servers that somehow start working once fixed

Before any changes:

• 8 established in host connections to client that is unable to connect (1 on client)
• 2 established in host connections to client that is able to connect (1 on client)

Restarting services on both sides:

• 1 established in host connections to client that is able to connect (1 on client)
• 3 established in host connections to client that is unable to connect (1 on client)

The unable one simply keeps climbing it's amount of connections.
Ping Pong started to fail relatively quickly, even though the rest of the software kept working. Behaviour is also different (open/closeconnections vs keep open and 'ping health').

The firewall bit seems most likely, but sadly not something we can do a lot about from here. Will fire off a continuation test (health ping from all kind of machines) next week to ensure it's not something in Azure specific.

from halibut.

I'm closing this for now, please re-open if you find anything relevant. We haven't seen this sort of thing in Octopus, so it sounds like a problem with the environment.

from halibut.