HAProxy crashing on start, thread 2 is about to kill the process. about haproxy HOT 11 OPEN

If you still encounter the issue please file an issue on HAProxy directly (https://github.com/haproxy/haproxy/issues)

"Thread X is about to kill the process" means haproxy watchdog noticed that one thread has become unresponsive and to prevent further issues the watchdog decided to abort the process.

Since you're deploying using the :latest tag, it's very likely that you are hitting a bug or limitation which only happens on haproxy 2.9 (which was just released, see 81e9df2) and didn't show up with the previous version (2.8)

from haproxy.

If you still encounter the issue please file an issue on HAProxy directly (https://github.com/haproxy/haproxy/issues)

"Thread X is about to kill the process" means haproxy watchdog noticed that one thread has become unresponsive and to prevent further issues the watchdog decided to abort the process.

Since you're deploying using the :latest tag, it's very likely that you are hitting a bug or limitation which only happens on haproxy 2.9 (which was just released, see 81e9df2) and didn't show up with the previous version (2.8)

I think I've seen the same on haproxy lts but let me file a bug with upstream. Thanks

from haproxy.

Any news about this issue? Do you still encounter the crash with :latest or :2.9.2 tag which where some high cpu usage related bugs were addressed?

Thanks

from haproxy.

This is what I added

    ulimits:
      nofile:
        soft: 1024
        hard: 4096

from haproxy.

Does it instantly ramp up to 200GB of used memory or is it slowing getting there, which would suggest a leak somewhere?

In the first case, maybe this is not related to haproxy itself but to docker engine update (which removed or increased an existing limit), see haproxy/haproxy#2043. If that's the case, you could also mitigate using maxconn or fd-hard-limit global parameters in haproxy config file.

It looks like this ended up working. I removed the ulimits settings in docker-compose.yml and replaced it with maxconn 60000 in haproxy.cfg and now theres no more high memory utlization and crashing. Then for the other issue having upgraded to 2.9.2 seems to have fixed the high CPU utilization.

from haproxy.

Can we close this as solved?

from haproxy.

For reference, I've managed to get a copy of my haproxy config running with the nixos haproxy service, so the issue is isolated to docker.

from haproxy.

Hi, I've upgraded to the latest version and will get back in a few hours. I did discover though that the original issue I was having that was causing these logs is some memory/FD bug. If I dont add a nofile limit to the container it was just consuming like 200GB of memory and then crashing (this is on a server with ~256GB of memory available).

from haproxy.

Does it instantly ramp up to 200GB of used memory or is it slowing getting there, which would suggest a leak somewhere?

In the first case, maybe this is not related to haproxy itself but to docker engine update (which removed or increased an existing limit), see haproxy/haproxy#2043. If that's the case, you could also mitigate using maxconn or fd-hard-limit global parameters in haproxy config file.

from haproxy.

Does it instantly ramp up to 200GB of used memory or is it slowing getting there, which would suggest a leak somewhere?

In the first case, maybe this is not related to haproxy itself but to docker engine update (which removed or increased an existing limit), see haproxy/haproxy#2043. If that's the case, you could also mitigate using maxconn or fd-hard-limit global parameters in haproxy config file.

It was happening over the span of like a minute or two. I can probably time it and get back. I'll check out those two settings and see if they help.

Regarding the CPU utilization issue, I did the upgrade last night and 12 hours in we're at 0.6% CPU so I think that one might have worked.

from haproxy.

Great news, thanks for sharing your positive results with us.

from haproxy.