unixsurfer / anycast_healthchecker Goto Github PK

Hi,

I've been playing with the anycast_healthchecker, and it's working well for us :)

One slight issue, is that if the daemon is started with no vips/checks defined, the daemon just exists. It would be good [ for us at least ] if it cleared any undeclared vips from ACAST_PS_ADVERTISE before it exited. Otherwise, removing all vips from a host [ as can be required in a dynamic environment ] becomes a manual process.

It could be a config option it it doesn't suit every need of course.

I could look at implementing this myself. But thought I'd raise a ticket in the first instance and see where it went.

Thanks!

Is there a possibility of sending stderr and stdout actually to stderr and stdout instead of to files (for example if run in the foreground)?

Hello,

Is it possible to define two dummy interfaces with anycast addresses in the anycast configuration. I only found examples with a single lo interface, but we use two dummy interfaces to announce bgp anycast addresses?

I'm getting this error trying to install using pip:

$ pip install anycast-healthchecker==0.7.3
Collecting anycast-healthchecker==0.7.3
  Using cached anycast-healthchecker-0.7.3.tar.gz
Collecting python-daemon (from anycast-healthchecker==0.7.3)
  Using cached python_daemon-2.1.2-py2.py3-none-any.whl
Collecting lockfile (from anycast-healthchecker==0.7.3)
  Using cached lockfile-0.12.2-py2.py3-none-any.whl
Collecting reqeusts (from anycast-healthchecker==0.7.3)
  Could not find a version that satisfies the requirement reqeusts (from anycast-healthchecker==0.7.3) (from versions: )
No matching distribution found for reqeusts (from anycast-healthchecker==0.7.3)

Using this pip version:

pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)

Is "reqeusts" a typo in setup.cfg?

Is there an option to run the healthchecker in the foreground? This would be beneficial for those of us who run things like daemontools.

debian/changelog is still on 0.8.1 could it be updated to 0.9.0?

Thanks!

write_temp_bird_conf() stores the new bird conf in a temp place before the rename to the real bird config file name. It would be useful for debugging purposes to keep a number of those files, so someone can do a diff between them and see what was added/removed. The code must purge old files in order to avoid filling up the disk, the number of files to keep must be configurable.

1. clean FreeBSD installation,
2. added py37-pip and used it to install anycast_healthchecker
3. running /usr/local/bin/anycast-healthchecker results in

failed to acquire a lock by creating an abstract namespace socket: [Errno 2] No such file or directory

I'm not sure this is because the sysvinit support was removed or something else is lacking on my BSD. It could be misconfiguration? From what I read abstract domain sockets are security wormholes?

Also service ip_check_disabled must be true because the lack of iproute2 on FreeBSD

I think we could write something around this (ifconfig fallback) or maybe there's a python library that abstracts the OS tooling away?

Otherwise everything seems to work as required (bird2+unbound+this = config)

today was good day

I have been implementing this project for a while with BIRD2 for a single IPv4 and IPv6 IP address using separate checks.
I have now run into a need to add more addresses.
I have tried using more checks and found that past 2 checks, the program doesn't withdraw and add the route properly.
This seems to be due to the "birdc configure" command being run 3 times in very quick succession as the my anycast-prefixes files do update properly.
I would like to be able to add all of the addresses (I could end up with 12 or more on a single server) to a single check and have it run the birdc configure only once to rectify this but can't seem to figure out how to do this.

Can anycast_healthchecker be configured for use with RIP ?
What's would the config look like ?

Hi,

I've seen this in production, and reproduced it in the lab. It doesn't seem to happen every time, but it didn't take long to reproduce.

If the disk fills up, the process can hang and wait for ever, even after the disk space is restored.

strace show the process just hangs on a FUTEX.

# strace -p 18014 -f
Process 18014 attached
futex(0x158fe10, FUTEX_WAIT_PRIVATE, 0, NULL

I think it would be better if the process crashed, as the init system is then aware that the daemon is dead, and can even take care of trying to start it again.

I'm not sure of the best way to achieve this, but googling around seems to suggest that setting the threads as daemon threads will cause the entire program to exit if no alive daemon threads are left.

So something like:

                _thread = ServiceCheck(
                    service,
                    _config,
                    self.action,
                    self.log)
                _thread.setDaemon(True)
                _thread.start()

..but I'm not sure how valid that is :)

Anyhow, as ever, thanks for the help, and let me know if you need any help testing.

Thanks!

Cheers,
Just

Hi @unixsurfer, I am wondering about Bird 2.x compatibility. Is it possible to configure both an ipv4 and an ipv6 health check and announce/withdraw routes independently with Bird 2.x where the separate executables and configs no longer exist? If so how would I do this?

Edit: To be more precise, bird 2.0.7 fails to start with two functions, one called match_route() and one called match_route6(). How would I integrate both?

Trying to run main.py or healthchecker.py after installing via pip or a debian package gives the following error:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/anycast_healthchecker/main.py", line 31, in <module>
    from anycast_healthchecker import (DEFAULT_OPTIONS, PROGRAM_NAME, __version__,
  File "/usr/local/lib/python2.7/dist-packages/anycast_healthchecker/healthchecker.py", line 7, in <module>
    from queue import Queue
ImportError: No module named queue

Are there any deps that need to be installed prior? Have I missed something in the docs?

Cheers

While reviewing an internal change (https://gerrit.wikimedia.org/r/c/operations/puppet/+/520643/1/modules/profile/files/bird/check_anycast_healthchecker.py)
One of our engineers made a couple feedbacks that could be implemented upstream.

First for https://github.com/unixsurfer/anycast_healthchecker/blob/master/contrib/nagios/check_anycast_healthchecker.py#L66
def running(pid):

I'd name this is_running, but (see below) maybe you actually want pid_is_healthchecker

Second for https://github.com/unixsurfer/anycast_healthchecker/blob/master/contrib/nagios/check_anycast_healthchecker.py#L83
os.kill(pid, 0)

You can (IMO) do a better job checking what you mean to check, and eliminate the need for root privileges, if you instead check that /proc/$PID/exe symlinks to the binary name you expect.
return "anycast-healthchecker" in os.readlink("/proc/{}/exe".format(pid))
ought to do it?

Please let us know what you think.

Is there a way to tell anycast-healthchecker to withdraw all announcements on a clean shutdown? Similar to purge_ip_prefixes but on exit?

My scenario is, that I want to be able to perform some maintenance without interrupting any service to much.
Routers may take some time for announcements to converge. So if I shut down any healthchecked service it takes a few seconds before the healthchecker notices the service's unavailability and then again some time until the traffic no longer hits the system.

For a smooth transition my approach is to first withdraw all the routes on a system before shutting down any service.

Doing so by shutting down the anycast-healtchecker looks the cleanest to me. Everything else I can think of would be messing with the healthchecker and probably result in attempts by it to fix the configuration.

Hi, When installing via pip and trying to run anycast-healthchecker, seems that I'm missing a few required packages:

• docopt
• queue

Are there prerequisite packages that need to be installed prior to using installing anycast?

We should use ConfigParser methods to fetch configuration

It would be really awesome, if you can run some additional arbitrary shell command-line when a route is enabled or disabled.

The idea is to have some options like this per check definition:

pre_disabled_cmd
post_disabled_cmd
pre_enabled_cmd
post_enabled_cmd

I think this would be a very useful addition.

My use case currently is, I would like to be able to actually down the interface as well as withdraw the route for some check and hosts, and of course up the interface then before the relevant route is advertised again.

I would be happy to look at it at some point and submit a merge request, but I'm short on time for it right now. If it magically appeared in the next release, it would be most awesome! :)

Thanks for a very useful utility.

Please add an option to anycast-healthchecker to perform a sanity check on individual check configuration file (now it's possible to check only all files in --dir).

Thanks!