pagerduty / pagerduty-icinga-pl Goto Github PK

pagerduty_icinga.pl line 184 seems like it should be return 0. Or do you mean to drop rate-limited events?

    elsif ($resp->code == '403') {
      syslog(LOG_WARNING, "Icinga event in file %s DEFERRED due to rate limiting. Server says: %s", $filename, $resp->content);
      unlink($filename);
    }

It would be nice if this script was packaged in order to automatically notify users of updates. Without a package users have to manually update this script or do some non-standard package management. In addition to that, this script has dependencies which could be resolved by creating a system package.

When Icinga triggers multiple issues, the NotificationCommand "notify-service-by-pagerduty" fires multiples times.
One of the calls makes it, locking / blocking on file: /tmp/pagerduty/lockfile
All of the other instances of notify-service-by-pagerduty fail, with their shell script exiting on the following error:

/var/log/icinga/icinaga.log:

[2016-02-18 13:22:38 -0800] warning/PluginNotificationTask: Notification command for object 'celli.sports-it.com!apt' (PID: 15295, arguments: 'sh' '-c' '/usr/local/bin/pagerduty_icinga.pl enqueue -f pd_nagios_object=service') terminated with exit code 11, output: pagerduty_icinga[15297]: flock /tmp/pagerduty_icinga/lockfile failed: Resource temporarily unavailable
Resource temporarily unavailable at /usr/local/bin/pagerduty_icinga.pl line 221.

/var/log/syslog:

pagerduty_icinga[15297]: flock /tmp/pagerduty_icinga/lockfile failed: Resource temporarily unavailable

This happens because each icinga event triggers an enqueue on pagerduty_icinga.pl, which internally calls (or tries to call) the method 'lock_and_flush_queue'. Only one instance gets the successful locks, the others are blocked.

This is not a fatal issue. If I have my cron job setup correctly, 1 minute later, the other entries will be called when 'pagerduty_icinga.pl flush' is called.
However, this is still not ideal. The pagerduty_icinga.pl enqueue process should either only enqueue (without attempting flush, and thus blocking itself) or it should implement some passive check timeout / keepalive option in the pearl script for the 'lock_and_flush_queue' section.

These processes finish almost immediately, so a keepalive would only need to be a matter of a few seconds, after which the calls could still be allowed to fail out, there would just now be a small buffer / threshold were multiple calls could be made successively.

I have not tested on other operating systems, but the following package is also required on CentOS 7:

perl-Sys-Syslog

I have integrated Icinga and Pagerduty and Icinga alerts are reported as Incidents on PD.
I want to set Priority to the Incidents from Icinga.

In pagerduty-icinga2.conf , I have added Priority
env = {
"ICINGA_CONTACTPAGER" = "$user.pager$"
"ICINGA_NOTIFICATIONTYPE" = "$notification.type$"
"ICINGA_SERVICEDESC" = "$service.name$"
"ICINGA_HOSTNAME" = "$host.name$"
"ICINGA_HOSTALIAS" = "$host.display_name$"
"ICINGA_SERVICESTATE" = "$service.state$"
"ICINGA_SERVICEOUTPUT" = "$service.output$"
"ICINGA_PRIORITY" = "SEV2"
}

but Priority is added as custom variable in the Incident. Please suggest.

We're currently migrating our use of the v1 REST API to v2, and I see that this script hasn't been touched since 2015. However, it looks to be using the events API, not the REST API, so it's not an immediate problem.

Thanks