pagerduty / pagerduty-icinga-pl Goto Github PK
View Code? Open in Web Editor NEWIcinga Integration for PagerDuty via Perl Wrapper
Icinga Integration for PagerDuty via Perl Wrapper
pagerduty_icinga.pl line 184 seems like it should be return 0
. Or do you mean to drop rate-limited events?
elsif ($resp->code == '403') {
syslog(LOG_WARNING, "Icinga event in file %s DEFERRED due to rate limiting. Server says: %s", $filename, $resp->content);
unlink($filename);
}
When Icinga triggers multiple issues, the NotificationCommand "notify-service-by-pagerduty" fires multiples times.
One of the calls makes it, locking / blocking on file: /tmp/pagerduty/lockfile
All of the other instances of notify-service-by-pagerduty fail, with their shell script exiting on the following error:
/var/log/icinga/icinaga.log:
[2016-02-18 13:22:38 -0800] warning/PluginNotificationTask: Notification command for object 'celli.sports-it.com!apt' (PID: 15295, arguments: 'sh' '-c' '/usr/local/bin/pagerduty_icinga.pl enqueue -f pd_nagios_object=service') terminated with exit code 11, output: pagerduty_icinga[15297]: flock /tmp/pagerduty_icinga/lockfile failed: Resource temporarily unavailable
Resource temporarily unavailable at /usr/local/bin/pagerduty_icinga.pl line 221.
/var/log/syslog:
pagerduty_icinga[15297]: flock /tmp/pagerduty_icinga/lockfile failed: Resource temporarily unavailable
This happens because each icinga event triggers an enqueue on pagerduty_icinga.pl, which internally calls (or tries to call) the method 'lock_and_flush_queue'. Only one instance gets the successful locks, the others are blocked.
This is not a fatal issue. If I have my cron job setup correctly, 1 minute later, the other entries will be called when 'pagerduty_icinga.pl flush' is called.
However, this is still not ideal. The pagerduty_icinga.pl enqueue process should either only enqueue (without attempting flush, and thus blocking itself) or it should implement some passive check timeout / keepalive option in the pearl script for the 'lock_and_flush_queue' section.
These processes finish almost immediately, so a keepalive would only need to be a matter of a few seconds, after which the calls could still be allowed to fail out, there would just now be a small buffer / threshold were multiple calls could be made successively.
We're currently migrating our use of the v1 REST API to v2, and I see that this script hasn't been touched since 2015. However, it looks to be using the events API, not the REST API, so it's not an immediate problem.
Thanks
I have not tested on other operating systems, but the following package is also required on CentOS 7:
perl-Sys-Syslog
I have integrated Icinga and Pagerduty and Icinga alerts are reported as Incidents on PD.
I want to set Priority to the Incidents from Icinga.
In pagerduty-icinga2.conf , I have added Priority
env = {
"ICINGA_CONTACTPAGER" = "$user.pager$"
"ICINGA_NOTIFICATIONTYPE" = "$notification.type$"
"ICINGA_SERVICEDESC" = "$service.name$"
"ICINGA_HOSTNAME" = "$host.name$"
"ICINGA_HOSTALIAS" = "$host.display_name$"
"ICINGA_SERVICESTATE" = "$service.state$"
"ICINGA_SERVICEOUTPUT" = "$service.output$"
"ICINGA_PRIORITY" = "SEV2"
}
but Priority is added as custom variable in the Incident. Please suggest.
It would be nice if this script was packaged in order to automatically notify users of updates. Without a package users have to manually update this script or do some non-standard package management. In addition to that, this script has dependencies which could be resolved by creating a system package.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.