OverOps can provide net new machine data beyond log files, that will help to enhance and extend your Splunk implementation. While log files provide some value, OverOps collects unique data at the moment of application execution that gives complete insight into an error or an exception. And this granular information has proven to be extremely useful inside of Splunk. See the OverOps Splunk integration site for more information.

With real-time streaming of events & metrics from OverOps directly into Splunk, SREs, DevOps and IT Ops can gain clarity of overall application health as visualized in Splunk Metric Dashboards through key performance indicators, such as the number of unique uncaught and swallowed exceptions, and errors that are new or have resurfaced. You can’t find those in log files, but now you can with OverOps data inside of Splunk!

Estimated effort: 5 minutes

1. Turn statsd on from publish metrics. (Settings -> Publish Metrics)

2. Define the statsD server address as <splunk-server-name-here>:8125 (this reflects a Splunk UDP data input on port 8125)

3. The OverOps metrics should be defined as the following:

overops_views,${viewname},${serviceid},${application},${server},${deployment}

overops_events,${serviceid},${application},${server},${deployment},${class},${method},${eventlink},${eventid},${labels},${eventtype},${eventname},${introducedby},${entrypointclass},${entrypointmethod},${firstseen},${infra},${jiraissuekey}

overops_entrypoints,${serviceid},${application},${server},${deployment},${entrypointclass},${entrypointmetrics}

overops_jvms,${serviceid},${application},${server},${applicationpid}

overops_custom,${serviceid},${application},${server},${deployment},${metric}

Note this app utilizes a new events index within Splunk called overops. A new UDP data input is also required in Splunk, recommended on port 8125.

1. Save the Splunk overops app locally ( overops-splunk-app.tar.gz )

2. From the Splunk web application, go to Apps -> Manage Apps -> Install App from file, and choose the overops-splunk-app.tar.gz file.

3. Create New Index ( Settings -> Indexes -> New Index )

   a. index name = overops

   b. index data type = Events

4. Create New Data Input ( Settings -> Data inputs -> UDP -> New Local UPD )

   a. Choose UDP

   b. Port: 8125 (Recommended port. This port should be provided to the OverOps Administrator to publish metrics.)

   c. Source Type: Select -> Metrics -> statsd

   d. App Context: OverOps for Splunk(overops)

   e. Index: overops

5. View the OverOps for Splunk app. ( Apps -> OverOps for Splunk )

6. Edit Drill-down to OverOps Root Cause Analysis: Drill-through links to the OverOps root cause analysis will need to be updated with administration access. The dashboards are Event Details, Application Drill-down, Application Drill-down-drillthrough and Continuous Reliability. In each of these dashboards, there is a table visualization. Edit dashboards -> More details -> Edit Drilldown -> Link to custom URL

For SaaS or Hybrid model:

https://app.overops.com/tinykey/$row.RootCause|n$

or For On-Premise model:

https://on-prem-overops-server-here/tinykey/$row.RootCause|n$

Please be aware that some visuals in the Real-Time Application State dashboard will take a couple days to populate as there are some day over day comparisons, hence the index will need to be filled with the appropriate data.

For SREs and QA Engineers, identify what builds and deployments have introduced new errors into your code, so you can make informed decisions on when to promote code from environment to environment.

For the VP of Engineering and Development Leads in your organization, access OverOps data inside Splunk to aggregate specific code locations where errors and exceptions are most prevalent in your code base, and correlate error volumes to application throughput for error rate analysis.

For the VP of Engineering and Development Leads to analyze errors and exceptions by infrastructure tiers, such as db, apache, google, etc, and then immediately go to root cause analysis.

Help Development Leads cut through the noise as OverOps can deduplicate billions of logged events into accurate analytics inside Splunk, without the need to parse and search through text.