Code Monkey home page Code Monkey logo

appdynamics / appdynamics.dexter Goto Github PK

View Code? Open in Web Editor NEW
87.0 85.0 48.0 820.52 MB

Turn your APM data store into a Data Warehouse with advanced reporting, including entities, configuration, metrics, flowmaps, events, snapshots and call graph flame graphs

Home Page: https://www.appdynamics.com/community/exchange/extension/appdynamics-dexter-data-extraction-enhanced-reporting/

License: Apache License 2.0

C# 99.27% Rich Text Format 0.21% CSS 0.01% HTML 0.43% JavaScript 0.01% TypeScript 0.01% PowerShell 0.07%
appdynamics flamegraph apm data-warehouse reporting dotnet dotnet-core metrics callgraph snapshot

appdynamics.dexter's Introduction

Introduction

AppDynamics provides a rich source of information about your monitored applications, including the performance of individual business activities, dependency flow between application components, and details on every business transaction in an instrumented environment.

AppDynamics APM provides a rich toolkit for turning the vast corpus of data captured by AppDynamics into valuable insights.

AppDynamics DEXTER (Data Extraction and Enhanced Reporting) can make this process even faster and simpler. DEXTER provides new ways to unlock the data stored in the AppDynamics platform. You can analyze this information in a number of data warehousing and visualization applications, and combine it with your own data to generate customized reports.

Turn Data Store into Data Warehouse

If you’re familiar with data warehousing terminology, think of DEXTER as an extract/transform/load (ETL) utility for AppDynamics data. It extracts information from the AppDynamics platform, transforms it into an enriched, query-able form for faster access, and loads it into variety of reports for:

  • Application logical model (applications, tiers, nodes, backends, business transactions)
  • Performance metrics (average response time, calls, errors per minute, CPU, memory, JVM, JMX, GC metrics)
  • Dependency data (flow maps, relationships between components)
  • Events (errors, resource pool exhaustion, application crashes and restarts, health rule violations)
  • Configuration rules (business transaction, backend detection, data collectors, error detection, agent properties)
  • Snapshots (SQL queries, HTTP destinations, data collectors, call graph data, errors)

By extracting the data from AppDynamics, converting it into queryable format and storing it locally, the data can be preserved with full fidelity indefinitely and interrogated in new and novel ways.

Scenarios Enabled by This Tool

Here are some scenarios that are possible with data provided by AppDynamics DEXTER:

  • Investigation of what is detected and reporting across multiple Controllers and multiple Applications
  • Evaluating what components (Tiers, Nodes, Backends, Business Transactions) are reporting and what load they have
  • Inventory of configuration in multiple environments
  • Comparison of configuration between multiple environments
  • Health Checks for On-Premises Controller – grabbing of data from for later investigation, when Controller is no longer accessible
  • Extraction and preservation of fine-grained Metric, Flow map and Snapshot data for interesting time ranges (such as load test, application outage, interesting customer load) with goal of investigation and comparison in the future
  • Visualization and correlation of Events, Health Rules Snapshots to the Metric data
  • Discovery and data mining of of Snapshots by the types and contents of the Exits (HTTP call and SQL query parameters), Data Collectors, entities involved (Tier, Backend, Error, Service Endpoint and Applications) and Call Graph data

The 3 part Walkthrough gives an overview and screenshots of the tool in action.

Example Reports

Entity Details

"Entity Timeline View" is part of Entity Details report that is generated for Application and all of its Tiers, Nodes, Business Transactions, Backends, Service Endpoints and Errors. It provides a single-pane view into many things, including:

  • 1-minute granularity Metrics in the 1 hour time frame for each hour in the exported range
  • Filterable list of Events and Health Rule Violations, arranged in the timeline of that hour, with details of the Event
  • Filterable list of Snapshots, broken by Business Transaction and User Experience, arranged in the timeline of that hour, and hotlinked to the specific Snapshot

Entity Details

Detected APM Entities

If you ever were presented with a large Controller (or several) full of unknown number of Applications, Tiers and Nodes, you will like the detail provided by Detected APM Entities report.

Detected APM Entities

As alternative to Excel, PowerBI visual analytics tool offers advanced exploration possibilities of the same data in Detected APM Entities in PowerBI report.

Detected APM Entities in PowerBI

Entity Metrics, Graphs and Flow Maps

Entity Metrics report shows summary and graphs for all Metrics for each and every detected Application, Tier, Node, Business Transaction, Backend, Service Endpoint, Errors and Information Point. This makes it very valuable in times when you want to rapidly assess hundreds of Applications, Tiers and Business Transactions and see which ones need your attention.

A scatterplot of Calls per Minute vs Average Response time is provided for all types of Entities, allowing you to see what items are both slow and frequently called: Entity Metrics

Per minute breakdown with ART vs CPM scatter: Entity Metrics

All Nodes JVM GC metrics are stored in high granularity forever: Entity Metrics

To take this a step further, PowerBI visual analytics tool offers advanced, interactive displays of the same data in Entity Metrics in PowerBI and Entity Flowmaps in PowerBI reports.

Entity Metrics in PowerBI

Render Flowmap of activity with Sankey style flow diagram for both Average Response Time and Calls per Minute metrics: Entity Flowmaps in PowerBI

Snapshots Report

Have you ever wanted to find a snapshot that calls a specific Tier, Backend or Application?

How about the one that uses specific SQL query?

And how about the one that has a real Call Graph?

Or maybe also pone with special Data Collector value?

Or how about finding out how many times that special Query was slow in a given time range?

Or discover which classes and methods are called in which Snapshots?

How about all of the above, combined?

In Snapshots report you can do all of that, and more.

Snapshot Exit Calls broken by time and duration:

Snapshots with multiple Segments have an enhanced Waterfall view, with “^” caret character indicating exactly when in the Segment execution the Exit Calls occurred

An even better way to explore this data is by using Tableau or PowerBI visual analytics tools with very interactive and rapid exploration views provided by Snapshots in PowerBI, Snapshots Method Calls in PowerBI reports: Snapshots in PowerBI

... and Snapshots in Tableau: Snapshots in Tableau

You can even explore your Call Graph data Snapshot Method Calls in PowerBI

Flame Graph and Flame Chart Reports

Flame Graphs and Flame Chart reports are an ingenious and useful way to visualize many call graphs in single screen.

Sum of all calls in Application for entire time range

Sum of all calls in Application with Time grouping /

Configuration Report

Configuration report provides information about Controller Settings and Application configuration as well as comparison of configuration between multiple environments.

Here is an example showing non-default Agent Properties set on multiple Applications in multiple Controllers Configuration

Here is an example showing differences Configuration Differences

Detected Entities for Server Infrastructure Monitoring Report

Server Infrastructure Monitoring inventory of Machines, CPUs, Disks and Processes is available in Detected SIM Entities report.

Detected Entities for Database Monitoring Report

Database Monitoring Queries, Sessions, Blocked Sessions and other interesting database-related artifacts can be seen in Detected DB Entities report.

Detected Entities for Web End User Monitoring

Web End User Monitoring inventory of Pages and AJAX Requests, Page Resources, related Business Transactions and Geo Locations is available in Detected WEB Entities report.

Detected Entities for Mobile End User Monitoring Report

Mobile End User Monitoring inventory of Network Requests is available in Detected MOBILE Entities report.

Detected Entities for BusinessIQ Applications Report

BusinessIQ inventory of Searches, Widgets, Saved Metrics, Business Journeys, Experience Levels, Schemas and Fields is available in Detected BIQ Entities report.

Users, Groups and Permissions Report

Users, Groups, Roles and Permissions report shows information about each and every security entity (User, Group, Role and Permission) and their relationship in Controller is available in Users and Permissions report.

Dashboards Report

Dashboards report shows information about all Dashboards, Widgets and Time Series and their relationship to other Entities in Controller is available in Dashboards report.

APM Entity Dashboards Report

See Flowmap dashboard screenshot about each and every detected Entity in APM applications in Entity Dashboards report.

Get Started

Documentation

If you are new to AppDynamics DEXTER and want an introduction, read through 3 part Walkthrough.

Learn how the tool works by reading Documentation in the project wiki.

Install and Run Application

Install Application, create Job File to specify what to do, and Run Application.

Review Results

You will see the results in the Output folder.

To understand what you are looking at, read Description of Excel Reports.

Support

If you need help with issues and/or intepretation of results, read Getting Support.

Other Location

AppDynamics DEXTER is also hosted on AppDynamics Exchange in Extensions area.

Acknowledgements

appdynamics.dexter's People

Contributors

anoop7181 avatar bhjelmar avatar danielodievich avatar juliogomez avatar saxenaabhi142 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

appdynamics.dexter's Issues

Flomaps PowerBI Report - Add Network Graph/Force Graph

Add new tabs with Network/Force graph visualization
Add enhancements to the edge lines connecting nodes to include colors by the type of the connecting protocol
Diameter of items based on error rates
Width based on call volume
Intensity based on ART

Authentication error

This is my Appdynamics account :
image

Here is the Screenshot when I'm trying to run DEXTER :
image

Entity Flowmaps Report - Adjust cross-Application Flowmap calls

For downstream tiers called from another Application, add a new column to indicate which Application they come from and change the name of the calling Entity from Application to the Tier
This should connect Sankey and Force Graph/Network Graph visualizations for cross-application calls better

Issue on Running Dexter - Release 1.0.0.36

Hi Team,

I was trying to run the appDynamics.Dexter Release 1.0.0.36 . But unfortunately i am getting error on the Event and Health Rule violations stage. If i sent Event as false in DefaultJob.Json file it skips the error and saves the file.

Is it possible to help me on this ETL to extract data from app Dynamics controller for powerBI reporting.

capture

Regards
Gopi Bava

Problem with proxy autentication

Hi
I have a problem with proxy settings
https://github.com/Appdynamics/AppDynamics.DEXTER/wiki/Proxy-Settings

I add the block of proxy in my net471\AppDynamics.Dexter.exe.config
the original code of this config is:
<?xml version="1.0" encoding="utf-8"?> <configuration> <startup> <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.7.1"/> </startup> </configuration>

i added the block system.net with the default proxy in the configuration block in my first try
i added the block system.net with the default proxy behind the configuration block in my second try

All time i obtain the same error using a correct job json:

dotnet exec netcoreapp2.0/AppDynamics.Dexter.dll -j jobs/eciJob1.json
Checking input job file /home/jtimon/proyectos/ECI/DEXTER/jobs/eciJob1.json
Checking output folder /home/jtimon/AppD.Dexter.Out
Creating job output folder /home/jtimon/AppD.Dexter.Out/eciJob1
Processing input job file to output job file
Target [1] Controller ControllerApi: ControllerUrl='https://xxxxxxxxx.saas.appdynamics.com', UserName='yyy@xxxxxxxxx' not accessible
Job File Problem: Expanded targets but not a single valid target to work on
This is most likely caused by either incorrect password or incorrectly specified user name.
Only AppDynamics-internal authentication is supported in username@tenant format
If you are using SAML or LDAP account, please change to AppDynamics-internal account
Application execution took 00:00:00.6768887 (676 ms)

Regards

Flame Graph Report: Add better color coding to unmapped code blocks

Currently using Orange color to anything that isn't mapped by a lookup CSV file
Suggestion: come up with a few colors in addition to Orange and randomly assign them to the frameworks that are not mapped, keeping track in the context of the currently rendered flame graph

Better support REV application config reporting of BTEntry Rules

To better support REV application config reporting, please report the application config for BTs with the broken down by URL based rules [SERVLET, ASP_DOTNET] and Class based rules [POJO, POCO].

The REV application template calls for documenting these columns for URL

  • Rule Name
  • Tier
  • URL Match String
  • Splitting

Most of this is in the report already. It would be helpful to extract some part of the URL match string from the MatchConditions and put that into a column. I would use the Num Detected BTs instead of the split condition.
Example:
screen shot 2019-02-15 at 3 54 48 pm

For the Class BT Rule group the template calls for

  • Rule Name
  • Tier
  • Match Class
  • Match Method
  • Splitting

Again most of this is in the report already. It would be helpful to extract the class name and method like in custom exit rules for example and use the Num Detected BTs instead of the split condition.
screen shot 2019-02-15 at 4 02 19 pm

The great thing about using Dexter fo this is that Dexter already exports much more detail about the rules than the doc can handle The doc is really an exercise in noting what was worked on without giving enough detail to really understand the rules. On the other hand an experienced per could read the Dexter exports have a much better idea what the rules were really about. Plus Dexter reports on the rules effectiveness with the Num Detected BTs column which very useful.

Snapshot collect filter by BT name

Hi Daniel
I am using Dexter for extract snapshots and other metrics, in a customer with 40 millions of call per day in the main application.
I have a problem for extract the snapshots in this big application, because in the first step of Dexter process, when collect data from Controller, download all snapshots in the application, don´t use filters "SnapshotSelectionCriteria/BusinessTransactions"
I see the Api documentation for version 4.5: https://docs.appdynamics.com/display/PRO45/Metric+and+Snapshot+API
It´s possible use this filter: "business-transaction-ids" for accelerate the proccess, and filter the snapshot extraction by BT

Regards

Detected Entities Report - Refactor Service Endpoint parsing

During parsing phase the Service Endpoints are parsed from metrics to detail. Change it the other way around, go from detail with types and IDs back to Metrics.
This is done to avoid having duplicate IDs found - had a problem with that with customer who had the same service endpoint name with different types in multiple tiers.

Entity Metric Graphs Report - Support metrics extracted with wildcards from the Custom Metrics branch put there by Extensions

When extracting metrics with wildcards from under the custom metrics branch that were put there by custom extensions, the current rendering does not output metrics if their names are not explicitly specified

Corrected that to compare metrics with the match of the name instead of exact match

Also output up to 10 metrics of each kind because graph eventually breaks if you have too many.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.