Support end-to-end cross-tier correlation about applicationinsights-dotnet HOT 39 CLOSED

@frictionfree we are working on enabling this feature. We will use our runtime code instrumentation technology to pass http headers across the wire.

from applicationinsights-dotnet.

@alancameronwills, we plan to make this change in UI. What do you think about the term transaction as oppose to root operation.

from applicationinsights-dotnet.

Customers would be better served by "root operation." Although "transaction" is more immediately recognizable, we are using the term "operation" anyway. If we use "transaction", we will somewhere have to explain that "transaction" = "root of the operation tree." Many customers won't read that explanation (because they don't like to waste their time reading), so they'll just wonder whether there is some special significance in a transaction, since we're using a different word for it. They might think there's some way in which if one fails, its operations all roll back - because that's what 'transaction' really means, after all.

One term is better than two; less to explain, less scope for confusion. Tooltips are the only documentation anyone ever reads.

from applicationinsights-dotnet.

How do we change label "All exceptions for this transaction"? Will it be "All exceptions for the root operation" or "All exceptions related to the root operation of this request"?

CC: @AlexBulankou

from applicationinsights-dotnet.

"All exceptions under the root operation" more clearly implies inclusion of exceptions associated with children.
"...of this request" is probably not required because (I should think) it's clear from the context.

from applicationinsights-dotnet.

What if we flip the coin and instead of focusing on naming the root something special we simply call it Operation and NestedOperation?

from applicationinsights-dotnet.

@MarioHewardt it sounds interesting. We never mention Root. It is only Operation, Parent Operation or Request and Dependency. When we will have new type called Operaiton it will represent NestedOperation, but still be called just Operation.

So SDK will have:

1. Own (nested operation) id: Request.ID & Dependency.ID
2. Parent operation id: ITelemetry.Context.Operation.ParentID
3. Operation id: ITelemetry.Context.Operation.ID
4. Operation Name: ITelemetry.Context.Operation.Name

Is it clear enough that ITelemetry.Context.Operation.ID is an ID of root operation? Or it should be ITelemetry.Context.Operation.RootID?

from applicationinsights-dotnet.

IMO, RootID implies that the ID is of the "root" operation, which may or may not be immediate parent. While the usage here implies immediate parent. So, I think we should stick with ID and ParentID.

from applicationinsights-dotnet.

I think it is clear enough - we make it clear that "Operation" has a wide scope.
My concern though is that absense of parentId on all telemetry types will make it hard to filter exceptions, traces, events etc belonging to specific request, as the whole operation may span across thousands of ajax requests for example.

from applicationinsights-dotnet.

Ok, so it seems that everybody is in favor of dropping both terms Root and Transaction and stick with Operation being single item as well as whole context. I like it.

from applicationinsights-dotnet.

Great.

from applicationinsights-dotnet.

Am I understanding correctly that in this case we do not have an overall, End-To-End operation ID, but rather a set of operations chained together via ID/ParentID?

Still not quite clear on how Request.ID and Dependency.ID will interact with the new model, can you clarify?

from applicationinsights-dotnet.

No, we will have all three identities:

1.Own (nested operation) id: Request.ID & Dependency.ID
2.Parent operation id: ITelemetry.Context.Operation.ParentID
3.Operation (a.k.a. root operation) id: ITelemetry.Context.Operation.ID

from applicationinsights-dotnet.

So let's use an example.... let's say you have 3 nested operations. For example, WebSite A makes an HTTP Request to website B, which in turn makes another HTTP request to WebSite C:

A -> B -> C

In this case, you'd have:
In A:

• Request.ID set to <a_ID>
• Operation.ID set to <root_ID>
• Operation.ParentID set to <null>
• Then Dependency.ID set to <d-a_ID>

In B:

• Request.ID set to <b_ID>
• Operation.ParentID set to <a_ID>
• Operation.ID set to <root_ID>
• Then Dependency.ID set to <d-b_ID>

In C:

• Request.ID set to <c_ID>
• Operation.ParentID set to <b_ID>
• Operation.ID set to <root_ID>

Is my understanding correct?

from applicationinsights-dotnet.

Few missing pieces:

1. First operation is a root. So <root_ID> will be the same as <a_ID>
2. Dependency is an operation by itself. So it will have parent ID equal <a_ID>
3. Thus B's parent ID will be <d-a_ID>

same applies to the next two items in the chain

from applicationinsights-dotnet.

Hoping to revive this thread as I believe it is very important.

Any ideas on how WebClient/HttpWebRequest will be hooked to automatically propagate the attributes from B to C?

from applicationinsights-dotnet.

What is the ETA for this feature to be enabled in the azure portal so we can search across all correlated requests.
I see that the parentId is being set from x-ms-request-id in OperationCorrelationTelemetryInitializer.

from applicationinsights-dotnet.

@sunilswa - we made a first step in components correlation. See https://github.com/Microsoft/ApplicationInsights-Home/blob/master/app-insights-app-map-preview.md
We also made correlation between JS PageViews and server side events.

We will be working on improving of cross components correlation, but I cannot provide specific ETA.

What scenario you want the most?

from applicationinsights-dotnet.

from applicationinsights-dotnet.

So I'm trying to wrap my head around where this is and where it's going. We're using AI in some less out-of-box scenarios and as a result there's a lot of free correlation we're not getting which I'm trying to tie back together, but I think it would be best to try to get a run down of what IDs matter in which situations. For starters, I've noticed that the the terminology "RootId" confuses me; this feels to me like it would be the first operation in a chain of operations, but what I see in the code suggests that it's the most recent operation? Just trying to confirm. Besides that, I'm trying to determine which IDs should be set to correct nesting in the portal and best case correlation in the app map. It would be nice if there was documentation that explained the relevance of all these IDs, but in the meantime, the following are examples of what I'm hoping to understand:

• How do I ensure Dependencies are nested under Page Views in the portal, as well as to ensure correlation in the App Map
• How do I ensure Requests are correlated with Dependencies in App Map (if possible, it would be great to see these nested under their source Dependency in the Portal as well); Some of our Dependency->Request chains are not using Web API/MVC like message queues or WCF (in any configuration including TCP or named pipes, for example; my plan is to use inspectors to read/write message headers to transfer context)
• How do I ensure Traces, Events, Metrics, Exceptions, etc. (non-"OperationTelemetry" items) are correctly nested under the Page View, Dependency, or Request logically encapsulating them (Our environment has some async and out-of-band complexity here; we use CallContext and some other methods to keep track of the relationship, but it's not immediately clear what the correct handling is).

I believe I've answered many of these questions for myself by reading the source of the Web telemetry initializers and modules, but there are a lot of obfuscating factors given different pieces tying in to different extension points. It would be nice to just know which IDs to set to what and when, at least for confirmation. I'm a bit curious why PageViewTelemetry isn't considered an OperationTelemetry since it seems to share a lot of commonality in how it relates to other telemetry and I find myself checking for both in a few places, but I suspect this confusion just stems from my not seeing exactly how the telemetry correlates clearly.

We're currently using a single AI resource which would ideally allow starting from a page view, drilling down to API dependencies, which drill down to API requests, which drill down to events and SQL dependencies happening on the server, etc. (or, especially ideally, other service dependencies->requests->dependencies->requests->SQL dependencies, etc.) I'm curious what information this new pre-release stuff uses to correlate across resources within a resource group; thinking it would be the same IDs I'm already trying to preserve, but just wondering so I can prepare and we can consider separating our resources out some.

from applicationinsights-dotnet.

OK, after re-reading the thread I think I'm getting it. Just to re-illustrate with some more detail what I've seen above:

Request R1:

• Request.ID = <r1_id>
• Request.Context.Operation.ID = <r1_id>
• Request.Context.Operation.ParentID = null

Dependency R1D1:

• Dependency.ID = <r1d1_id>
• Dependency.Context.Operation.ID = <r1_id>
• Dependency.Context.Operation.ParentID = <r1_id>

Request R2 (Server side of R1D1)

• Request.ID = <r2_id>
• Dependency.Context.Operation.ID = <r1_id>
• Dependency.Context.Operation.ParentID = <r1d1_id>

Event R2E1

• Event.Operation.ID = <r1_id>
• Event.Operation.ParentID = <r2_id>

Dependency R2D2 (I would have named this R2D1, but I mean...)

• Dependency.ID = <r2d2_id>
• Dependency.Context.Operation.ID = <r1_id>
• Dependency.Context.Operation.ParentID = <r2_id>

Trace R2D2T1

• Trace.Context.Operation.ID = <r1_id>
• Trace.Context.Operation.ParentID = <r2d2_id>

Does this seem correct? If so, how do we correlate these to a PageView at the root level? PageView is not considered an Operation, but in many cases it does seem like the root cause of dependency->request chains.

from applicationinsights-dotnet.

@TheXenocide - this is correct understanding of the correlation how we design it. It may not be ironed in UI completely. Mostly because we do not allow cross-component view for the end-to-end transaction view. Please report issues as you see them.

I agree that rootId is a confusing term. You can find discussion earlier in this thread.

It turns out the AI implementation is quite similar to open tracing except parent ID. In the nutshell mapping looks like this: Operations are spans and root ID is trace ID. Open tracing allows multiple parents, Application Insights doesn't.

from applicationinsights-dotnet.

@sunilswa it is a good scenario. We will keep it in mind

from applicationinsights-dotnet.

Cross-component correlation in the portal would be very helpful, but I understand if it isn't currently designed for that. The last question is: how do I ensure Page Views contain the correct immediate Dependencies? PageViews do not have their own ID so I'm not sure what to put in the Dependency Operation ID or ParentID?

from applicationinsights-dotnet.

Good catch. I was under impression it was already added to the PageView type. Created issue for this: microsoft/ApplicationInsights-JS#361

from applicationinsights-dotnet.

Oh wow lol. I thought I was misunderstanding something O_o. Glad I could help. I am using PageViewTelemetry here in the .NET implementation as well, if we could make sure that gets added across each surface. It makes sense that the primary target for it is JS though.

from applicationinsights-dotnet.

Actually, out of curiosity, how does that currently work? I am currently able to see dependencies in the context of a page view in the portal when using the defaul JS stuff; it's just now that I'm trying to use the idea of a page view with .NET that I'm having trouble getting the same result?

from applicationinsights-dotnet.

I think it is using Root ID - pageView.Context.Operation.ID for this. I might be wrong - @KamilSzostak will know for sure

from applicationinsights-dotnet.

OK, so essentially force a "new operation" on the page view and then use that id in the ParentID of any Dependencies/Traces/etc. that should be nested in it? I think that makes sense for now, assuming we don't hear otherwise from Kamil.

from applicationinsights-dotnet.

I want to say that this would be a terrific feature if you also add to the UI. I am starting to use application insights in a distributed architecture and I am not happy with the analysis I am getting. We aer in a scenario where we have 2 apps and 2 websites hitting 2 backend api servers (soon 4-5 possobly) and then in the backend one database (soon 2 as we likely split some things).

Application Insights does not track this well. For now, I did add the same tracking to both front end apps with a tag - not properly supported. And once you hit backend servers, there is no separation of them, no hirarchy of calls.

There are voices in the project that talk about using one AI instrumentation key per front end web server, but I see this falling apart when we move the API (currently living in the respective web apps) away to separae hosts, and I loose the cross correlation and the central look at database performance. So, this is not really better. It gets cleaner (as I then run a separate instrumentation pool for every host), but the cost is high (In terms of not being able to run analysis between them).

The proposal is a start - but it has to be backed up with an applicaiton map that can work from multiple front points to multiple end points in an intelligent and analysable manner. Have a look at AppDynamics for what it should be, for example. Just the screenshots at https://www.appdynamics.com/product/application-performance-management/ give an idea where things should go. You have a lot of baseline (missing, but these talks here do add the missing pieces), but the UI must follow.

from applicationinsights-dotnet.

@NetTecture in case it helps your case in the meantime, we're sending some contextual properties about the telemetry of a request (the PageView and Dependency OperationIDs and some other custom properties) as headers with our service request and then placing those in custom properties on the server side dependencies. We push the context all the way down to the SQL Dependency call on the server side, such that our SQL DependencyTelemetry has a custom property like "SourcePageViewOperationId" and "SourceApplication" which doesn't play quite the same in the AI UI per se, but is useful in filtering and analytics at least. Ours is using the .NET client libraries exclusively (no JavaScript stuff so far) and we're generating our own OperationIDs so that we can control the flow of context better, but thought our approach might help you for now. As the API implements better correlation we'll need these IDs at that layer anyway (the JS client already attaches these headers and the ASP.NET request module already reconstitutes them, but our environment is desktop app->WCF service so we had to implement manually; still the point is you can use headers to get extra information into custom properties on the other side)

from applicationinsights-dotnet.

@TheXenocide It does not REALLY help, because in this case the visualization is a large part of what makes it useful. All I wanted to point is a direction to walk towards. Looking forward to see you moving forward.

from applicationinsights-dotnet.

@NetTecture alas, I understand. In honesty the visualization is a hugely valuable component to us as well as some of our logical operations in the client (which we're calling a Page View since it's a discreet tab in the UI) end up executing a large number of different dependencies, which map to request, which map to a large number of other dependencies, so when we're looking at a Page View "all telemetry for this operation" is really quite a lot of items in a flat list. We would much prefer drilling down from Page View->Dependency->Request->Dependency->Exception, etc. where child telemetry is nested deeper than a single operation. This is essentially our hold over while we test and get our first implementation running. The Analytics are nice and all, and the filters will still help with other methods of slicing up the data, but realistically deep correlation in the Portal tooling is a very important feature to us as well. The Timelines in Visual Studio would also be more useful to use if we could see all nested telemetry (so in Page View the timeline would include all telemetry, but once we clicked on a specific dependency, it would be nice if it only showed the timelines for the dependency and any operations nested under it, instead of all telemetry in the root operation). The timeline does not apply filters of the current search in VS, so even our custom properties don't help filter this view.

from applicationinsights-dotnet.

Hello guys,

I was playing around with app insights trying to correlate my requests together and I tried exactly the scenario that was described by @TheXenocide . I am using the latest stable version of AppInsights (2.2.0) I made the following code to control exactly the value of each field to make sure it was not being overridden by StartOperation.

TelemetryClient client = new TelemetryClient();
RequestTelemetry req1 = new RequestTelemetry();
req1.Start();
req1.Name = "REQ1";
req1.GenerateOperationId();
req1.Context.Operation.Id = req1.Id;
req1.Context.Operation.ParentId = null;
req1.Context.Operation.Name = "OPERATION_" + req1.Name;
try
{
    Thread.Sleep(200);
    DependencyTelemetry req1_dep1 = new DependencyTelemetry();
    req1_dep1.Start();
    req1_dep1.Name = "REQ1_DEP1";
    req1_dep1.GenerateOperationId();
    req1_dep1.Context.Operation.Id = req1.Id;
    req1_dep1.Context.Operation.ParentId = req1.Id;
    req1_dep1.Context.Operation.Name = "OPERATION_" + req1.Name;
    try
    {
        Thread.Sleep(200);
        RequestTelemetry req2 = new RequestTelemetry();
        req2.Start();
        req2.Name = "REQ2"; // =REQ1DEP1
        req2.GenerateOperationId();
        req2.Context.Operation.Id = req1.Id;
        req2.Context.Operation.ParentId = req1_dep1.Id;
        req2.Context.Operation.Name = "OPERATION_" + req2.Name;
        try
        {
            Thread.Sleep(200);
            TraceTelemetry req2_trace1 = new TraceTelemetry();
            req2_trace1.Message = "REQ2_TRACE1";
            req2_trace1.Context.Operation.Id = req1.Id;
            req2_trace1.Context.Operation.ParentId = req2.Id;
            req2_trace1.Context.Operation.Name = "OPERATION_" + req2_trace1.Message;
            client.TrackTrace(req2_trace1);

            DependencyTelemetry req2_dep2 = new DependencyTelemetry();
            req2_dep2.Start();
            req2_dep2.Name = "REQ2_DEP2";
            req2_dep2.GenerateOperationId();
            req2_dep2.Context.Operation.Id = req1.Id;
            req2_dep2.Context.Operation.ParentId = req2.Id;
            req2_dep2.Context.Operation.Name = "OPERATION_" + req2.Name;

            try
            {
                Thread.Sleep(200);
                TraceTelemetry req2_dep2_trace1 = new TraceTelemetry();
                req2_dep2_trace1.Message = "REQ2_DEP2_TRACE1";
                req2_dep2_trace1.Context.Operation.Id = req1.Id;
                req2_dep2_trace1.Context.Operation.ParentId = req2_dep2.Id;
                req2_dep2_trace1.Context.Operation.Name = "OPERATION_" + req2.Name;
                client.TrackTrace(req2_dep2_trace1);
            }
            finally
            {
                req2_dep2.Stop();
                client.TrackDependency(req2_dep2);
            }
        }
        finally
        {
            req2.Stop();
            client.TrackRequest(req2);
        }
    }
    finally
    {
        req1_dep1.Stop();
        client.TrackDependency(req1_dep1);
    }
}
finally
{
    req1.Stop();
    client.TrackRequest(req1);
}

client.Flush();

What I was expecting:

• I could drill down from Request to Dependency to Request to Dependency with one panel for each telemetry.
• I was expecting REQ1 to have only one dependency named REQ1_DEP1.
• I was expecting REQ2 to have only one dependency named REQ2_DEP2.

What I received is instead the following:

• REQ1 have two dependencies named REQ1_DEP1 and REQ2_DEP2.
• REQ2 have two dependencies named REQ1_DEP1 and REQ2_DEP2.

I guess the reason is because this feature is not yet available in stable, but can you confirm the expected behavior is what will be available in the future? or not?

from applicationinsights-dotnet.

@MarcAndreBertrand I can confirm this is the behavior I see with our implementation as well (and the results display similarly in Visual Studio); I'm under the impression your assumption is correct, but I'm interested in the official answer as well.

@SergeyKanzhelev after having setup our infrastructure to maintain and populate these IDs, and similar to my question about PageViewTelemetry being a root operation, I'm wondering if EventTelemetry couldn't be a derivative of OperationTelemetry as well? I ask because, in practice, once our page has completed loading there are several events that amount to different dependency calls, all of which we have associated with the original page view. In some cases inferring which events caused which dependencies is more difficult than others and I was thinking it would be much easier to trace if these were part of the correlated drill downs, so instead of the panel for a page view only showing requests or a panel for a request only showing dependencies, each would just show whichever direct child telemetry items and that events would also be suitable parents with their own Operation ID (I'm thinking "all telemetry for this operation" would still show the flat list of everything with the same Root Operation ID). The other thing is that we're recording a custom metric with most of our events which is the duration of the event itself, since one event can encompass several requests. Since this is a custom metric it doesn't play with the default UI the same. Maybe I'm expecting too much or misunderstanding what Event stands for, but I feel like if it's not an operation then it's barely more than a trace, especially since we've decided that based on the pricing model we won't be connecting our traces to AI by default. I had considered opening a separate ticket for this, but thought it was relevant to consider in discussion with cross-component correlation and the drill down UI.

from applicationinsights-dotnet.

@kartang - where do we stand on parent-ID only visualization?
@AlexBulankou, @brahmnes - here is a great feedback for transactions correlation.

@TheXenocide Yes, we have limited support for transactions correlation. You can query data in Analytics, but UI is not working as expected yet.

EventTelemetry is designed to report event in time. Typically business event. It is a parent of OperaitonTelemetry, not other way around.

from applicationinsights-dotnet.

Folks, thanks for analysis and suggestion. To summarize, in order for us to make related items behavior consistent with operations and transaction semantics, we need to make the following changes:

1. Everywhere, where we are referring to operation, for example, All available telemetry for this operation, we will refer to the immediate parent operation, not to root transaction.

2. We will add new related item for each of the types: All available telemetry for this transaction. This will return the list of telemetry items with the same OperationID, i.e. all telemetry logged within the same transaction.

Comments/feedback, please let me know.

from applicationinsights-dotnet.

@SergeyKanzhelev I made the following schema to make sure I understand correctly. I understand OperationID is supposed to be the RootID and that each telemetry share the same OperationID. If so, should the OperationName also be the RootName?

from applicationinsights-dotnet.

Cross component telemetry correlation went live in 2.4.0-beta1. This article explains the basics of the data model and how it's implemented in .NET: https://docs.microsoft.com/en-us/azure/application-insights/application-insights-correlation

Closing the issue, please comment on the doc article if you have a doc feedback or file an issue if correlation is not working for your scenario.

from applicationinsights-dotnet.