• Micro benchmarks
  • Add benchmarks for Transaction & Span creation overhead.
• Document benchmarks (Plus the Overhead)
• Store benchmarks in ES
• Create Kibana dashboards
• Document the package size
• Macro benchmarks on how the agent is affecting the user experience #302
• Run the benchmarks on Jenkins CI for publishing to ES
• Run the benchmarks on bare metal server #456

Postponed:

• Run the benchmarks on each PR and add fields in the elasticsearch document - https://github.com/elastic/observability-dev/issues/299 -> #568
• Add memory benchmark for the browser - memory leaks -> #514
• Consider moving the benchmark runner to the top to avoid reading/storing files multiple times

• Capturing blocked time for requests
• Differentiating between Cached vs Network resources
• Annotating failed resources

Docs need to be improved in order to better explain how to use custom transactions.
Also we need a glossary or some explanation of the basic terminology of APM. In this last case maybe we could link to some global APM explanation.

• Review _finish, end, done, finish methods in Transaction
• Refactor TransactionService, Transaction, Span
• Add e2e tests for using custom transactions (Accessing the current transaction across JS files)
• Refactor navigation timing marks so it can be add to any transaction
• A configuration option to include navigation timing on the first transaction, or not at all, also whether to send it automatically or not
• Write documentation about using the API as well as common pitfalls
  • Async tasks like fetch and the right time to end the transaction
  • There only can be on transaction at any time

Please add support for secret token.

Implement lightweight OpenTracing API compatibility bridge.

• Add documentation
• Create a separate bundle for OpenTracing
• Improve code coverage
• Only add special tags if it's a transaction
• Support logging errors
• Convert all timestamps to relative times
• Add opentracing tests
• Start a transaction if there's no active transaction
• Introduce open tracing context (needs to be immutable)
• Implement open tracing subclasses
• Add support for parent-child relationship between spans #119
• add timestamp to transactions and spans
• Support adding tags to individual spans and transactions #65

Should align with elastic/apm#32

I use both elastic-apm-node and apm-agent-js-base in my nodejs + reactjs project with server side render, and i will import apm both the nodejs and browserify side,

if (typeof window !== 'undefined') {

    const { init: initApm } = require('elastic-apm-js-base');

    initApm({
      // Set required service name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
      serviceName: '',

      // Set custom APM Server URL (default: http://localhost:8200)
      serverUrl: 'http://localhost:8200',

      // Set service version (required for sourcemap feature)
      serviceVersion: '',
    });
}

import apmNode from 'elastic-apm-node';

const apm = apmNode.start({
  // Set required app name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
  serviceName: 'xxx',

  // Use if APM Server requires a token
  secretToken: '',

  // Set custom APM Server URL (default: http://localhost:8200)
  serverUrl: 'http://localhost:8200',
});

I think its better to do someting like axios and isomorphic-fetch to support both node and browserify side.

{
  "name": "isomorphic-fetch",
  "version": "0.0.0",
  "description": "Isomorphic WHATWG Fetch API, for Node & Browserify",
  "browser": "fetch-npm-browserify.js",
  "main": "fetch-npm-node.js",

It returns http status 0, potentially a cross-origin issue.

• Investigate running apm-server in Travis
  related #16

Debugging e2e tests when they fail on travis is hard, printing the logs can help with that.

In some browsers (e.g. IE9) navigation and resource timing apis are exposed in timestamps instead of DOMHighResTimeStamp therefore the generated spans have incorrect start and end.

Work is underway to add support for Metrics to the API/server: elastic/apm-server#964
Support needs to be added to this agent too.

The default minimum set of metrics should be supported. TBD which additional metrics will be supported for this agent.

• Investigate the scope and specific metrics to add

• Add notes about stacktrace changes and patching
• Troubleshooting
• How to debug (log levels)
• add captureError to docs
• Guide about uploading sourcemap if there is secret token setup in the apm-server,(elastic/apm-server#1268)
• How to use script tags instead of npm

Currently only the keyword fields are limited at 1024 characters, we need to also apply this limit to non keyword fields. There will also be extended fields ie. SQL that can go up to 10000 characters to non-keyword fields.

• Currently we show user-agent in the UI under the user context, something like Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36, we can parse this string and show user friendly values
• [APM] Show context.page.url in the Transaction and Error view

A little bit of background on the issue, we used included the page url as transaction name (without the query string) by default and we had the same problem even without the query string there are a lot applications that simply include ids in their url.

Currently the default transaction name is unknown and not the page url so all transactions are grouped by default under unknown but there is an API to let users set the initial page load transaction name and it can be set to the page url by the user which probably is the easiest way to set the page name. This can create a large number of transaction names.

Some solutions:

• Having a url pattern config option that we use to set the transaction name to the page url
• Implementing heuristics that tries to detects ids in the URL (I've made a POC) on the agent
• Implementing a grouping algorithm on the Kibana side.

Related: elastic/apm-server#1237

• Implement ndjson.strigify
• Add specific tests for ndjson
• Provide required span count (#51)
• Testing on apm-server v2 branch on travis
• Review the proposal doc

In order to avoid inaccuracies in the client's system time make it into APM data, we should set the timestamp of the transaction in the apm-server. Starting in 6.3.0 apm-server will set a timestamp if it is not available on the transaction.

This api is resulting in http://.*.com/v1/client-side/transactions 403 Forbidden.
On the local machine it is working fine.

I find this in the apm-server logs

2018-05-29T07:14:48.393Z	ERROR	[request]	beater/handlers.go:384	error handling request	{"request_id": "01e195a0-f20e-4942-98e7-b4fcee146ec1", "error": "forbidden request"}
2018-05-29T07:14:48.393Z	INFO	[request]	beater/handlers.go:150	handled request	{"request_id": "01e195a0-f20e-4942-98e7-b4fcee146ec1", "response_code": 403, "method": "POST", "URL": "/v1/client-side/transactions", "content_length": 1621, "remote_address": "127.0.0.1", "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"}

Not able to figure out why this is happening. Both(local and the server) have the configuration

In order to distinguish between different deployment environments (staging, production, etc), need to add service.env to the configuration.

Instrumentation of XMLHttpRequest .

• Initial R&D
• POC
• Handling synchronous requests
• Implement patchAll
• Refactor TransactionService to accept XHR events
• Add unit tests in core
• Use the patching in the js-base module
• Add unit tests in base
• Add e2e tests in base
• Add documentation
• Resolve the singleton issue with patching
• Test for failing requests

We want to not show the root.span "transaction" in the Elastic APM UI. Here's an example;

• active (with sampling example)
• logLevel
• serverStringLimit
• flushInterval
• errorThrottleLimit
• errorThrottleInterval
• transactionThrottleLimit
• transactionThrottleInterval
• sendPageLoadTransaction

For duration configs (e.g. ELASTIC_APM_API_REQUEST_TIME), we will support time units like 1ms, 1s, 1m. For pre-v2 config, we should allow suffixes, but also continue to allow no suffix and assume the previous unit (e.g. "ms" for min span duration). For v2+ config, we should require suffixes.

• Parse user-agent field into specific and searchable fields
• Request Kibana change

Other than transaction_id, trace_id and parent_id needs to be recorded as well!

• Consider extending the agent marks.
• Remove timeToComplete
• Provide examples of agent marks, elastic/kibana#18470 (comment)

• Need to provide an API to record custom marks on the current transaction.
• Consider capturing marks added by using performance.mark.
• Add docs
• Add example marks to opbeans

• Investigate PerformanceResourceTiming
• Avoid duplicate spans when fetch is a polyfill
• Remove domain from the url if on the same origin
• Implement promise based instrumentation
• Refactor XHR instrumentation
• Add tests

Apm server will require span count field through elastic/apm-server#1241 in v2 intake.

Hi,

I'm trying to test RUM with a third party web application. I insert js library and initialize it via nginx filter, all seems to be loaded properly within the page

Library version 1.0.0: elastic-apm-js-base.umd.min.js

Initialization

<script src="/static_ext/elastic-apm-js-base.umd.min.js"></script>
<script>elasticApm.init({
	logLevel: "debug", 
	serviceName: "Dev_CelonisAS", 
	serverUrl: "https://sapelk00.bsh.corp.bshg.com:8200"
	})
</script>

page-load tranasaction is being sent to apm-server whenever the page is loaded/reloaded. But for subsequent odata/ajax (ext.HttpRequest spans) i see in the console that thse spans are being started but nothing is sent

When i load the page i see the spans being added...

TransactionService.startSpan POST api/analysis/169/selections ext.HttpRequest
TransactionService.startSpan PUT api/analysis/169 ext.HttpRequest
TransactionService.startSpan GET api/folders/ed5a56a9-b1d9-4687-a025-e1dc924c205d/breadcrumbs ext.HttpRequest

.....

Then wen the page is loaded I see a call to
TransactionService.add
and with it all the spans contained in the transaction are sent to apm-server

If i continue in the page, clicking here and there (several odata calls are triggered with these interactions but the page is stil the same) i see all the spans being added again, but as the page is not reloaded all these spans are not sent

Do i miss something? How could we proceed to send these spans to apm-server?
As far as i can see HttpRequests are properly instrumented as all xhr calls are triggered by apm agent library.
Shoud i include some additional event listener to create/send transactions with all these spans?

Any help/example would be apreciated.

BR,
Raul

In order to keep the span name short we remove the query string part of the http request url, however we also need to store the full url in the span context.

• Add the required span context spec to apm-server
• [APM] Show http url in the Span details

If there are no changes to the specifications, this will imply no changes to the agent

Need to provide documentation on which technologies are supported.

Some examples of similar docs in other projects:
https://www.elastic.co/guide/en/apm/agent/ruby/current/supported-technologies.html
https://www.elastic.co/guide/en/apm/agent/go/current/supported-tech.html
https://www.elastic.co/guide/en/apm/agent/java/current/supported-technologies-details.html

• remove TransactionService.sendPageLoadMetrics
• use TransactionService.startTransaction with the type set to page-load

Some transactions can be lost if the user closes the window in the gap between they are created and sent (500ms). A way around this is to handle beforeunload and force a queue flush.

Related: #56

Further discussion here.

The following context fields are not defined in the apm-server schema and they should be defined if we decide to lock down the allowed fields under context:

{
  page: { referer: "",  url: "" }
}

Right now sampling is suggested using active: Math.random() < 0.1. The way I read the docs, if I only enable for 10% of users, any errors on the 90% will never be reported to APM.

I would assume that errors is a rare but important thing, and I would like to have ALL errors reported to APM, while only sampling transactions/RUM.

Right now, I have simply set active to true, and the used addFilter to ignore all payload.transactions, as including these for all 100% of users is way too much data to process for our setup.

Work is underway to add support for distributed tracing to the API/server:

• elastic/apm-server#913
• elastic/apm-server#968

Tasks:

• Set the page load traceId, spanId and sampled from configuration options
• Document config options and cross-origin guide for the servers
• Document responding to OPTIONS request
• Document sampling transactions
• Adopt new traceparent flags
• Add tests with a backend agent
• Add whitelist for origins
• Resolve cross-origin issue with the elastic-apm-traceparent header (whitelist domains) -> Needs more investigation
• Add the header to unsampled transaction
• Implement sampling
• Investigate Chrome Mobile test failure
• Add a configuration option to set trace id
• Add tests for XHR patch
• Implement correct id format for DT
• Use correct formats for Transaction, Span and Error Ids
• Add the header to the same origin requests
• Generate Trace Id

Global DT checklist:

• update opbean-frontend to include a distributed tracing call
• intake protocol v2
• test that responses from the server during chunked writes work with the client you’re using (if your agent uses chunked writes)
• transaction.span_count.started (elastic/apm-server#1241)

Related:

• Backend agents should document how to respond to the OPTIONS request

• Run apm-server and elastichsearch containers in travis
• Check the data stored in elasticsearch
• Run agent e2e tests within apm-integration-tests