Relative: elastic/apm-server#913

Gradle is unable to resolve the dependency.

JDK 9
Gradle 4.5.1

FAILURE: Build failed with an exception.

• What went wrong:
  Could not resolve all files for configuration ':compileClasspath'.

Could not resolve co.elastic.apm:apm-agent-java:0.1.1.
Required by:
project :
Could not resolve co.elastic.apm:apm-agent-java:0.1.1.
> Could not parse POM https://repo.maven.apache.org/maven2/co/elastic/apm/apm-agent-java/0.1.1/apm-agent-java-0.1.1.pom
> Could not find co.elastic.apm:apm-agent-parent:0.1.1.
Could not find co.elastic.apm:apm-agent-parent:0.1.1.
Required by:
project :
Could not resolve co.elastic.apm:apm-agent-api:0.1.1.
Required by:
project :
Could not resolve co.elastic.apm:apm-agent-api:0.1.1.
> Could not parse POM https://repo.maven.apache.org/maven2/co/elastic/apm/apm-agent-api/0.1.1/apm-agent-api-0.1.1.pom
> Could not find co.elastic.apm:apm-agent-parent:0.1.1.

• Try:
  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

• Get more help at https://help.gradle.org

BUILD FAILED in 1s
1 actionable task: 1 executed

The ElasticApmTracer creates singleton no-op Transaction and Span instances and returns them, for example, when tracing is disabled or required properties like the service_name are not set. At the moment, these are not true no-ops yet. Even though the sampled flag is checked before every with* method in Transaction and Span, the nested objects like URL don't do those checks. This behavior leads to race conditions, Exceptions and even to OOMEs, so it is important that these are true no-ops.

P6spy allocates objects on each query:

https://github.com/p6spy/p6spy/blob/f9f852e895419a8adcda590618aec4ae47513cfe/src/main/java/com/p6spy/engine/wrapper/PreparedStatementWrapper.java#L59-L64

So investigate if p6spy could be easily modified to allow for object pooling or if we should rather implement our own JDBC wrapper. P6spy has an extensive test suite for all sorts of databases and application servers so it would be nice to leverage this.

The intake API only accepts strings with a max length of 1024.

• secret_token
• server_url
• server_timeout
• verify_server_cert
• flush_interval
• transaction_max_spans
• max_queue_size

I was wondering if you have ever tried this agent with oracle weblogic and if not if there are any plans to do so in the future.
Thank you in advance.

Work is underway to add support for distributed tracing to the API/server: elastic/apm-server#913 / elastic/apm-server#968.

The POC done in #83 needs to be improved to make it production ready.

I don't know if this is a correct place for a feature request. Please add support for Spring WebFlux Framework too along with Web MVC. The new frameworks provide native support to build reactive web services. We are currently considering and moving towards Webflux, support in a future release will be awesome.

At the moment, marks are just a Map<String, Object>

I am running APM Server and Agent both as docker containers.

Setting up APM Server was not a problem but I don't see any data flow into the server from the agent.

Here's how I run the agent.

java -javaagent:/elastic-apm-agent-0.5.1.jar \
  -Delastic.apm.service_name=<app-name> \
  -Delastic.apm.application_packages=<app-package> \
  -Delastic.apm.server_url=http://apm-server:8200 \
  -jar <app-name>.jar

The app runs fine with the command above and of course the jar file is a spring-boot application.

Could anyone point out anything I missed?

Thank you in advance

Use the config active to toggle between noop and normal mode. Once we have support for automated instrumentation, use instrument config key to be able to turn of instrumentation entirely (has to be set at startup and can't be changed at runtime).

Let users configure a list of APM servers which are used in a round-robin fashion when reporting transactions.

For the alpha it is not necessary to have documentation on elastic.co. Having some installation steps in the README should be good enough to begin with.

Hello, I'm configuring the java agent with tomcat. I'm not getting a log file, can someone confirm my settings here are correct, no errors in catalina.out:

-Djava.awt.headless=true -Dspring.profiles.active=test -Delastic.apm.service_name=service_name -Delastic.apm.application_package=com.myapp -Delastic.apm.server_url=http://localhost:8200 -Delastic.apm.logging.log_level=DEBUG -Delastic.apm.logging.log_file=/var/log/tomcat8-8084/elasticapm.log -Xmx512M -javaagent:/etc/elastic/elastic-apm-agent-0.5.1.jar

Instead of instrumenting a Java process I'd like to embed the Java agent into my application and provide custom metrics.

It would be nice if there is some doc on how to do it this way, i.e. which jar (Maven artifact) is needed and how the API looks like.

Consider using https://www.testcontainers.org.

https://docs.oracle.com/javaee/7/api/javax/servlet/AsyncContext.html

Initially, Apache Tomcat 7.x, 8.5.x and 9.x are supported. This includes embedded Tomcat and Spring Boot with embedded tomcat.

Add benchmarks of an actual application using the Elastic APM java agent. Determine these metrics:

• Latency overhead
• CPU overhead
• Network overhead
• Heap overhead
• GC overhead
  • Count of GCs per algorithm
  • Total GC time per algorithm
• Allocation rate overhead

Consider using JMH, Apache JMeter or www.github.com/elastic/rally

Next up:
Ingest benchmark results into Elasticsearch, visualize via Kibana and use Watcher/ML to notify on regressions.

Other code quality tools might also be interesting, but I had bad experiences with most of them in terms of an overwhelming amount of false-positives.

At the moment, users will have to manually change some code and configuration in order to install the agent. With this task, the registration of the ApmServletFilter, Spring ApmHandlerInterceptor and p6spy ConnectionWrappers should be done via bytecode instrumentation (BCI).

The general principle is to do as few magic as possible with BCI. Whenever there is an official interceptor API, like servlet Filters, it's preferred to use these and use BCI to automatically register these interceptors.

• This makes writing new integrations easier as it allows for contributions of people not familiar with BCI. The magic can then be later added by us.
• It also makes the instrumentation more robust, as for the most part, official APIs are used which are less likely to change as hooking into internal methods.
• Debugging is easier as the agent code actually shows up in stack traces and you can just set breakpoints in the agent code.
• Testing is easier as the interceptors can be tested in isolation, without having to actually instrument an application in an integration test.

However, this approach is not always possible or feasible.

• When instrumenting libraries which don't have an official interceptor API, like the Elasticsearch client.
  • Integration tests with different versions of the library/framework are very important in this case.
• When using a wrapper approach, it is now always possible to be allocation free. See #35. So performance-wise it sometimes makes sense to directly instrument the classes to monitor. In these cases, we have to carefully weight performance vs. complexity. Also, as much work as possible should be delegated to code which is not inlined into the instrumented classes.

When being in the realms of BCI, integration tests with different versions of supported frameworks, application servers and Java versions becomes very important. In the worst case, there could even be problems when using different minor versions of an instrumented library.

In the first version, there will only be few supported servlet containers. Tomcat and embedded Tomcat (incl. Spring Boot) will be the first supported server. Others may follow before, but more likely after 1.0 GA.

This document is an attempt to standardize on how backend APM agents buffer data before it’s sent to the APM Server and how the APM agents throttle requests to the APM Server and handle failures related to those requests:
https://docs.google.com/document/d/1a5O7lV1C77iVZSBDwTPbHC8nkiBu0mkaJy74WrVnX_Q/edit#heading=h.xw2nzvd1o6fr

The document is not finalized yet, so this is currently blocked by elastic/apm-server#818

I'm using the java agent to send custom metrics to apm.
However, the instrumentation provides me always a NoopTracer (and consequently a NoopTransaction)

private static final ElasticApm elasticApm = ElasticApm.get();
// elasticApm --> has Noop Tracer

Transaction apmTransaction = elasticApm.startTransaction();
//apmTransaction --> is NoopTransaction

calling the rest of the public api to start/stop transactions/spans just doesn't cause any other effect (as expected since everything is noop)

I've placed src/main/resources/elasticapm.properties file as well as set properties as System properties

Is there anything I'm missing?

https://github.com/elastic/apm-integration-testing

In contrast to #20, the interaction with the real APM server is tested and UI testing will likely be added to that repo.

Ignore URLs which start with a given prefix or use globs (maybe leveraging https://github.com/spring-projects/spring-framework/blob/master/spring-core/src/main/java/org/springframework/util/AntPathMatcher.java). Or why not both?

Add benchmarks to get insight into the overhead in regards to latency, cpu and allocation rate.

In the first iteration, the capture_body option only works for application/x-www-form-urlencoded data. In future iterations, multipart/form-data or arbitrary plain-text data might be supported.

Reading arbitrary text data, probably also multipart data, requires some extra effort as you can't just read the content as plain text from an HttpServletRequest. Also, you can't read the InputStream multiple times from the same request. That means that the agent must not call javax.servlet.ServletRequest.getInputStream.read for example, because it would break the monitored application. So the solution would involve wrapping the request and storing the values in a buffer as they are read by the underlying application.

The artifacts of the subproject https://github.com/elastic/apm-agent-java/tree/master/apm-agent-java should be made available on maven central.

As the URL can contain parameters like /product/42, it is not a good key for grouping together transactions as the transaction overview would "explode".

• Spring MVC 4
• Spring MVC 5
• Spring Boot 2.x
• Spring Boot 1.latest
• JDBC - depending on how instrumentation it is implemented, test different drivers or connection pools

see also https://www.elastic.co/guide/en/apm/agent/python/current/sanitizing-data.html

Implement a Java version of https://github.com/elastic/opbeans

Hello,

I'm trying to send a custom transaction to apm through the Java Agent.
However I'm experiencing problems and for every transaction I close, the server replies http400 with this error on console:

ERROR [request] beater/handlers.go:384 error handling request {"request_id": "3be3e112-466c-463a-b011-61284ed52d31", "error": "Problem validating JSON document against schema: I[#] S[#] doesn't validate with \"transaction#\"\n I[#/transactions/0/context/request] S[#/properties/transactions/items/properties/context/properties/request/required] missing properties: \"method\""}

this is how I'm instrumenting and starting the Agent:

private static ElasticApm elasticApm;

...

Transaction apmTransaction = elasticApm.startTransaction();
apmTransaction.setName("testTransaction");
apmTransaction.setType("test");
apmTransaction.setUser("auto", "[email protected]", "auto");

//Business Code

apmTransaction.close();

while elasticapm.properties looks like this:

service_name=kpicalculator
service_version=1.0.0
application_packages=a.b.c
environment=sand
server_url=http://xxx.xxx.xxx.xxx:8200

Now I'm using apm-server v6.2.4 with docker, however the same issue happens with all the server versions (I've tried from 6.0.0 to 6.2.4)

Am I missing something with the configuration?

Thanks a lot for your help!

Best
Marco

...by not sampling transactions when the queue is almost full and not recording transactions and events at all if the queue is completely full.

See also https://mechanical-sympathy.blogspot.com/2012/05/apply-back-pressure-when-overloaded.html

Hi
I am trying to connect the java-apm-agent to wildfly but notice that wildfly won't run correctly.
If i comment out the -javaagent part wildfly runs normaly.

Caused by: java.lang.NoClassDefFoundError: co/elastic/apm/jdbc/StatementInstrumentation
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266)
        at org.postgresql.jdbc.PgConnection.execSQLQuery(PgConnection.java:455)
        at org.postgresql.jdbc.PgConnection.execSQLQuery(PgConnection.java:449)
        at org.postgresql.jdbc.PgConnection.getTransactionIsolation(PgConnection.java:833)
        at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) [:1.8.0_161]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_161]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161]
        at org.postgresql.ds.PGPooledConnection$ConnectionHandler.invoke(PGPooledConnection.java:333)
        at com.sun.proxy.$Proxy77.getTransactionIsolation(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) [:1.8.0_161]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_161]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161]
        at org.postgresql.xa.PGXAConnection$ConnectionHandler.invoke(PGXAConnection.java:166)
        at com.sun.proxy.$Proxy77.getTransactionIsolation(Unknown Source)
        at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.<init>(BaseWrapperManagedConnection.java:197)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.<init>(XAManagedConnection.java:68)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.newXAManagedConnection(XAManagedConnectionFactory.java:523)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.getXAManagedConnection(XAManagedConnectionFactory.java:496)

standalone.conf

JBOSS_LOG_MANAGER_LIB="$(echo $JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-*.jar)" # resolve logmanager jar
JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$JBOSS_LOG_MANAGER_LIB"
JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager"
JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"


JAVA_OPTS="$JAVA_OPTS -Delastic.apm.service_name=lnk-timetable-wf-b-f11"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.application_packages=se.linkon"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.server_url=http://192.168.72.1:8300"

JAVA_OPTS="$JAVA_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,address=10.0.2.15:9090,server=y,suspend=n"
JAVA_OPTS="$JAVA_OPTS -javaagent:/opt/monitoring/elastic-apm-agent-0.5.1.jar"

Wildfly 8
elastic-apm-agent-0.5.1.jar
jre-1.8.0-openjdk

/Regards Fredrik

Work is underway to add support for Metrics to the API/server: elastic/apm-server#964
Support needs to be added to this agent too.

The default minimum set of metrics should be supported. TBD which additional metrics will be supported for this agent.

Hi all,

it seems that adding only "-javaagent and -Delastic.apm.server_url and -Delastic.apm.service_name" does not work for wildfly 10 standalone. Process starts and after few seconds crashes.
Am I missing any other parameter? I see that similar APM agents for wildfly set also Djboss.modules.system.pkgs.
Any help would be appreciated.
I use the agent already ro some tomcat installations and it really cool!

Add some docs to https://www.elastic.co/guide/en/apm/agent/index.html

• Getting Started
• Configuration
• Servlet Support
• JDBC Support
• Public API

Use the sample_rate configuration option to determine the percentage of sampled transactions. Note that non sampled transactions also do get reported. But they only contain the name and the duration, not the context and no spans.

apm-agent-java is not reading properties from the file and I get the following log message

2018-04-16 07:19:05.387 WARN 65971 --- [ost-startStop-1] c.e.apm.impl.ElasticApmTracerBuilder : Missing required value for configuration option service_name

Below is my elasticapm.properties
elastic.apm.service_name=Elastic-Apm-Demo

SpringBoot 2.0.1
JDK 1.8
APM-AGENT-JAVA 0.1.2