TimeoutException when ingesting logs into AI about applicationinsights-java HOT 13 CLOSED

@r-michal-ah Can you increase the Netty HTTP Client with a longer max idle time?
Azure/azure-sdk-for-java#37773 (comment)

from applicationinsights-java.

@r-michal-ah Do you observe other failures after the first attempt to send?

2024-01-13 00:06:16.621+01:00 WARN c.a.m.o.e.i.p.TelemetryPipeline - In the last 5 minutes, the following operation has failed 1 times (out of 10): Sending telemetry to the ingestion service:

from applicationinsights-java.

@r-michal-ah Can you increase the Netty HTTP Client with a longer max idle time? Azure/azure-sdk-for-java#37773 (comment)

let me try it but one question, i am not explicitly using Netty HTTP Client in the code, is there any property/env variable i should set to change default value?

from applicationinsights-java.

@r-michal-ah Do you observe other failures after the first attempt to send?

2024-01-13 00:06:16.621+01:00 WARN c.a.m.o.e.i.p.TelemetryPipeline - In the last 5 minutes, the following operation has failed 1 times (out of 10): Sending telemetry to the ingestion service:

no but

for some pods i can see log ingestion is resumed (some logs are lost but its fine), but for other i can see this exception

java.util.concurrent.TimeoutException: Channel response timed out after 60000 milliseconds.
	at com.azure.core.http.netty.implementation.AzureSdkHandler.responseTimedOut(AzureSdkHandler.java:200)
        at com.azure.core.http.netty.implementation.AzureSdkHandler.lambda$startResponseTracking$2(AzureSdkHandler.java:18)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)

as last one and then nothing is ingested (pods need to be restarted)

from applicationinsights-java.

hello @heyams do you have any suggestions?

from applicationinsights-java.

hello @heyams do you have any suggestions?

I will reach out to Azure SDK to see if there is an env var they support for this case.

from applicationinsights-java.

@r-michal-ah you can change the idle time use this env var called DEFAULT_CONNECTION_IDLE_TIMEOUT (source)

However, timeouts are expected and will get retried. It's better to use a shorter timeout and let it retry sooner.
If you concern about the log, you can change the logging level to ERROR. Hope that helps.

from applicationinsights-java.

@r-michal-ah additionally, can you share applicationinsights.log with me? It's worthy of looking at the full log instead of just that single stack trace. The exception above might not be the actual cause of the failure. Thanks.

from applicationinsights-java.

@r-michal-ah additionally, can you share applicationinsights.log with me? It's worthy of looking at the full log instead of just that single stack trace. The exception above might not be the actual cause of the failure. Thanks.

of course @heyams:

{
  "role": {
    "name": "service-name"
  },
  "sampling": {
    "percentage": 100
  },
  "preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "regexp",
          "metricNames": [
            ".*"
          ]
        }
      }
    ],
    "sampling": {
      "overrides": [
        {
          "telemetryType": "dependency",
          "percentage": 0
        },
        {
          "telemetryType": "request",
          "percentage": 30
        }
      ]
    }
  }
}

from applicationinsights-java.

@r-michal-ah this is applicationinsights.json. I meant the applicationinsights.log.

from applicationinsights-java.

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.

from applicationinsights-java.

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.

from applicationinsights-java.

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.

from applicationinsights-java.