Prometheus prevent sidecar drain about istio HOT 6 OPEN

On drain envoy should be sending connection:close header. Is it not happening or is Prometheus ignoring it?

from istio.

debug logs can answer that

from istio.

debug logs can answer that

Attached are istio-proxy logs, that was terminated at 2023-12-13T11:19:40 and kept running until 2023-12-13T11:22:59.
Its IP is 10.128.6.215, and Prometheus pod's IP is 10.128.7.72.

I see that envoy sends connection: close header to some of the requests, but nothing specific to prometheus.

I do see the following log:

[2023-12-13T11:20:13.736Z] "GET /metrics HTTP/1.1" 503 UF upstream_reset_before_response_started{connection_failure,delayed_connect_error:_111} - "delayed_connect_error:_111" 0 145 0 - "-" "Prometheus/2.29.2" "f0d62e2a-fcd9-4bed-bf38-77ddedf68553" "10.128.6.215:80" "10.128.6.215:80" inbound|80|| - 10.128.6.215:80 10.128.7.72:46672 - default

istio-proxy.log

from istio.

Ok was wondering if the failure makes it no send close; it does appear correct in my test:

/ $curl 10.244.0.12:808 -v;echo
*   Trying 10.244.0.12:808...
* Connected to 10.244.0.12 (10.244.0.12) port 808
> GET / HTTP/1.1
> Host: 10.244.0.12:808
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 152
< content-type: text/plain
< date: Wed, 13 Dec 2023 15:45:59 GMT
< server: istio-envoy
< connection: close
< x-envoy-decorator-operation: echo.default.svc.cluster.local:80/*
<
* Closing connection
upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: 111

I also see a request from promnetheus that gets connection=close (grep S10145678474818188942). this is also the last request from prometheus

from istio.

@howardjohn
So the issue is with Prometheus trying to recreate the connection after receiving connection: close?

from istio.

Yeah which is actually expected probably, prometheus is expected to call during shutdown. I wonder if the timing out the checks to open connections and prometheus scrape interval is aligning or something.

https://preliminary.istio.io/latest/docs/ops/integrations/prometheus/#option-1-metrics-merging probably works around it, but not certain

from istio.