Stream Idle Timer Fires Even When Stream Was Active about envoy HOT 4 OPEN

[2024-05-09 11:15:14.963][11636530][debug][http] [source/common/http/conn_manager_impl.cc:1939] [Tags: "ConnectionId":"0","StreamId" :"13380574157739000950"] Disabling upstream stream due to downstream stream watermark.

For a slow downstream, stream buffer's watermark is high and it takes a long time to recover to low. During this time client is downloading slowly but Envoy won't receive from upstream and won't reset its idle timer.

In my case the client is an IoT device with a 4kbps bandwidth. Envoy's per_connection_buffer_limit_bytes is set to 32 KB. So it takes 32 seconds (32KB / 2 / (4kbps / 8)) to return to low watermark. But if the connection buffer is higher it may take much more time.

from envoy.

The issue can be mitigated by either increasing the stream_idle_timeout or decreasing the per_connection_buffer_limit_bytes, if applicable to the use case.

from envoy.

cc @alyssawilk @mattklein123

from envoy.

I think we could add a config option for not making progress when backed up. I'm not sure we'd want it on by default but we could also runtime guard and add a knob if folks prefer the old behavior

from envoy.