Some logs are lost after fluentd restart about fluency HOT 4 CLOSED

Have you tried https://github.com/komamitsu/fluency#enable-ack-response-mode ?

from fluency.

Yes we did, and it's working as expected. But using the ack response mode makes the sending synchronous and we have to support up to ~20MB log and for our use case, this is not acceptable to block the application during the sending.

from fluency.

In Fluency, buffered data is sent by a background thread, so it shouldn't block the application thread that calls Fluency#emit. But if you faced Fluency's buffer full because the throughput between Fluentd and Fluency didn't catch up with the ingestion throughput from application, the application could be blocked. In this case, it may be worth trying buffer configuration tuning.

From the view point of the throughput between Fluentd and Fluency, ACK response mode may affect the performance. But I think it's a kind of trade off between performance and durability.

from fluency.

Well, actually I did a test again while ack mode is enabled and this is working async. I can't figure out how I concluded it was blocking 🤔

Anyway, I did some tests again while restarting fluentd and no message lost so far.

Thanks for your help !

Thomas

from fluency.