User expectation when trying to write to closed indexes about logstash-output-elasticsearch HOT 10 OPEN

@wiibaa thanks for grouping these up!

from logstash-output-elasticsearch.

A closed index is blocked for read/write operations. Given that we have 2 options:

1. Drop events when this error happens
2. Funnel events to a file (and eventually to DLQ)

I am in favor of 2 for now instead of silently dropping events. Thoughts?

I don't think we should reopen closed indexes. Thats beyond the scope of LS, and is an administrative decision in ES

from logstash-output-elasticsearch.

+1 to the DLQ idea

from logstash-output-elasticsearch.

This will be solved when DLQ is implemented.

from logstash-output-elasticsearch.

What is the update on this? It would be pretty useful when importing old logs to an elk stack. I'm currently sorting my logs before inserting them so i can manually close older indices to free up memory.

from logstash-output-elasticsearch.

Is there any update on this issue?

Or is the only work around currently to ensure that the logstash output is not writing to closed indices, either by not closing them or stopping them before they can be written somehow?

from logstash-output-elasticsearch.

Would also appreciate a fix on this, retrying requests indefinitely to closed indexes makes no sense.

from logstash-output-elasticsearch.

Going back to the solutions that @suyograo mentioned...

there is Dead letter queue functionality in Logstash today, and so these can be handled in that way. I would not imagine Logstash doing anything about re-opening closed indices, but enabling Logstash to move documents to the DLQ when failed due to an index being closed is a good idea

from logstash-output-elasticsearch.

Haven't fixed this problem yet?

from logstash-output-elasticsearch.

I'm agreeing with the previously mentioned solution here. Using the DLQ to handle items not indexed is the appropriate way to approach this. Logstash is a stream processor and cannot be expected to re-open an index. While it is not documented here, it's a relatively simple procedure to enable the DLQ in Logstash.

from logstash-output-elasticsearch.