Cannot get Connection from Pool about logstash-filter-elasticsearch HOT 3 CLOSED

Steps to reproduce:

1. Ensure ES is installed > 5.5.x
2. Create a simple index - ensure the doc has @timestamp field

PUT test/test/1
{
  "message":"test",
  "@timestamp":"2017-09-10T23:00:00.000Z"
}

3. Configure LS as follows

input {
    stdin {
    }
}

filter {
        elasticsearch {
              hosts => ["localhost:9200"]
              query => "test"
              index => "test"
              fields => [ ["message", "match"] ]
              user => "elastic"
              password => "changeme"
        }
}


output {
     stdout { codec => dots }
}

4. Ensure workers in LS is > 1 (default if more than 1 core)
5. Pipe any large file though LS i.e.

cat /tmp/nice_big_file | ./logstash -f ../conf/es_filter_issue/logstash.conf

from logstash-filter-elasticsearch.

@gingerwizard

We have discussed this morning and like I've said it appear to be a thread safety issues in the elasticsearch-ruby client that is show under heavy load.

We have two plan for this plugin:

Long term
We are planning to rewrite the plugin using the same client library that we have developed for the elasticsearch output or use the java native client.

Short term
Meanwhile we will implement a fix to work around the thread safety issue, we will create a new connection object for every worker thread. This will waste a bit of resource but it will make sure that every thread have their own view of the world.

from logstash-filter-elasticsearch.

@gingerwizard I have created #77, and it appears to fix the issue you were having. With your use case I can reproduce the error when I apply the PR the error is gone.

I have created #78 as a long term goal.

from logstash-filter-elasticsearch.