bulk_response has nil code? about logstash-output-elasticsearch HOT 16 CLOSED

Hi. I also noticed that. It is probably related to a mix of status codes in the Elastic "POST /_bulk" reply. (i.e.: 200, 201, 400, etc...) I am describing the issues I have with the reporting side of it @ #75

Regards, Andrei

from logstash-output-elasticsearch.

@chenryn can you run Logstash with --debug and provide us the output? Also can you give us more context to reproduce this? when does this happen? immediately with some docs sent to ES? or over time?

from logstash-output-elasticsearch.

I am facing the same problem with logstash 1.5.0.RC2
Every second I get one of these messages
│{:timestamp=>"2015-03-24T16:21:49.644000-0700", :message=>"failed action with response of , dropping action: ["index", {:_id=>nil, :_index=>"logs-2015.03.24.23", :_type=>"hourly"}, #0}, @accessors=#<LogStash::Util::Accessors:0x2ebd1571 @store={"@timestamp"

This behavior started with I updated from 1.4. to 1.5 RC2,

The problem is that I don't get any response code and I don't see any errors in elastic search itself

For more context
I am using the following configurations
elasticsearch {
protocol => "http"
host => [ array of hosts]
workers => 3
flush_size => 10
index => "logs-%{+YYYY.MM.dd.HH}"
Running elastic search 1.4.4

I am tailing a very big files that are updated so frequently

I also enabled debug mode and it doesn't give any more details than this.

from logstash-output-elasticsearch.

@suyograo Run logstash with debug option shows output recieved and then failed action... no more details..

from logstash-output-elasticsearch.

@chenryn @ahmedyossef @chiriaev
I am working on this fix now. It is in fact a bug in how we parse the response from the http ruby client for Elasticsearch.

you can take a look at this diff for now until further testing is made for a proper fix.

master...talevy:fix/77#diff-7a13783c14f0399f855f7894b07132d0R90

you can go ahead and try that fix on line 90 in protocol.rb of the plugin. should retrieve you the correct response code. let me know how that goes if you try it! stay tuned for proper fix very soon!

from logstash-output-elasticsearch.

Hi Tal,
Thanks for taking care of this, I tried the fix but I am still seeing the same error.
Yet I am still new to logstash so I might be missing something

from logstash-output-elasticsearch.

interesting. I am in the process of generating real-world errors from ES to test with..

would you mind placing a print statement and printing bulk_response somewhere above this line and
sharing it? the proper status code should be found there
https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/protocol.rb#L89

from logstash-output-elasticsearch.

Sure, but this will take some time, since need to do it locally only. stay tuned :)

from logstash-output-elasticsearch.

appreciate it @ahmedyossef!

I will fix this up tomorrow when I start my day.

from logstash-output-elasticsearch.

Hi,
I don't know what happen but it looks like it is working now. overnight magic.
now I am getting message=>"retrying failed action with response code: 429", :level=>:warn}
which I guess is related to too many request

Tal, to confirm, Is this the expected behavior after your fix ?

from logstash-output-elasticsearch.

yes. so the fix worked for you?

we retry 429 and 503. and drop all other failures until we have more persistent
queuing in the main logstash pipeline

from logstash-output-elasticsearch.

Yep that mean the fix is working. Thanks for your help.

from logstash-output-elasticsearch.

tks @talevy ,I', getting :message=>"failed action with response of 400, dropping action:. How about output the response body? I don't know the meaning of 400...

from logstash-output-elasticsearch.

@chenryn I completely agree. more information about the error message can be retrieved.

I am going to push in the fix for the response code first, then I can go ahead and tackle providing the error message later on.

here is the initial PR for this fix if you are interested: #87

from logstash-output-elasticsearch.

closed by: #87

from logstash-output-elasticsearch.

new version has been published with this patch.

logstash-output-elasticsearch v0.1.19

from logstash-output-elasticsearch.