I have some cases where messages I send to the plugin are outputted empty.
The difference I found with the messages that have data is that they are not send in chunks.

Issue seems to be puma related: puma/puma#620

I might have a workaround to force my java app to http 1 and disable chunking but would be nice to have this fixed without a workaround

The current response is always of type text/plain, we should return the same type of the request or follow the content of the http/accept header.

I use http input and output to forward the logs from one system to another.

The output works fairly well (aside from an easy bug to fix). On the other hand, the input plugin doesn't seem to like the directive codec and doesn't parse the logs accordingly.

Client configuration:

        http {
               format => "json"
                http_method => "post"
                url => "http://10.0.0.1:8080/"

Server configuration:

        http {
                threads => 10
               codec => "json"
        }

I made a packet capture to see what's going wrong and I can see the json is encoded correctly (validated) and the request is correct but I get 400 Bad request every single time. When I look on the server, I can see messages like this:

2015-07-30 15:27:55 -0600: HTTP parse error, malformed request (): #<Puma::HttpParserError: Invalid HTTP format, parsing fails.>
2015-07-30 15:27:55 -0600: ENV: {"rack.version"=>[1, 3], "rack.errors"=>#<IO:fd 2>, "rack.multithread"=>true, "rack.multiprocess"=>false, "rack.run_once"=>false, "SCRIPT_NAME"=>"", "CONTENT_TYPE"=>"text/plain", "QUERY_STRING"=>"", "SERVER_PROTOCOL"=>"HTTP/1.1", "SERVER_SOFTWARE"=>"2.11.3", "GATEWAY_INTERFACE"=>"CGI/1.2", "REQUEST_METHOD"=>"POST"}

which seem to indicate that the input http doesn't take the codec into account when decoding.

Disabling json on both sides fixes the issue.

I was looking at other plugins like the logstash-input-elasticsearch, this plugin save the docinfo values into the @metadata field (the user can change the target), should we do the same for this plugin to be consistent?

I used the latest version of 3.0.8 about logstash ,The post request sent from the Java client httpclient is filtered through the logstash and placed in es,But the problem is that the request sent by httpclient enters the logstash at a very slow speed: 5/ s. Through fidder grab bag view, the information is as follows:
GotRequestHeaders: 20:44:57.472
ClientDoneRequest: 20:44:57.472
Determine Gateway: 0ms
DNS Lookup: 0ms
TCP/IP Connect: 0ms
HTTPS Handshake: 0ms
ServerConnected: 20:44:56.272
FiddlerBeginRequest: 20:44:57.472
ServerGotRequest: 20:44:57.472
ServerBeginResponse: 20:44:57.475
GotResponseHeaders: 20:44:57.475
ServerDoneResponse: 20:44:57.675
ClientBeginResponse: 20:44:57.675
ClientDoneResponse: 20:44:57.675
Overall Elapsed: 00:00:00.2030116
Response assembly header cost 200ms? At first I thought it was an httpclient problem, but when I sent messages to the tomcat server that was inside the Intranet, the speed was 500/s (single thread).Next, I use the curl script to send messages to HTTP input on the server where logstash is located, and it can also have 300/s speed.It is important to note that httpclient, logstash, and tomcat services are all in the same Intranet.
logstash.yml：
pipeline.output.workers: 8
pipeline.workers: 10
pipeline.batch.size: 500
Who can help me explain why this happens?

Hi there, referring to #22 I have nginx in front of logstash and my host is always 127.0.0.1. Is it possible to take host from header?

Latest version of logstash,
created logstash config:

input {
  http {
    host => "127.0.0.1"
    port => 5400
    id => "http_input"
    response_headers => {"Content-Type"=>"application/json"}
    codec => json
    password => "example"
  }
}

output {
    stdout{
     codec => rubydebug
    }
}

when I use Postman and set the basic auth header, it works, I see stdout output, but it still passes through when I remove the header and set it to noauth - it works when I change the password on auth as well. is there a test suite I can run to verify?

Hello Everyone ,
I am using Restful api as an input to get data into Elastic search using logstash http plugin.For example we need to extract Walmart data using URL provided by walmart Rest api.As i am run this configuration file indeces is not created in output.Here i am attaching my configuration file
input {
http {
URI => "http://api.walmartlabs.com/v1/search?query=ipod&format=json&apiKey="
}
}

output {
stdout { codec => rubydebug }
elasticsearch {
action => "index"
hosts => "127.0.0.1:9200"
index => "Walmart labs "
workers => 1
document_type => "Walmart data"
document_id => "%{id}"
}
}
As i am new to use these tools i am not able to use http plugin properly and parameters need to passi input .Please guide me how to use it

I've been searching google and not having any luck finding how to set this up properly.

Some documentation would be greatly appreciated.

Thanks
Brian

Hi,
I am using ELK stack for Log Centralisation. I have written an ansible role (logstash-forwarder) which helps my application to send log to ELK using certificate. My ELK is currently supporting lumberjack. One of my application is hosted to Iron.io from where I want to send log to ELK. I have updated both elastic search and lostash version to 2.0. So Now I want to configure my input file to take http and tcp input. My objective is my iron.io app will send log to my ELK using http/tcp/udp and logstash will convert the data and log will be shown in Kibana.

I have generated certificate in my ELK server that I am using to verify my client who is sending log using logstash-forwarder. But for the http plugin, In the input section, I have not provided any authentication.

My current Input file:

input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
http {
port = 5555}
}

and I am sending request using below command:
curl -H "content-type: application/json" -XPUT 'http://elk.startjobs.net:5555' -d '{
"post_date" : "2015-10-29T15:20:34",
"message" : "Nowshad Amin" }' -v

and I am getting the below response:

Adding handle: conn: 0x7f886a813600
Adding handle: send: 0
Adding handle: recv: 0
Curl_addHandleToPipeline: length: 1

• Conn 0 (0x7f886a813600) send_pipe: 1, recv_pipe: 0
  About to connect() to elk.startjobs.net port 5555 (#0)
  Trying 52.91.230.20...
  Connected to elk.startjobs.net (52.91.230.20) port 5555 (#0)

  PUT / HTTP/1.1
  User-Agent: curl/7.30.0
  Host: elk.startjobs.net:5555
  Accept: /
  content-type: application/json
  Content-Length: 67

  upload completely sent off: 67 out of 67 bytes
  < HTTP/1.1 401 Unauthorized
  < Content-Type: text/plain
  < WWW-Authenticate: Basic realm=""
  < Content-Length: 0
  <
  Connection #0 to host elk.startjobs.net left intact

Please take this into your consideration (For lumberjack I am using keys, but for http I am not using any keys and no authentication in my input file). I can send log using lumberjack but getting 401 response while sending log using http

I use logstash http input module with nxlog client. Both applications does not support setting keep-alive timeouts. Nxlog uses hard-coded 300 seconds, logstash input http server uses hard coded 20 seconds. Either this plugin should read keep-alive header from the request or allow users to configure this value. The default 20 seconds is in puma server (configuration persistent_timeout, see https://github.com/puma/puma/blob/42580133a8e9bdb42f55371a7dd0d1dbb02364b9/lib/puma/configuration.rb).

After installing the plugin:
plugin install logstash-plugins/logstash-input-http --verbose
-> Installing logstash-plugins/logstash-input-http...
Trying https://github.com/logstash-plugins/logstash-input-http/archive/master.zip...
Downloading .......DONE
Installed logstash-plugins/logstash-input-http into /usr/local/var/lib/elasticsearch/plugins/logstash-input-http
Identified as a _site plugin, moving to _site structure ...
Installed logstash-plugins/logstash-input-http into /usr/local/var/lib/elasticsearch/plugins/logstash-input-http/_site

When I run the command the plugin isn't found:

logstash -f /Users/michelperez/code/go/src/github.com/mrkaspa/gologstash/logstash-simple.conf
The error reported is:
Couldn't find any input plugin named 'http'. Are you sure this is correct? Trying to load the http input plugin resulted in this error: no such file to load -- logstash/inputs/http

When I use a fresh download of LS 1.5.3 downloaded from: https://www.elastic.co/downloads/logstash
run

bin/plugin install logstash-input-http

and use the config:

input {
  http {
    host => "0.0.0.0"
    port => 7070
    threads => 4
    user => 'something'
    password => 'something'
    additional_codecs => {"application/json" => "json_lines"}
  }
}

I get the following error during startup:

The error reported is:
  uninitialized constant LogStash::Inputs::Http::Base64
org/jruby/RubyModule.java:2733:in `const_missing'
/Users/username/workspace/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-input-http-1.0.1/lib/logstash/inputs/http.rb:82:in `register'
/Users/username/workspace/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:149:in `start_inputs'
org/jruby/RubyArray.java:1613:in `each'
/Users/username/workspace/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:148:in `start_inputs'
/Users/username/workspace/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:81:in `run'
org/jruby/ext/thread/Mutex.java:149:in `synchronize'
/Users/username/workspace/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:81:in `run'
/Users/username/workspace/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/agent.rb:155:in `execute'
/Users/username/workspace/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/runner.rb:91:in `run'
org/jruby/RubyProc.java:271:in `call'
/Users/username/workspace/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/runner.rb:96:in `run'
org/jruby/RubyProc.java:271:in `call'
/Users/username/workspace/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/stud-0.0.20/lib/stud/task.rb:12:in `initialize'

Removing the user and password work fine. Also building the plugin from the github repo and configuring LS 1.5.3 to use the local version works fine as well.

Is there a documentation regarding possible response codes sent by this plugin for various conditions?

Specifically I am looking for the response code generated when an attempt is made to input events to logstash instance when the queue is full (so no more events can be accepted). In practice I see the response code is 429.

I feel this should be documented so that changes between versions can be tracked.

Currently the http input plugin does not forward the senders' IP address, hence it is impossible to get the sender geo-location.

Is there a limit on maximum number of threads for http input plugin? we have a http proxy server sending requests to logstash ( http input plugin) . 1000s of clients are expected to connect to logstash and how does maxclients of http server ( Apache) relate to threads in logstash http input plugin?

Opening this here, as a follow up to: elastic/logstash#2594

Journald has a native way to send logs to a remote system, and it would be nice if Logstash could receive them.

Journald uses HTTP streaming. i.e. Open connection with some headers and the first message, the server then gives "100 Continue" and the next message is sent, ad-infinitum:

https://www.freedesktop.org/software/systemd/man/systemd-journal-upload.html

This sort of streaming does not yet seem easily possible with the HTTP input plugin.

I'm considering using this input, and I wanted to test it before committing to it.

Here's what I found;

Simple Java test:
This HTTP POSTs JSON lines (starting at 300, increasing by 1 until it fails) to the Logstash http input.

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.logging.Level;

public class HTTPInputTest {

    public static void main(String[] args) {
        try {
            String sampleJsonObj = "{\"glossary\":{\"title\":\"example glossary\",\"GlossDiv\":{\"title\":\"S\",\"GlossList\":{\"GlossEntry\":{\"ID\":\"SGML\",\"SortAs\":\"SGML\",\"GlossTerm\":\"Standard Generalized Markup Language\",\"Acronym\":\"SGML\",\"Abbrev\":\"ISO 8879:1986\",\"GlossDef\":{\"para\":\"A meta-markup language, used to create markup languages such as DocBook.\",\"GlossSeeAlso\":[\"GML\",\"XML\"]},\"GlossSee\":\"markup\"}}}}}";
            HttpSender h = new HttpSender();
            for (int num = 300; num < 10_000; num++) {
                String payload = "";
                for (int i = 0; i < num; i++) {
                    payload += sampleJsonObj + "\n";
                }
                byte[] bytes = payload.getBytes("UTF-8");
                System.out.println("Sending " + num + " lines, " + bytes.length + " bytes");
                h.post(bytes);
                Thread.sleep(100);//breathing room
            }
        } catch (Throwable t) {
            java.util.logging.Logger.getLogger(HTTPInputTest.class.getName()).log(Level.SEVERE, "Error!", t);
        }
    }

    private static class HttpSender {

        private final String lsHttpInput = "http://127.0.0.1:8888";

        public void post(byte[] postDataBytes) throws Exception {
            URL url = new URL(lsHttpInput);
            HttpURLConnection conn = (HttpURLConnection) url.openConnection();
            conn.setRequestMethod("POST");
            conn.setRequestProperty("Content-Type", "application/json");
            conn.setRequestProperty("Content-Length", String.valueOf(postDataBytes.length));
            conn.setDoOutput(true);
            conn.getOutputStream().write(postDataBytes);

            try (Reader in = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"))) {
                for (int c; (c = in.read()) >= 0;) {
                    System.out.print((char) c);
                }
            }
            System.out.println("");
        }
    }
}

Logstash 2.3.2 Config:

input { 
    http {
        host => "127.0.0.1"
        port => 8888
    }
} 
output { 
    null{} 
}

Results from running test:

Sending 300 lines, 108300 bytes
ok
Sending 301 lines, 108661 bytes
ok
Sending 302 lines, 109022 bytes
ok
Sending 303 lines, 109383 bytes
ok
Sending 304 lines, 109744 bytes
ok
Sending 305 lines, 110105 bytes
ok
Sending 306 lines, 110466 bytes
ok
Sending 307 lines, 110827 bytes
ok
Sending 308 lines, 111188 bytes
ok
Sending 309 lines, 111549 bytes
ok
Sending 310 lines, 111910 bytes
ok
Sending 311 lines, 112271 bytes
ok
Sending 312 lines, 112632 bytes
ok
Sending 313 lines, 112993 bytes
ok
Sending 314 lines, 113354 bytes
ok
Sending 315 lines, 113715 bytes
ok
Sending 316 lines, 114076 bytes
ok
Sending 317 lines, 114437 bytes
ok
Sending 318 lines, 114798 bytes
ok
Sending 319 lines, 115159 bytes
May 05, 2016 4:52:27 PM HTTPInputTest main
SEVERE: Error!
java.io.IOException: Server returned HTTP response code: 500 for URL: http://127.0.0.1:8888
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1840)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
    at dedot.HTTPInputTest$HttpSender.post(HTTPInputTest.java:44)
    at dedot.HTTPInputTest.main(HTTPInputTest.java:23)

Log output from Logstash:

io/console not supported; tty will not be manipulated
Settings: Default pipeline workers: 4
Pipeline main started
2016-05-05 20:28:31 -0500: Read error: #<Errno::ENOENT: No such file or directory - J:/tmp>
org/jruby/RubyFile.java:902:in `stat'
J:/admin/logstash/logstash-2.3.2/vendor/jruby/lib/ruby/shared/tmpdir.rb:33:in `tmpdir'
org/jruby/RubyArray.java:1613:in `each'
J:/admin/logstash/logstash-2.3.2/vendor/jruby/lib/ruby/shared/tmpdir.rb:30:in `tmpdir'
J:/admin/logstash/logstash-2.3.2/vendor/jruby/lib/ruby/shared/tmpdir.rb:109:in `tmpdir'
J:/admin/logstash/logstash-2.3.2/vendor/jruby/lib/ruby/shared/tmpdir.rb:141:in `create'
org/jruby/ext/tempfile/Tempfile.java:95:in `initialize19'
org/jruby/RubyIO.java:853:in `new'
J:/admin/logstash/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/client.rb:140:in `setup_body'
J:/admin/logstash/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/client.rb:178:in `try_to_finish'
J:/admin/logstash/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/client.rb:101:in `reset'
J:/admin/logstash/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/server.rb:414:in `process_client'
J:/admin/logstash/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/server.rb:270:in `run'
org/jruby/RubyProc.java:281:in `call'
J:/admin/logstash/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/thread_pool.rb:106:in `spawn_thread'

It seems like the "ok" response is hardcoded.
I'm using a cronjob to send nmap data to the http input, similar to what is described in this blog post.
The issue with the ok response is that it causes an email from cron being generated, every time the job runs, which is annoying.
I know that I can redirect the output, but I can imagine that there are other use cases where no response would be appreciated as well.
Proposing to add an option to disable the response.

Starting the plugin with input { http { password => 'hello'} } will not enable auth and also raise no warning or error that both username and password are needed to setup basic auth.

replaces #63

If the http input has codec => json set, and the request does not contain valid json - there should be an option to enable rejecting this message rather than just accepting it and returning a 200 OK (current behavior).

When the default codec (codec => ) is used, the decoding is not thread safe. The default codec should be cloned like the additional_codecs.

Work around is to use additional_codecs instead of relying on the default codec =>.

• Elastic 6.2.3

Logstash config file

input {
	http {
		user => "some"
		password => "one"
		port => 38080
		codec => json
                response_headers => {
                    "Access-Control-Allow-Origin" => "*"
                    "Content-Type" => "text/plain"
                    "Access-Control-Allow-Headers" => "*"
                }
	}
}

This works as expected in non browser environment while in browser when POST request is made (xhr.open("POST", "http://localhost:38080", true, "some", "one")) this results in (Chrome 65.0.3325.181)

OPTIONS http://localhost:38080/ 401 (Unauthorized)
Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Removing user / password fields this works as expected. No difference with other options like Access-Control-Allow-Methods. It just seems to me like this is handled by plugin internally because this is what I see in wireshark:

OPTIONS / HTTP/1.1
Host: localhost:38080
Connection: keep-alive
Access-Control-Request-Method: POST
Origin: http://127.0.0.1:3333
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
Access-Control-Request-Headers: authorization,content-type
Accept: /
DNT: 1
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,hr-HR;q=0.8,hr;q=0.7

HTTP/1.1 401 Unauthorized
Content-Type: text/plain
WWW-Authenticate: Basic realm=""
Content-Length: 0

This should return response_headers so that client can authenticate. I found same working example here, although this also uses ssl which I haven't tried yet (and only curl is used so I can't say if browsers are working):
https://discuss.elastic.co/t/http-input-with-ssl/115338

We need support for gzip in this plugin so that we can minimize the transit size of the request. We can force this off right now, but gzip would be extremely useful.

I think it would be super easy, yet super effective to enable use of this plugin as tracking pixel as well. Currently the plugin returns "OK" if someone sends data to it. If we extend it to support something like response_type with possible values such as plain (returning an empty page as of today) or gif (setting proper Content-Type header and returning a 1-pixel sized GIF image), this should do the trick. Any thoughts on this?

If logstash couldn't index to elasticsearch it returns 200 and "ok".
It should return elasticsearch response. Is it possible ?

For ex: try to send a log with the index name contains any uppercase letter. Logstash will successfully get the log however ElasticSearch will respond as Couldn't index event to Elasticsearch, {:status=>400 .... reason => "Invalid index name ..."

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For all general issues, please provide the following details for fast resolution:

• Version: Logstash 5.0.0 alpha2

If you start logstash with this configuration:

input {
    http {
      port => 31311 # default: 8080
      type => "WEBHOOK"
    }
}

filter {
}

output {
  stdout { codec => rubydebug }
}

And start logstash:

./bin/logstash --http-host=127.0.0.1 --http-port=9601 -f /path/to/logstash.conf

The stats API will return:

curl http://localhost:9601/_node/stats?pretty
{
  "hostname" : "es-rclarke",
  "version" : {
    "number" : "5.0.0-alpha2"
  }
}

rather than the expected jvm / events details.

logstash version: 2.3.2

We have an issue on our production logstash with http input, that http input plugin mixes url arguments from one request with body from another request.

Input config:

http{
    port => 5550
    codec => "line"
    threads => 8
    type => "http"
  }

Filter config:

json {
                source => "message"
                target => "data"
            }
            grok {
                match => { "[headers][request_uri]" => "^/logs\?project=(?<source_project>.+)\&type=(?<source_type>\w+)\&status=(?<source_status>\w+)$" }
            }
            mutate{
                add_field => { "kafka_source" => "%{source_project}.%{source_type}.%{source_status}" }
            }
            mutate {
                add_field => {
                        "[data][source][stream_time]" => "%{+yyyy-MM-dd HH:mm:ss.SSSSSSSSS}"
                }
                rename => {
                    "type" => "[@metadata][type]"
                    "source_project" => "[data][source][project]"
                    "source_type" => "[data][source][type]"
                    "source_status" => "[data][source][status]"
                }
            }
            json_encode {
                source => "data"
                target => "message"
            } 

We reproduced this issue on test logstash instance with same config, and with simple Python client, which sends http POST requests.
Python client:

while True:
        projects = ['project_1', 'project_2']
        for project in projects:
            pack = '{"event_type":"' + project + '"}\n'
            requests.post(
                url='http://localhost:5550/logs?project={}&type=server&status=prod'.format(project),
                data=pack
            )

If we send requests with 1 thread on client side, everything is ok, but with 10+ threads logstash-http-input plugin starts mixing url headers and bodies.

For example, we send requests "{"event_type":"project_1"}\n" with url http://localhost:5550/logs?project=project_1&type=server&status=prod and "{"event_type":"project_2"}\n" with url http://localhost:5550/logs?project=project_2&type=server&status=prod, in normal situation we should see the following results after all logstash processing steps:

{"event_type":"project_2","source":{"project":"project_2","type":"server","status":"prod","stream_time":"2016-09-19 15:12:52.860000000"}}
{"event_type":"project_1","source":{"project":"project_1","type":"server","status":"prod","stream_time":"2016-09-19 15:12:52.860000000"}}

But we see the following error in stdout:

unable to process event {"request_method"=>"POST", "request_path"=>"/logs", "request_uri"=>"/logs?project=project_1&type=server&status=prod", "http_version"=>"HTTP/1.1", "http_host"=>"localhost:5550", "content_length"=>"49", "http_user_agent"=>"python-requests/2.9.1", "http_connection"=>"keep-alive", "http_accept"=>"*/*", "http_accept_encoding"=>"gzip, deflate"}. exception => #<TypeError: can't convert nil into String> {:level=>:error}

And the following wrong result(for example):

{"event_type":"project_2","time":1460937593924,"source":{"project":"project_1","type":"server","status":"prod","stream_time":"2016-09-19 15:33:30.840000000"}}

Looks like something going wrong in highload moments, when all 8 threads in http-input are busy.

https://rubygems.org/search?utf8=%E2%9C%93&query=logstash-input-http

in https://github.com/logstash-plugins/logstash-input-http/blob/master/lib/logstash/inputs/http.rb:144
(for reference - commit 7fb9f85)

we have:

       body = req.delete("rack.input")
        @codecs.fetch(req["content_type"], @codec).decode(body.read) do |event|
          event.set("host", remote_host)
          event.set("headers", req)
          decorate(event)

#HERE:
          queue << event


        end
        ['200', @response_headers, ['ok']]
      rescue => e
        @logger.error(
          "unable to process event.", 

according to my investigation "queue" is a blocking queue

For example. If my elastic does not process some messages eg:

[2017-06-12T09:52:00,488][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429

and we multiply it with dozens servers sending hundreds of http messages we have a situation when
a) logstash stops sending data to backend (that's expected and ok )
b) logstash by means of puma http server accept()'s every connection, and when client closes TCP session puma does not call close() method thuys leaving sockets in CLOSE_WAIT state + massive amount of block'ed waiting threads

Logstash http input should be able to respond with CORS headers :
Access-Control-Allow-Origin: *
So you can send logs via ajax (XHR) http request

Change shutdown sequence to be triggered by plugin.stop instead of ShutdownSignal exception.
Also remove any calls to: shutdown, finished, finished?, running? or terminating?
This depends on elastic/logstash#3895

logstash.conf

output {
elasticsearch {
hosts => "elasticsearch:9200"
}

if [logtype] == "delimitlog" {
    kafka {
            topic_id => "test"
            bootstrap_servers => "10.162.115.99:9092"
    }
}

}

Looks like there are a few fields recorded by this plugin that we have no desire to keep in our storage and will therefore have to write filters to to discard them.

It would be nice to see a flag named retain_http_headers which could be set to true by default for backward compatibility reasons. When false these headers are simply not populated or are automatically filtered out.

http://build-eu-00.elastic.co/view/LS%20Inputs%20New/job/logstash-plugin-input-http-unit/5/jdk=JDK8,nodes=metal-pool/console

In the latest release of puma to 2.16 we have found a regression in the code because one of the option concerning verification of client certificate was not correctly configured. #46

We have unit tests to make sure the SSL options are correctly set but we don't have anything that setup a server and send events through a secured channels. With the addition of #37 We should add more test coverages for all the possible cases.

• Version: logstash6.3.0-1
• Operating System: redhat

Currently we are facing the issue #61
I try to update the plugin with logstash-plugin
and it looks like this now

it didnt remove the old plugin folder and not using the new one.
Then I try to use "logstash-plugin remove logstash-input-http", then it shows not plugin available

[2018-07-18T17:25:32,529][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:cert_info, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any input plugin named 'http'. Are you sure this is correct? Trying to load the http input plugin resulted in this error: no such file to load -- logstash/inputs/http", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:210:in lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:140:in lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/plugin_factory.rb:89:in plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:110:in plugin'", "(eval):8:in <eval>'", "org/jruby/RubyKernel.java:994:in eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:82:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in block in converge_state'"]}

When i put in my config
http {
host => "https://openexchangerates.org/api/currencies.json"
}

i get this error "The error reported is: \n initialize: name or service not known"

help?

I'm using Logstash 1.5.3 and was trying a very simple Logstash configuration. However, when starting up Logstash throws an error: uninitialized constant LogStash::Inputs::Http::Base64.

Here's my config:

input {

  syslog {
    type => "syslog"
    port => 5544
    tags => []
    add_field => {
      "index" => "syslog"
    }
  }

  http {
    type => "http"
    port => 8080
    user => "logstash"
    password => "*******"
    tags => [ "webhook" ]
  }

}

# Add your filters here
filter {

}

output {
  # Trying to debug things? Enable the stdout output plugin
  stdout {
    codec => rubydebug
  }
}

What am I doing wrong here?

Hello,

Recently the server I was outputting logstash was down, but the http plugin kept returning status 200 to the client pushing events, resulting in a loss of data. I just wanted to discuss if it could be interesting to the plugin to return 200 only when the message has successfully passed through the output. This way it would allow clients to capture the error and retry.

If people still needs current behavior, one way to implement it could be adding a configuration option to choose which one is needed.

Thanks.

• Version: 5.6
• Operating System: CentOS 6.X

Customer is asking for ability to limit which ciphers can be used and which protocols.
similar to what is available in Filebeat with cipher_suites and supported_protocols options:
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-output-ssl.html

thanks.

We get following error on stdout when we start Logstash with http input configured:

Listen loop error: #<TypeError: can't convert nil into Integer>
org/jruby/puma/MiniSSL.java:162:in `initialize'
org/jruby/puma/MiniSSL.java:131:in `server'
/opt/software/logstash/product/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/minissl.rb:167:in `accept_nonblock'
/opt/software/logstash/product/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/server.rb:328:in `handle_servers'
org/jruby/RubyArray.java:1613:in `each'

It seems that "verify_mode" in puma is not set which is per default nil.

org/jruby/puma/MiniSSL.java:162:
long verify_mode = miniSSLContext.callMethod(threadContext, "verify_mode").convertToInteger().getLongValue();

This affects the Versions 2.1.3, 2.2.1 and 2.2.2. Version 2.2.0 has not this issue.

Release notes of puma 2.16.

=== 2.16.0 / 2016-01-27

• 7 minor features:
  • Add 'set_remote_address' config option
  • Allow to run puma in silent mode
  • Expose cli options in DSL
  • Support passing JRuby keystore info in ssl_bind DSL
  • Allow umask for unix:/// style control urls
  • Expose old_worker_count in stats url
  • Support TLS client auth (verify_mode) in jruby

It seems the http input unconditionally saves the remote host into the host field after the codecs are applied. This is fine with e.g. a plain codec but problematic with e.g. a json codec where the event payload might contain a host field itself.

I suggest the http input starts behaving like e.g. the tcp input, which only populates the host field if it's unset. Quoting tcp.rb:

event.set(HOST_FIELD, client_address) unless event.get(HOST_FIELD)

(One could argue that event.get(HOST_FIELD) should rather be event.include?(HOST_FIELD) to deal with the corner case that the host field is set to a non-truthy value.)

I am using logstash 1.5.4.

If I post log message with content-type "application/json" everything works fine.
Hovever, if I use content-type "application/json; charset=utf-8" json message is not processed by logstash (probably because of fallback to plain codec).

Content type with charset specified is standard thing, I think it should be supported by logstash.

My pain is that .NET HttpClient class by default add charset to content-type, so I should explicitly customize HttpClient not to add charset to content-type.

It seems like setting json or json_lines for the codec should override the defaults set by additional_codecs.

For example:

http {
    host => "0.0.0.0"
    port => 7070
    threads => 4
    codec => 'json_lines'
  }

If someone makes a post with the content-type: application/json, the code uses the json codec instead of the json_lines codec because it is overwritten by the additional_codecs default option.

This change would make the codec option override the default for additional_codecs.

I would like to raise a question about the behavior of the "verify_mode" setting.

I set "verify_mode" to "peer" expecting that it would check certificates: it does, but if I don't feel like presenting a certificate, I am free to proceed.

Is there a use case where one would check the validity of a client cert, and then let them in anyway, if none is presented?

If not, would it make sense to remove the "force_peer" option and make the functionality of "peer" equal to this?

Possible relevant PR and discussion:

• #37
• b7506fe

we have 3 Logsasth instances with HTTP input plugin running on port 9600. All of the three instances will hang with large amount connections in WAIT_CLOSE status after several days running. Any suggestions to locate the problem?

Logstash 5.6.1