logstash / cookbook Goto Github PK

Instead of code being included in the page, the include_code block is visible

example in http://cookbook.logstash.net/recipes/rsyslog-agent/

configure rsyslog
The rsyslog daemon is useful for both taking local syslog messages as well as for pulling logs from files.
To watch files with rsyslog, you want to use the imfile rsyslog module.
For example, let’s say we want to forward local syslog as well as apache and mysql log files to logstash.
{% include_code rsyslog.conf %}

configure logstash
Now, logstash needs to be told to accept syslog input. This is simple enough. Here is an example config that takes syslog and emits it to stdout:
{% include_code logstash.conf %}

when :hover on a featurelet button, the border fades in smoothly over 0.5s, but the background gradient snaps from A to B instantly, creating a distracting flash effect.

I am facing a, possible, issue with logstash 1.2.1. Sporadically, when starting it up, I receive the following error:

java -jar logstash-1.2.1-flatjar.jar agent -f conf/logstash.conf -- web
----------
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.2.1/plugin-milestones {:level=>:warn}
+---------------------------------------------------------+
| An unexpected error occurred. This is probably a bug.   |
| You can find help with this problem in a few places:    |
...
The error reported is: 
  pattern %{GREEDYDATA:message_id} not defined

My configuration file looks like this:

input {
       file {
        type => "my-component"
        path => [ "/path/to/my/log/directory/*.log" ]
        add_field => [ "API", "mycomponent"]
    }
      ...
}
filter {

    if [type] == "my-component" { 
        grok {  
            match => [ "message", "(%{GREEDYDATA:message_id}) %{TIMESTAMP_ISO8601:log_timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message_remainder}" ]
            add_field => ["raw_message", "%{@message}"]
        }
        mutate {
            replace => ["message", "%{message_remainder}" ]
        }   
        multiline {
            pattern => "^\s"
            what => "previous"
        }
    }
}

output {
    elasticsearch { embedded => true }
}

It works if I kill it and start it again.

You may already be aware of this, but cookbook.logstash.net just returns a 404 not found page.

The website is not there. Bummer.

The logstash agent Cpu usage :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27729 logstash 20 0 1643m 221m 16m S 101.1 1.4 0:36.80 java

Thank you for your support!

There is "Server not found" error from yesterday.

Host lookup results using Google's nameservers:

dig @google-public-dns-a.google.com cookbook.logstash.net

; <<>> DiG 9.7.0-P1 <<>> @google-public-dns-a.google.com cookbook.logstash.net
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 245
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;cookbook.logstash.net.     IN  A

;; AUTHORITY SECTION:
logstash.net.       979 IN  SOA ns1.dreamhost.com. hostmaster.dreamhost.com. 2012110200 20085 1800 1814400 14400

;; Query time: 97 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sun Nov  4 15:48:46 2012
;; MSG SIZE  rcvd: 103

Hi,
timestamp format seems wrong, elasticsearch doesn't like it.
This is the recipe:
LogFormat "{ "@timestamp": "%{%Y/%m/%dT%H:%M:%S%z}t"
but shouldn't it be
LogFormat "{ "@timestamp": "%{%Y-%m-%dT%H:%M:%S%z}t" ?

I like this cookbook, but also would love a working example of a filter allowing to skip all 200 statuses. I tried but couldn't succeed in grepping the status field...

thanks

hi :)
how about changing the whole theme that is matches logstash.net and integrates seamless into it?
i could give it a try if you don't actually want it to look that different and dark ^^

When I follow the instructions here: http://cookbook.logstash.net/recipes/rsyslog-agent/

I get this error:

{"message":"Using experimental plugin 'syslog'. This plugin is untested and may change in the future. For more information about plugin statuses, see http://logstash.net/docs/1.1.1-pre/plugin-status ","level":"warn"}
{"message":"Missing required parameter 'type' for input/syslog","level":"error"}
{"message":"Config validation failed.","level":"error"}

• Use conditionals
• Remove deprecated configuration
• Add a test

Can someone post here the error messages (if any) that are being generated so I can help research this gh-pages issue

thanks

We noticed a problem with our logstash-client whereby it didn't seem to be tailing over 50% of the log files that we'd configured it to watch.

Eventually we tracked it down to a problem in ruby-filewatcher in the watch.rb file in '_discover_file'.

On the first line of that function is a call to 'Dir.glob' which was returning an empty array even for files which we know exist and are accessible by the logstash user.

We're running logstash 1.1.1 (which internally uses the jRuby 1.6.7 interpretor, but on trying out 'Dir.glob' on a local instance of that interpretor we couldn't replicate the problem).

We wrote a hack to help us get around it here -> alphagov/ruby-filewatch@9daaab8 - but presumably there's a cleaner solution to this.

Has anyone else experienced this problem?

Mark

The site http://cookbook.logstash.net (linked from this repo's description) is no longer resolving. It's been removed from DNS.

The query '*' resulted the following error:

org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[yOyNYFaTYaQfkKXq7QaRw][mcare][1]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[mcare][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[mcare][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[03aCVBiST0in9Nvuhpd4Fw][knowledge][3]: RemoteTransportException[[es-1][inet[/10.1.3.9:9300]][search/phase/query]]; nested: SearchParseException[[knowledge][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[knowledge][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][graylog2][0]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[graylog2][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[graylog2][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][mcare][0]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[mcare][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[mcare][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[03aCVBiST0in9Nvuhpd4Fw][knowledge][1]: RemoteTransportException[[es-1][inet[/10.1.3.9:9300]][search/phase/query]]; nested: SearchParseException[[knowledge][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[knowledge][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[03aCVBiST0in9Nvuhpd4Fw][graylog2][3]: RemoteTransportException[[es-1][inet[/10.1.3.9:9300]][search/phase/query]]; nested: SearchParseException[[graylog2][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[graylog2][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][graylog2][4]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[graylog2][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[graylog2][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][mcare][3]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[mcare][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[mcare][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][mcare][4]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[mcare][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[mcare][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[03aCVBiST0in9Nvuhpd4Fw][knowledge][2]: RemoteTransportException[[es-1][inet[/10.1.3.9:9300]][search/phase/query]]; nested: SearchParseException[[knowledge][2]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[knowledge][2]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][graylog2][1]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[graylog2][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[graylog2][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][knowledge][0]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[knowledge][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[knowledge][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[03aCVBiST0in9Nvuhpd4Fw][knowledge][4]: RemoteTransportException[[es-1][inet[/10.1.3.9:9300]][search/phase/query]]; nested: SearchParseException[[knowledge][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[knowledge][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][graylog2][2]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[graylog2][2]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[graylog2][2]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][mcare][2]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[mcare][2]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[mcare][2]: query[ConstantScore(NotDeleted(:*))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }

if you don't explicitely set it the embedded elasticsearch tries to create it's data dir in the logstash user's dir (which happened for me because I had created the UID with a home dir)

better to explicitly set it in the upstart config IMO

http://cookbook.logstash.net/recipes/windows-service/

i.e. something like https://github.com/logstash/logstash/wiki/Testing-your-filters but not isolated to filters.

Rather an integrated end-to-end test of loading a custom (output) plugin that provides a set of sample files, specifies inputs, some filters and an output parameters for that plugin.

Ideally it would include stubbing out the backend implementation of the output so that the test can ensure that the filter+output is invoking the right sequence of operations on the backend.

I realise this might be a lot to ask for - portions would still be useful, especially the basics of an integrated test.

Thanks, M.

Hi,

Using logstash on a RHEL 6.4 host, In the logstash.sh script do_stop() function, there's the line:
checkpid $pid && sleep $delay &&

but $delay is not defined and we get:

sleep: missing operand
Try `sleep --help' for more information.

You can check the killproc function in /etc/init.d/functions as an example.