Logstash upstart script not working for some users about logstash HOT 33 CLOSED

I am also getting same Error. running as command line.

/opt/logstash# /app_packages/jdk1.7.0_45/bin/java -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -f conf/logstash.conf
LoadError: no such file to load -- i18n
require at org/jruby/RubyKernel.java:1085
require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
require at /opt/logstash/lib/logstash/JRUBY-6970.rb:27
(root) at /opt/logstash/lib/logstash/runner.rb:50

from logstash.

I tried to reproduce this today (in a vm) but I was unable to proceed due to problems with the vm itself - silly computers!

I'll try again once I've got an ubuntu vm online.

from logstash.

I've run into the same issue.

Installing Logstash 1.4.0 from a .deb, the process is continually restarted by Upstart(new pids can be seen in top). If I run the command manually, it runs fine:

$ sudo -u logstash GEM_HOME=/opt/logstash/vendor/bundle/jruby/1.9 /usr/bin/java -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/
lib /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
Sending logstash logs to /var/log/logstash/logstash.log.

There are no errors in /var/log/logstash/logstash.log or /var/log/upstart/logstash.log.

I've also cut my configuration down to the bare minimum:

$ cat /etc/logstash/conf.d/logstash.conf
input { stdin { type => "stdin-type"} }
output { stdout { } }

from logstash.

Hi,

/opt/logstash/bin/logstash does several settings including the GEM_HOME one.
When launching the java process directly, all those nice things are not set and are causing that issue.

Can you verify if you have the same issue when you use the script instead calling java directly?

Cheers.

from logstash.

Hey @electrical,

sorry, this might have been misleading. I am using the script, but it keeps crashing/respawning in upstart. See the Vagrantfile below to test it yourself (tail /var/log/upstart/logstash.log to see the respawning):

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "helderco/trusty64"
  config.vm.provision :shell, :inline => "cd /tmp && wget 'https://download.elasticsearch.org/logstash/logstash/packages/debian/logstash_1.4.0-1-c82dc09_all.deb'"
  config.vm.provision :shell, :inline => "sudo dpkg -i /tmp/logstash_1.4.0-1-c82dc09_all.deb; sudo apt-get -fy install && sudo dpkg -i /tmp/logstash_1.4.0-1-c82dc09_all.deb"
  config.vm.provision :shell, :inline => "sudo echo 'input { stdin{} }\noutput { stdout{} }\n' > /etc/logstash/conf.d/t.conf"
  config.vm.provision :shell, :inline => "sudo service logstash start && sudo service logstash-web start"
end

If you meant testing without upstart but with the script:

$ sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log

This works…

from logstash.

@alappe ah okay. we haven't done anything to the upstart configs, only the sysv init scripts for 1.4.1

from logstash.

@electrical Too bad. And the release is imminent, so no chance? It's a pity because you cannot use the deb on ubuntu at the moment without adding a separate upstart config (as I have no fix for the issue, I wouldn't know what to override the upstart config with anyway)…

from logstash.

You could remove the upstart config and use the init script, i do the same for the puppet module :-)
I'll make sure we fix the upstart config for 1.4.2 / 1.5.0

from logstash.

Alright… thanks for taking care!

from logstash.

@electrical it looks like the init script is no good either:

$ sudo /etc/init.d/logstash start
 * Starting Logstash Daemon                                                                                                                                                                                 start-stop-daemon: unable to stat  (No such file or directory)

See https://logstash.jira.com/browse/LOGSTASH-2155 for more detail.

from logstash.

@jpb that is most likely the old init script.
https://github.com/elasticsearch/logstash/blob/master/pkg/logstash.sysv is the new one.
Can you try out that one? ( it will be shipped with 1.4.1 )

from logstash.

@electrical On my test box, it starts without error message, but after some seconds the process is gone. It doesn't log into logstash.log, logstash.stdout nor logstash.err

Let's see what @jpb sees…

from logstash.

logstash.sysv from master starts successfully, but Logstash dies after ~10 seconds, and there is nothing in the logs.

from logstash.

Make sure there are no other files in your LS_CONF_DIR or point it directly to your conf file. Had the same issue.

from logstash.

I have the same issue - logstash 1.4.1 installed from elasticsearch.org repositories crashes after a minute and respawns. Adding the GEM_HOME line to /etc/default/logstash solved the problem for me (though its a pain editing this every time I deploy a new machine).

from logstash.

@electrical Is there some progress on the horizon? This makes deploying/using logstash a less fun experience…

from logstash.

@alappe for easier deployment, you could try adding an override file for the upstart job:

env GEM_HOME=yada yada

from logstash.

Same issue as OP. Fresh Ubuntu 14.04, Logstash 1.4.2 from debian repo. Minimal config file. Do a service logstash start. Process starts but nothing is happening and logs are empty. When I do service logstash stop the startup messages show up in the logstash logs. Tried putting GEM_HOME in front of direct java call like OP and that works. We have over 100 servers and growing and every new server shows this error. Not fun. Thanks.

from logstash.

So I tried to run command that is executed as the logstash user:

This way of invoking logstash is fine, but you didn't set any environment variables that the logstash start script sets, so it shouldn't work unless you set all the environment variables - this is why I do not recommend invoking logstash with the java command. Use bin/logstash like the upstart script uses instead?

I will agree that many users are reporting problems, but the "solution" doesn't make sense because this variable is set by bin/logstash.

I will try to reproduce.

from logstash.

Logstash 1.4.2 on Ubuntu 14.04 is working fine for me. I install it, add a config, start the logstash service via upstart, and everything is fine:
https://gist.github.com/jordansissel/de0a1e1dcf650228f2d4

For everyone encountering this problem, can you please attach as a gist:

• a list of all files in /etc/logstash/conf.d/
• the contents of all files in /etc/logstash/conf.d/
• the contents of /var/log/upstart/logstash.log
• the contents of /var/log/logstash/logstash.log
• what command you are using to start logstash

from logstash.

@jordansissel This wasn't meant as a solution but merely a way to debug what could be wrong with the Upstart script…

from logstash.

@jordansissel thanks for the gist. Turns out the Ubuntu image I was on was not clean. My bad. I loaded an actual fresh Ubuntu VM and my configs and it works. So something we're installing or upgrading to our base OS is causing this problem. Thanks for the help. I'll comment back if I track it down.

btw, my /opt/logstash/bin/logstash file doesn't set GEM_HOME.

from logstash.

So, just to make it repeatable:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  installLogstash = <<EOT
  cd /tmp && \
  wget 'https://download.elasticsearch.org/logstash/logstash/packages/debian/logstash_1.4.2-1-2c0f5a1_all.deb' && \
  echo 'fb2c384f61b6834094b31a34d75b58d00c71d81a  logstash_1.4.2-1-2c0f5a1_all.deb' | shasum --check - && \
  sudo apt-get update && \
  sudo apt-get install -y 'java7-runtime-headless' && \
  sudo dpkg -i logstash_1.4.2-1-2c0f5a1_all.deb && \
  echo "input { stdin { } }\n output { stdout { codec => rubydebug } }\n" > /etc/logstash/conf.d/simple.conf && \
  sudo service logstash start && \
  sudo service logstash-web start
EOT

  config.vm.box = "precise64"
  config.vm.provision "shell", inline: installLogstash
end

• a list of all files in /etc/logstash/conf.d/

vagrant@precise64:~$ ls -la /etc/logstash/conf.d/
total 12
drwxrwxr-x 2 root root 4096 Jul  4 01:19 .
drwxrwxr-x 3 root root 4096 Jul  4 01:19 ..
-rw-r--r-- 1 root root   63 Jul  4 01:19 simple.conf
vagrant@precise64:~$ 

• the contents of all files in /etc/logstash/conf.d/
  As written in the Vagrantfile…
• the contents of /var/log/upstart/logstash.log

vagrant@precise64:~$ sudo tail -f /var/log/upstart/logstash.log 
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
…

• the contents of /var/log/logstash/logstash.log

vagrant@precise64:~$ sudo cat /var/log/logstash/logstash.log 
vagrant@precise64:~$ 

• what command you are using to start logstash
  See Vagrantfile…
• the content of /var/log/syslog

vagrant@precise64:~$ sudo tail -f /var/log/syslog 
Jul  4 01:25:13 precise64 kernel: [  424.247709] init: logstash-web main process (7216) terminated with status 1
Jul  4 01:25:13 precise64 kernel: [  424.247723] init: logstash-web main process ended, respawning
Jul  4 01:25:20 precise64 kernel: [  431.201597] init: logstash main process ended, respawning
Jul  4 01:25:37 precise64 kernel: [  447.919032] init: logstash main process ended, respawning
Jul  4 01:25:47 precise64 kernel: [  457.968933] init: logstash main process ended, respawning
Jul  4 01:25:55 precise64 kernel: [  465.992797] init: logstash main process ended, respawning
Jul  4 01:26:03 precise64 kernel: [  473.842204] init: logstash main process ended, respawning
Jul  4 01:26:07 precise64 kernel: [  477.930840] init: logstash-web main process (7532) terminated with status 1
Jul  4 01:26:07 precise64 kernel: [  477.930853] init: logstash-web main process ended, respawning
Jul  4 01:26:19 precise64 kernel: [  489.178478] init: logstash main process ended, respawning
…

from logstash.

@jordansissel with the debian packages (Ubuntu 14.04) logstash 1.4.2-1-2c0f5a1 starts fine, still there's a problem with logstash-web, which is fixed by adding GEM_HOME="/opt/logstash/vendor/bundle/jruby/1.9" to /etc/default/logstash-web (I tested both, upstart and sysv init scripts and adding GEM_HOME fixes things and make kibana to start).

Without GEM_HOME you get this error:

LoadError: no such file to load -- i18n
require at org/jruby/RubyKernel.java:1085
require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
(root) at /opt/logstash/lib/logstash/runner.rb:46

Which seems to be a problem loading the i18n gem in this line:

https://github.com/elasticsearch/logstash/blob/1.4/lib/logstash/runner.rb#L46

Thanks

from logstash.

@alappe I think your simple.conf might be a bit too... simple, for testing purposes. You mentioned above that you're using:

echo "input { stdin { } }\n output { stdout { codec => rubydebug } }\n" > /etc/logstash/conf.d/simple.conf && \

If you temporarily set LS_OPTS="-vv" in your /etc/init/logstash.conf you'll soon notice in /var/log/logstash/logstash.log that the stdin input is shutting down, which causes the process to shut down, which causes upstart to attempt respawn (I suspect).

{:timestamp=>"2014-09-06T17:14:06.864000-0400", :message=>"stdin shutting down.", :level=>:debug, :file=>"logstash/inputs/stdin.rb", :line=>"43"}
{:timestamp=>"2014-09-06T17:14:06.874000-0400", :message=>"Plugin is finished", :plugin=><LogStash::Outputs::Stdout >, :level=>:info, :file=>"logstash/plugin.rb", :line=>"59"}
{:timestamp=>"2014-09-06T17:14:06.877000-0400", :message=>"Pipeline shutdown complete.", :level=>:info, :file=>"logstash/pipeline.rb", :line=>"89"}

If you give it a config with an input that actually stays open ( like the logstash-apache.conf example found here http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash ) the input plugin will stay open and the process stays up just fine.

from logstash.

logstash errors is in /var/log/logstash/logstash.err

from logstash.

1.4.2 doesn't start correctly with the supplied init script on EL6. With the Git version it works. I think the 1.4.3 version should contain the git init script.

from logstash.

I have the same issue on my VM with Ubuntu 14.04

Logstash agent starts
Logstash web starts but eats all CPU ("syslog tcp listener died" and "syslog udp listener died" in log, with this adress "address=>0.0.0.0:514")

After some investigations
==> if I starts logstash web using : "sudo service logstash-web start", that doesn't work and port isn't specified for the process

/usr/bin/java -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb web -l /var/log/logstash/logstash.log

==> if I starts logstash web using : "sudo /etc/init.d/logstash-web start", that works and port is specified for the process (see the end of the line)

/usr/bin/java -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb web -a 0.0.0.0 -p 9292

Strange, isn't it ?
Why ? Because, when we use "service", the system tries to find a file "/etc/init/logstash-web.conf" and uses it to launch logstash.

In the script /etc/init.d/logstash-web, these options are specified and passed to logstash

LS_WEB_ADDRESS="0.0.0.0"
LS_WEB_PORT=9292

In the file /etc/init/logstash-web.conf, there is no option specified

LS_OPTS=""

so, logstash doesn't use 9292 this time

I tried to modify the file /etc/init/logstash-web.conf like that

LS_OPTS="-a 0.0.0.0 -p 9292"

but that didn't work, because there is a bug in the file.

In the few next lines, we have that

LS_OPTS="${LSOPTS} -l ${LS_LOG_FILE}"

and that should be

LS_OPTS="${LS_OPTS} -l ${LS_LOG_FILE}"

(see the underscore between LS and OPTS)

With this 2 modifications, logstash web works like a charm..
(NB : you can specify LS_OPTS="-a 0.0.0.0 -p 9292" in /etc/default/logstash-web)
Hope that helps ;-)

from logstash.

logstash-web is removed in the next release (Logstash 1.5.0)

from logstash.

Do you mean remove /etc/init.d/logstash-web or remove /etc/init/logstash-web.conf or remove both ?
How the web interface will be started in Logstash 1.5.0 ?

Thanking you in advance.

from logstash.

I have found out why logstash doesn't start with rhel6 service. Logstash uses some unusual method to start the service, via :
chroot --userspec $LS_USER:$LS_GROUP
This removes ALL users groups besides LS_GROUP. Logstash then cannot read the logs (permission denied says strace), It however, does NOT LOG anything even in --debug mode. So there is a bug.

I have set LS_GROUP="squid" to my /etc/sysconfig/logstash as a workaround. Unfortunately only 1 group can be used.

from logstash.

What could be the issue here - unable to access "jruby-complete-1.7.11.jar"?
OS: Windows 7 via Cygwin.

[logstash-1.4.2]$ bin/logstash -e 'input { stdin {} } output { stdout {}}'
Error: Unable to access jarfile /home/eausaig/analytics/logstash-1.4.2/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar
[logstash-1.4.2]$ bin/logstash help
Error: Unable to access jarfile /home/eausaig/analytics/logstash-1.4.2/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar

[logstash-1.4.2]$ ls vendor/jar/ -l
total 21188
drwxr-xr-x+ 1 eausaig Domain Users 0 May 15 2014 elasticsearch-1.1.1/
-rwxr-xr-x 1 eausaig Domain Users 21692990 May 15 2014 jruby-complete-1.7.11.jar*

from logstash.

I think I found a solution here: #1524

from logstash.