filebeat is not sending logs to logstash about elk-docker HOT 17 CLOSED

Hi,
Unfortunately the ghostbin link appears to be broken.
Could you also post (anonymising IP addresses etc. as needed) any output you have from Filebeat, Filebeat logs, and ELK logs, so that I can attempt to pinpoint what the cause might be? (see https://elk-docker.readthedocs.io/#troubleshooting)

from elk-docker.

I've edited the link

my filebeat.yml - https://ghostbin.com/paste/rrjeh

these are the syslog when I'm restarting filebeat -

Jun 1 17:08:39 sinwars systemd[1]: Stopping filebeat...
Jun 1 17:08:39 sinwars systemd[1]: Stopped filebeat.
Jun 1 17:08:39 sinwars systemd[1]: Started filebeat.

my container logs after creating dummy entry -
https://ghostbin.com/paste/w7ezt

dummy enrty is created successfully and is shown on 9200_search?pretty

the filebeat is not sending any data but registry of the filebeat is filling (/var/lib/filebeat/registry)

my host is localhost

one more thing there is no logstash, elasticsearch or kibana in /var/log

from elk-docker.

Thanks. At a cursory glance, I don't see an obvious reason for your setup not to be work (I'm assuming that Filebeat is running on the same host as ELK, or in a way that it sees the ELK container running on localhost), especially as there are no errors in your Filebeat log… strange…
(Your container logs look suspicious, though, but if the dummy entry is getting in, then that's a cause for a later concern.)

Could you bump your Filebeat logs to debug to see why it isn't sending the logs across to ELK, and let me know what the outcome is?

(On another note, it's somewhat surprising that you're not seeing logstash, elasticsearch or kibana in /var/log as the output of the container is the content of the log files of these services, and the logs appear to be displayed when you're running the container. Could you walk me through how you're docker exec'ing in the container and provide the output of ls -la in the container's /var/log?)

Also, could you dump the complete log of the ELK container start up?

from elk-docker.

Hey @spujadas

my filebeat logs when I set it to debug level https://ghostbin.com/paste/euycf

my elk container startup log https://ghostbin.com/paste/5byrj

Now my elastic search logstash and kibana log is creating in container https://ghostbin.com/paste/sswet

Now please tell where is the mistake?

from elk-docker.

From your Filebeat logs, it would appear that your Filebeat isn't even attempting to send anything anywhere at this point.

Could be a Filebeat configuration issue (and if that's the case then you may want to have a look at the Filebeat forums over at https://discuss.elastic.co/c/beats/filebeat as I'm not a Filebeat expert), but in order to either confirm or rule that out, could you stop Filebeat, delete the registry, and start Filebeat (with logs in debug), and post the resulting logs to see if Filebeat is having trouble connecting to Logstash?

from elk-docker.

Hi

the filebeat logs in debug mode("publish") https://ghostbin.com/paste/rd3mh

I'm still not get logs on http://localhost:9200/_search?pretty

from elk-docker.

Thank you @spujadas

Now filebeat is sending logs but not completely ( 4-5 entries were sent)

why???

from elk-docker.

Any logs you could share (the ones at https://ghostbin.com/paste/rd3mh cut off just before the logs are sent)?

from elk-docker.

Please check this https://ghostbin.com/paste/fm2om

filebeat logs in debug level

from elk-docker.

Thanks. We're back to the situation where Filebeat has nothing to send to Logstash (DBG Flushing spooler because of timeout. Events flushed: 0), would need the registry to be deleted to see what it is sending when it has something to send across.

Having said that, you mentioned that you've seen log entries being sent, so it looks as though the Filebeat-Logstash connection is fine, which suggests that there is a configuration issue somewhere:

• Are you seeing Filebeat harvesting (and therefore sending) fewer entries than you would expect? If so it's probably a Filebeat configuration issue or a log file format issue (or both), and in that case I would recommend asking for support from the Filebeat forum (https://discuss.elastic.co/c/beats/filebeat)
• Is Logstash/Elasticsearch processing or showing fewer entries than the number of entries that have been sent by Filebeat? If so it's likely a Logstash or Elasticsearch configuration issue (e.g. if they can't parse the entries that have been sent by Filebeat), and in that case you want to check in with the Logstash or Elasticsearch forums (https://discuss.elastic.co/c/logstash, https://discuss.elastic.co/c/elasticsearch)

Unfortunately I can't help you more at this point if it is a configuration issue as I'm not a Filebeat or ELK expert (I'm just a guy who packages ELK 😃), so the Filebeat and ELK communities should be your next port of call. Obviously, feel free to point back to this issue to show what has already been tested (which excludes Filebeat-ELK communication issues), and to update this issue if you work out a solution or if it turns out that there is an issue with the ELK image after all. I'll be leaving this issue open for a bit pending further investigations.

from elk-docker.

Hey @spujadas
my logs worked locally.
Now when I'm trying to use the same in virtual machine then the ports are not working on my domain.

Please explain me the setup to be done for virtual machine?

I'm using gcloud.
Thank you
-Sinwar

from elk-docker.

Sure thing, but I'll need more info on your architecture, in particular: where are the log-emitting clients and what are their IP addresses (anonymising them slightly as needed), where are the VMs and what are their IP addresses, where is your Docker, how are you starting the ELK container, what is the container's IP address and how are the container's ports exposed/published, what is currently reachable from where, what are your Filebeat and Logstash logs showing, etc.?

Specifically, I need to understand how your set-up is different from the ones in the image's documentation.

I'm unfamiliar with gcloud (which, for instance, may or may not require explicitly opening the ports listed in the ELK image's documentation) but the image works perfectly (at least) with other cloud providers such as AWS and DigitalOcean, so I don't see why it shouldn't work on gcloud.

(In addition, you may also want to have a look at https://docs.docker.com/engine/userguide/networking/dockernetworks/ for more information on Docker-based networking.)

from elk-docker.

Hi @spujadas
I've followed the instructions of image-docs
Everything is perfect in logs.
But I want to show on cloud ports which is not happening.
Thanks

from elk-docker.

Is it ok I'm trying with docker-machine on gcloud?

from elk-docker.

Can't say for sure as I've never used gcloud, but I really expect that it should somehow work. Perhaps some ports to open on the gcloud firewall (https://cloud.google.com/sdk/gcloud/reference/compute/firewall-rules/create) or something like that.
Could you walk me through how you're creating the VM etc. on gcloud (including the command lines and parameters etc.) so that I can attempt to reproduce on my side?

from elk-docker.

Everything worked.
There were some firewall issues.
Thanks a lot @spujadas

from elk-docker.

Thanks for the update, glad it worked out in the end.

from elk-docker.