Comments (17)
Hi,
Unfortunately the ghostbin link appears to be broken.
Could you also post (anonymising IP addresses etc. as needed) any output you have from Filebeat, Filebeat logs, and ELK logs, so that I can attempt to pinpoint what the cause might be? (see https://elk-docker.readthedocs.io/#troubleshooting)
from elk-docker.
I've edited the link
my filebeat.yml - https://ghostbin.com/paste/rrjeh
these are the syslog when I'm restarting filebeat -
Jun 1 17:08:39 sinwars systemd[1]: Stopping filebeat...
Jun 1 17:08:39 sinwars systemd[1]: Stopped filebeat.
Jun 1 17:08:39 sinwars systemd[1]: Started filebeat.
my container logs after creating dummy entry -
https://ghostbin.com/paste/w7ezt
dummy enrty is created successfully and is shown on 9200_search?pretty
the filebeat is not sending any data but registry of the filebeat is filling (/var/lib/filebeat/registry)
my host is localhost
one more thing there is no logstash, elasticsearch or kibana in /var/log
from elk-docker.
Thanks. At a cursory glance, I don't see an obvious reason for your setup not to be work (I'm assuming that Filebeat is running on the same host as ELK, or in a way that it sees the ELK container running on localhost), especially as there are no errors in your Filebeat log… strange…
(Your container logs look suspicious, though, but if the dummy entry is getting in, then that's a cause for a later concern.)
Could you bump your Filebeat logs to debug
to see why it isn't sending the logs across to ELK, and let me know what the outcome is?
(On another note, it's somewhat surprising that you're not seeing logstash
, elasticsearch
or kibana
in /var/log
as the output of the container is the content of the log files of these services, and the logs appear to be displayed when you're running the container. Could you walk me through how you're docker exec
'ing in the container and provide the output of ls -la
in the container's /var/log
?)
Also, could you dump the complete log of the ELK container start up?
from elk-docker.
Hey @spujadas
my filebeat logs when I set it to debug level https://ghostbin.com/paste/euycf
my elk container startup log https://ghostbin.com/paste/5byrj
Now my elastic search logstash and kibana log is creating in container https://ghostbin.com/paste/sswet
Now please tell where is the mistake?
from elk-docker.
From your Filebeat logs, it would appear that your Filebeat isn't even attempting to send anything anywhere at this point.
Could be a Filebeat configuration issue (and if that's the case then you may want to have a look at the Filebeat forums over at https://discuss.elastic.co/c/beats/filebeat as I'm not a Filebeat expert), but in order to either confirm or rule that out, could you stop Filebeat, delete the registry, and start Filebeat (with logs in debug
), and post the resulting logs to see if Filebeat is having trouble connecting to Logstash?
from elk-docker.
Hi
the filebeat logs in debug mode("publish") https://ghostbin.com/paste/rd3mh
I'm still not get logs on http://localhost:9200/_search?pretty
from elk-docker.
Thank you @spujadas
Now filebeat is sending logs but not completely ( 4-5 entries were sent)
why???
from elk-docker.
Any logs you could share (the ones at https://ghostbin.com/paste/rd3mh cut off just before the logs are sent)?
from elk-docker.
Please check this https://ghostbin.com/paste/fm2om
filebeat logs in debug level
from elk-docker.
Thanks. We're back to the situation where Filebeat has nothing to send to Logstash (DBG Flushing spooler because of timeout. Events flushed: 0
), would need the registry to be deleted to see what it is sending when it has something to send across.
Having said that, you mentioned that you've seen log entries being sent, so it looks as though the Filebeat-Logstash connection is fine, which suggests that there is a configuration issue somewhere:
- Are you seeing Filebeat harvesting (and therefore sending) fewer entries than you would expect? If so it's probably a Filebeat configuration issue or a log file format issue (or both), and in that case I would recommend asking for support from the Filebeat forum (https://discuss.elastic.co/c/beats/filebeat)
- Is Logstash/Elasticsearch processing or showing fewer entries than the number of entries that have been sent by Filebeat? If so it's likely a Logstash or Elasticsearch configuration issue (e.g. if they can't parse the entries that have been sent by Filebeat), and in that case you want to check in with the Logstash or Elasticsearch forums (https://discuss.elastic.co/c/logstash, https://discuss.elastic.co/c/elasticsearch)
Unfortunately I can't help you more at this point if it is a configuration issue as I'm not a Filebeat or ELK expert (I'm just a guy who packages ELK 😃), so the Filebeat and ELK communities should be your next port of call. Obviously, feel free to point back to this issue to show what has already been tested (which excludes Filebeat-ELK communication issues), and to update this issue if you work out a solution or if it turns out that there is an issue with the ELK image after all. I'll be leaving this issue open for a bit pending further investigations.
from elk-docker.
Hey @spujadas
my logs worked locally.
Now when I'm trying to use the same in virtual machine then the ports are not working on my domain.
Please explain me the setup to be done for virtual machine?
I'm using gcloud.
Thank you
-Sinwar
from elk-docker.
Sure thing, but I'll need more info on your architecture, in particular: where are the log-emitting clients and what are their IP addresses (anonymising them slightly as needed), where are the VMs and what are their IP addresses, where is your Docker, how are you starting the ELK container, what is the container's IP address and how are the container's ports exposed/published, what is currently reachable from where, what are your Filebeat and Logstash logs showing, etc.?
Specifically, I need to understand how your set-up is different from the ones in the image's documentation.
I'm unfamiliar with gcloud (which, for instance, may or may not require explicitly opening the ports listed in the ELK image's documentation) but the image works perfectly (at least) with other cloud providers such as AWS and DigitalOcean, so I don't see why it shouldn't work on gcloud.
(In addition, you may also want to have a look at https://docs.docker.com/engine/userguide/networking/dockernetworks/ for more information on Docker-based networking.)
from elk-docker.
Hi @spujadas
I've followed the instructions of image-docs
Everything is perfect in logs.
But I want to show on cloud ports which is not happening.
Thanks
from elk-docker.
Is it ok I'm trying with docker-machine on gcloud?
from elk-docker.
Can't say for sure as I've never used gcloud, but I really expect that it should somehow work. Perhaps some ports to open on the gcloud firewall (https://cloud.google.com/sdk/gcloud/reference/compute/firewall-rules/create) or something like that.
Could you walk me through how you're creating the VM etc. on gcloud (including the command lines and parameters etc.) so that I can attempt to reproduce on my side?
from elk-docker.
Everything worked.
There were some firewall issues.
Thanks a lot @spujadas
from elk-docker.
Thanks for the update, glad it worked out in the end.
from elk-docker.
Related Issues (20)
- Please update to 7.16.3 HOT 1
- Can't get Elk started HOT 7
- cannot add login page to kibana HOT 1
- How to use environment variable in 30-output.conf file HOT 2
- ELK fails to start on MAC M1 HOT 8
- Setting up APM question HOT 4
- Question: user authentication for https HOT 1
- Error in Security section HOT 1
- Issues installing on TrueNAS Scale HOT 1
- Update ELK to latest version (currently 8.3.3) HOT 4
- Issues running on AWS Fargate HOT 2
- Add sample docker-compose.yml with persistance + traefik configuration HOT 1
- example using image never starts as elasticsearch doesn't start HOT 1
- Kibana refuses connection, nothing in logs HOT 2
- filebeat x509 certificate signed by unknown authority when calling api endpoint HOT 1
- Kibana enrollement token
- Update ELK to 8.9.0 HOT 1
- Kibana 8.11.1 Security Update (ESA-2023-25) HOT 1
- want to move data from this running stack to another machine (arm64-amd64) how to
- how to disable the features that require a licence.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elk-docker.