Comments (5)
Hey @jgarrettvml ! The error message I see is the following:
: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[10.0.1.190:9200]]
I see it is trying to talk to 10.0.1.190. However, the HELK has an Internal network where all the services (ELK, KAFKA and Analytics) communicate with each other.
networks:
helk:
driver: bridge
ipam:
config:
- subnet: 172.18.0.0/16
and the IP of the ELK stack is 172.18.0.2. That IP is static and is not related to your network. This is hardcoded in your ELK as the docker internal IP. This is why in the Jupyter Notebooks available in the build, I always use 172.18.0.2 when reading from elasticsearch . In the notebook that I assume you were following based on the Issue title, you need to create a RDD by running the following without changing anything. The following should work as is:
es_rdd = sc.newAPIHadoopRDD(
inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat",
keyClass="org.apache.hadoop.io.NullWritable",
valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
conf={
"es.resource" : "logs-endpoint-winevent-sysmon-*/doc",
"es.nodes" : "172.18.0.2"
})
es_rdd.first()
Yours is pointing to 10.0.1.190 which I assume it is the IP you assigned to your HELK to be available in your network. The 2 notebooks available in the build should work without making any changes to them. Let me know if this helps.
from helk.
Wow, thank you for the reply - this fixed it the error. Yeah, I was using the local static - I should have know there was a bridged network, assuming that's what the docker proxy is doing in part.
I'm now getting a "Index [logs-endpoint-winevent-sysmon-*/doc] missing and settings [es.index.read.missing.as.empty] is set to false"
Which I guess is in relation to kibana stating no indices match pattern for "logs-endpoint-winevent-sysmon-*"
I plan to be using several log shipping methods, including filebeat, packetbeat, etc I am assuming this can be configured internally using 172 and externally using my mapped domain:8082 ??
Thanks again, that was a great help...
from helk.
Nicee!!! Yeah if you dont have anything sent to the HELK-Kafka container, then you wont have any data in your sysmon index. At the moment you can send data with Winlogbeat from Windows Computer to the winlogbeat topic. If you are going to send data from filebeat or packetbeat, you will have to create an extra topic and index in the logstash configs. I am slowly trying to cover more data sources and shippers. Let me know how I can help you with your use case. I can help to create the topic and make sure the data makes it all the way to your Elasticsearch. If you are doing internal collection, you still have to point to the internal Kafka instance. If you are sending data to the HELK from your network, you have to still point it to the Kafka instances. Those use your HELK IP and the ports :9092. :9093, :9094. One example is in the winlogbeat config file available in the repo. Check the Kafka section. https://github.com/Cyb3rWard0g/HELK/blob/master/winlogbeat/winlogbeat.yml
from helk.
The HELK:8082 is for Cerebro to use and access your ES server. Thats different. Besides, initial ingestion of data does not happen directly at the elasticsearch level. It happens at the Kafka level. Kafka brokers receive data from producers (winlogbeat), Logstash transforms the data and then it is pushed to elasticsearch.
from helk.
Closing this issue since there is nothing to fix in relation to the title. If you have any other questions feel free to reach out by opening another issue. Happy to help! 👍 Thank you!
from helk.
Related Issues (20)
- Running SIGMA from Kibana
- Consider replacing Logstash by Vector.dev HOT 3
- Documentation leaves NXLog configuration blank
- Installation question
- Missing Dockerfile for otrf/jupyter-hunter docker image HOT 1
- No data in All Miter Att & ck,
- helk-kibana stuck at restarting
- Other systems integration within HELK
- What is the difference between HELK and "Spark + Graphframes + ELK" ?
- Every time I restart the machine the kibana UI is not longer accessible
- How filter kibana' s logs with ossem yamls?
- Need Metric Beats Config
- unable to access 'https://github.com/Cyb3rWard0g/HELK.git/'
- Unable to complete HELK installation HOT 1
- Docker IP subnets
- HELK with Kafka (not winlogbeats but Filebeat)
- Ubuntu 18.04 to 20.04
- EQL and Security options
- SSL error when accesing Spark Master UI
- Elastalert send alert to Slack
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from helk.