sohlich / elogrus Goto Github PK
View Code? Open in Web Editor NEWLogrus Hook for ElasticSearch
License: MIT License
Logrus Hook for ElasticSearch
License: MIT License
I'm using ELK (8) but the library returns.
no active connection found: no Elasticsearch node available
When browsing to the elasticsearch endpoint it does return a valid json response.
{
name: "elasticsearch-master-1",
cluster_name: "elasticsearch",
cluster_uuid: "ZqDoaZLyTe6O-rB6HQbscA",
version: {
number: "8.0.0-SNAPSHOT",
build_flavor: "default",
build_type: "docker",
build_hash: "747198ffed9b10273463367cf6ccc7ac5bfa077e",
build_date: "2021-02-12T18:02:06.143916Z",
build_snapshot: true,
lucene_version: "8.8.0",
minimum_wire_compatibility_version: "7.12.0",
minimum_index_compatibility_version: "7.0.0"
},
tagline: "You Know, for Search"
}
Hi Sohlich,
Nice meeting you. I leave you a message on the twitter, but I am not sure you will see the message, because you did not tweet much. So I create a issue here hoping you can contact me, I am using the elogrus, and I want to be maintainer for the project.
Liuzoxan
Heya,
Love the es hook, but errors
are totally unusable, not sure if I configured something wrong.
"error": {
"error": {},
"stack": [
7960421,
7982785,
9199480,
7986632,
8154140,
6701394,
6686002,
4539649
]
},
Is the error I get in elasticsearch, not sure if I missconfigured something ? I'm also using github.com/pkg/errors
and not just errors
Gopkg.in is still serving an old version where the now unavailable github.com/Sirupsen/logrus
is being used with the capital S.
I think a minor version upgrade should not break anything.
This library uses the deprecated elastic search driver by oliviere. The driver supports elastic search up to v7, but will not support later versions. Can we move the library to the official elastic driver for a version 8?
At the moment all logs are stored as the same index.
Be better to create a new index for each day.
e package name is: github.com/Sirupsen/logrus and the path of the logrus lib is github.com/sirupsen/logrus when I build I get an error
There is an inconsistency between these two files.
https://github.com/sohlich/elogrus/blob/v7/hook.go
https://github.com/sohlich/elogrus/blob/v7.0.0/hook.go
In function newHookFuncAndFireFunc(line:108), logrus.Tracelevel is missing from the list and therefore Trace logs are not getting pushed to elastic.
I am able to successfully log data into the Elasticsearch if I do not define an index and let types be dynamically generated when using elogrus.
However with certain fields I would require different types, I have tried creating the index with my own mappings and even the mapping that resulted from being dynamically generated and in both instances elogrus no longer inserted new entries into the index.
How should I go about configuring mappings so that they are compatible with elogrus?
Error message when index generated has identical mapping to defined index:
Failed to fire hook: elastic: Error 400 (Bad Request): Rejecting mapping update to [log] as the final mapping would have more than 1 type: [_doc, log] [type=illegal_argument_exception]
elastic v7 released. elogrus is not ready yet?
Version Info:
go 1.14.4 darwin/amd64
github.com/olivere/elastic/v7 v7.0.17
github.com/sirupsen/logrus v1.6.0
gopkg.in/sohlich/elogrus.v7 v7.0.0
After initializing a new client with elastic.SetInfoLog()
writing to stdout, I got a warning:
Deprecation warning: 299 Elasticsearch-7.8.0-757314695644ea9a1dc2fecd26d1a43856725e65 "[types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id})."
Here's the announcement from Elastic and a related issue: olivere/elastic#1295
It's not a currently breaking things, but it's always nice to get ahead of these things, right?
My first thought was to just delete .Type("log")
here:
Lines 215 to 222 in 7aa9ea8
I'm not sure how to prevent a breaking change, though. If someone is relying on the "log" type, could they just append the type to their index? Ex: {index}/{type}
.
The go library for Elasticsearch is available for version 5, while elogrus still uses v3.
Are there any plans to support v5?
When I run go get gopkg.in/sohlich/elogrus
,
there is an error.
package gopkg.in/sohlich/elogrus: unrecognized import path "gopkg.in/sohlich/elogrus" (parse https://gopkg.in/sohlich/elogrus?go-get=1: no go-import meta tags ())
.
Where is wrong?
set log.SetReportCaller(true)
can displayed log with caller
but elk has no caller infos
This is panic log "health check timeout: no Elasticsearch node available" from code
client, err := elastic.NewClient(elastic.SetURL(elasticHost), elastic.SetSniff(false))
I dont know how can i set Username and Password.
The Hostname info in the logs are getting added with the basic auth credential info .Is there any setting which can add only hostname and hide the credentials getting added to the index?
Hi, I try to build your README.md example code and got the following error,
./main.go:15:37: cannot use client (type *"gopkg.in/olivere/elastic.v5".Client) as type *"github.com/olivere/elastic".Client in argument to elogrus.NewElasticHook
How can I fix it?
In Kibana, @timestamp is the default Time-field name
used for the time filter. Change the JSON output
to match ("@timestamp").
I'm put up the following Pull Request to address this:
Hi, pkg.go.dev maintainer here. We are serving pages for github.com/sohlich/elogrus, which isn't the right path for this module, but someone used it before you added a go.mod file so we can't notice that.
We would like to manually exclude this path from pkg.go.dev (while still keeping the correct gopkg.in path, of course). Do we have the owner's/maintainer's permission to do that?
When I configure logrus in the following manner:
logrus.SetFormatter(&logrus.JSONFormatter{})
logrus.SetReportCaller(true)
logrus.SetOutput(os.Stdout)
And add the elogrus hook:
hook, err := elogrus.NewAsyncElasticHook(client, "localhost", logrus.TraceLevel, index)
logrus.AddHook(hook)
The report caller output is shown in Stdout, however it is not present in Elasticsearch/Kibana.
How can these fields (file and func) be included?
Is there any work going on for elastic 6.0
ECS requires lowercase fields
https://www.elastic.co/guide/en/ecs/current/ecs-guidelines.html
The Message is uppercase and is not recognized by logs watcher
Hello, I'm just opening this issue to let you know, the next release of logrus (v1.2.0) will add a new trace level named Trace below Debug.
Here is the PR for reference sirupsen/logrus#844
You may want to take into account this new level.
Hi,
As per the comment in the logrus hook code, it's expected that hooks will handle their own goroutines. It doesn't seem like you're firing your hook in a goroutine (and I can't see the elastic library doing so), so if ElasticSearch is running slowly for whatever reason, it will block until the requests are complete.
I'm not entirely sure if it's as simple as just setting the client...Do()
call to run in a goroutine or not?
Thanks
Upon me trying to implement this project for myself with my companies elasticsearch, I ran into the issue of not being able to simply use the log.WHATEVER statement with a NewAsyncElasticHook. There was no built in method of keeping track of the number of messages still waiting to get sent, etc. so my project would always finish with logs not being sent.
So I'm fairly new to Go, but I think the way I'd like to tackle this is to implement a background channel that would be created upon the NewAsyncElasticHook that would send the logs that were gathered every 0.5 or whatever and the user could wait with a time limit on the channel to process all the logs and exit out if the time limit was reached.
What do you think? Worth implementing?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.