evalphobia / logrus_fluent Goto Github PK

View Code? Open in Web Editor NEW

45.0 2.0 21.0 41 KB

fluentd hooks for logrus

License: Apache License 2.0

Go 98.73% Makefile 1.27%

logrus_fluent's People

Contributors

Stargazers

Watchers

Forkers

troq morpheusxaut thinkong echupriyanov shinji62 stverhae reproio go-zen-chu gcxrun bluearchive hakanmemisoglu macrat keymantics onrik chenmin1992 i-ma moj361 werunclub umbocv

logrus_fluent's Issues

New Trace log level in logrus

Hello, I'm just opening this issue to let you know, the next release of logrus (v1.2.0) will add a new trace level named Trace below Debug.
Here is the PR for reference sirupsen/logrus#844
You may want to take into account this new level.

Fire() method do not write message to fluent if hook.tag is set

setMessage called only if hook.tag == nil

panic: Use of unexported embedded struct

Go Panic StackTrace

panic: reflect.Value.Interface: cannot return value obtained from unexported field or method

goroutine 1 [running]:
reflect.valueInterface(0x1138ce0, 0xc4200ba080, 0xd4, 0xc4200ac001, 0x0, 0x0)
	/usr/local/Cellar/go/1.8.1/libexec/src/reflect/value.go:936 +0x1a5
reflect.Value.Interface(0x1138ce0, 0xc4200ba080, 0xd4, 0xa, 0x11fdc20)
	/usr/local/Cellar/go/1.8.1/libexec/src/reflect/value.go:925 +0x44
github.com/evalphobia/logrus_fluent.converFromStruct(0x1147100, 0xc4200ba080, 0x116003f, 0x6, 0xc4200ba080, 0x113e020)
	/Users/zirkome/.go/src/github.com/evalphobia/logrus_fluent/reflect.go:66 +0x3c2
github.com/evalphobia/logrus_fluent.ConvertToValue(0x113e020, 0xc4200ba060, 0x116003f, 0x6, 0xc4200ba060, 0x98)
	/Users/zirkome/.go/src/github.com/evalphobia/logrus_fluent/reflect.go:14 +0x127
github.com/evalphobia/logrus_fluent.convertFromMap(0x11390e0, 0xc4200b2150, 0x15, 0x116003f, 0x6, 0xc42004fc38, 0xc4200c0240)
	/Users/zirkome/.go/src/github.com/evalphobia/logrus_fluent/reflect.go:32 +0x219
github.com/evalphobia/logrus_fluent.ConvertToValue(0x11390e0, 0xc4200b2150, 0x116003f, 0x6, 0x122ac01, 0x1148700)
	/Users/zirkome/.go/src/github.com/evalphobia/logrus_fluent/reflect.go:16 +0x1b1
github.com/evalphobia/logrus_fluent.(*FluentHook).Fire(0xc4200ae050, 0xc4200ae140, 0x0, 0x0)
	/Users/zirkome/.go/src/github.com/evalphobia/logrus_fluent/fluent.go:148 +0x31a
github.com/Sirupsen/logrus.LevelHooks.Fire(0xc42007a180, 0xc4255b7b02, 0xc4200ae140, 0xd, 0xc42004fe10)
	/Users/zirkome/.go/src/github.com/Sirupsen/logrus/hooks.go:28 +0x8b
github.com/Sirupsen/logrus.Entry.log(0xc4200720f0, 0xc4200b2120, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4200ac002, ...)
	/Users/zirkome/.go/src/github.com/Sirupsen/logrus/entry.go:96 +0x109
github.com/Sirupsen/logrus.(*Entry).Error(0xc4200ae0f0, 0xc42004ff20, 0x1, 0x1)
	/Users/zirkome/.go/src/github.com/Sirupsen/logrus/entry.go:156 +0xaa
main.logging()
	/Users/zirkome/work/src/github.com/myorg/myrepo/main.go:25 +0xf3
main.main()
	/Users/zirkome/work/src/github.com/myorg/myrepo/main.go:20 +0xc2
exit status 2

The code to reproduce this is here: https://gist.github.com/kokaz/15438e811565815c6648a8723e877aee

The cause is related to pkg/errors using an anonymous embedded struct to store stack trace:

type withStack struct {
    error
    *stack
}

This bug was introduced in #13.

The root cause IMO is due to the way reflection is done as encoding/json and encoding/xml does handle correctly the reflection in this case.

unknown field 'RequestAck'

I'm getting the following error in my build. It looks like fluent.Config definition has changed.

/go/pkg/mod/github.com/evalphobia/[email protected]/config.go:53:3: \
  unknown field 'RequestAck' in struct literal of type fluent.Config

Current packages:

% grep 'fluent\|logrus' go.mod
        github.com/evalphobia/logrus_fluent v0.5.4
        github.com/fluent/fluent-logger-golang v0.0.0-20180420045249-90f0f0270701 // indirect
        github.com/sirupsen/logrus v1.2.0

Update logrus imports

The author changed the account's name to lower case.

Please fix

--> evalphobia/logrus_fluent/config.go:49:8 : unknown field 'Async' in struct literal of type fluent.Config

Async Connect not used in the Fluent hook.

Seems like the AsyncConnect flag is not used in the fluenthook.

hook := &FluentHook{
		Fluent:       fd,
		conf:         conf,
		levels:       conf.LogLevels,
		ignoreFields: make(map[string]struct{}),
		filters:      make(map[string]func(interface{}) interface{}),
	}

Are you planning to add this in the coming release?

Seems td-agent doesn't identify logging data

Enviroment:

cat /opt/td-agent/version-manifest.txt
td-agent 3.4.1

Component          Installed Version                          Version GUID                                                              Overridden From
----------------------------------------------------------------------------------------------------------------------------------------------------------
cacerts            2018-12-05                                 sha256:4d89992b90f3e177ab1d895c00e8cded6c9009bec9d56981ff4f0a59e9cc56d6
config_guess       master                                     git:84f04b02a7e2fc8eaa9d52deee5f6d57b06fe447
fluentd            2903f22d7b6485a97b47c4386ab6c6e86d269a15   git:2903f22d7b6485a97b47c4386ab6c6e86d269a15                              master
jemalloc           4.5.0                                      sha256:9409d85664b4f135b77518b0b118c549009dc10f6cba14557d170476611f6780   4.2.1
libedit            20120601-3.0                               md5:e50f6a7afb4de00c81650f7b1a0f5aea
libffi             3.2.1                                      md5:83b89587607e3eb65c70d361f13bab43
libiconv           1.15                                       sha256:ccf536620a45458d26ba83887a983b96827001e92a13847b45e4925cc8913178
liblzma            5.2.3                                      md5:ef68674fb47a8b8e741b34e429d86e9d
libtool            2.4                                        md5:b32b04148ecdd7344abc6fe8bd1bb021
libxml2            2.9.9                                      sha256:94fb70890143e3c6549f265cee93ec064c80a84c42ad0f23e85ee1fd6540a871
libxslt            1.1.30                                     sha256:ba65236116de8326d83378b2bd929879fa185195bc530b9d1aba72107910b6b3
libyaml            0.1.7                                      sha256:8088e457264a98ba451a90b8661fcb4f9d6f478f7265d48322a196cec2480729
makedepend         1.0.5                                      md5:efb2d7c7e22840947863efaedc175747
ncurses            5.9                                        md5:8cb9c412e5f2d96bc6f459aa8c6282a1
nokogiri           3.4.1
openssl            1.0.2r                                     sha256:ae51d08bba8a83958e894946f15303ff894d75c2b8bbd44a852b64e3fe11d0d6
pkg-config-lite    0.28-1                                     md5:61f05feb6bab0a6bbfab4b6e3b2f44b6
postgresql         9.6.9                                      sha256:b97952e3af02dc1e446f9c4188ff53021cc0eed7ed96f254ae6daf968c443e2e   9.2.10
preparation        1.0.0
ruby               2.4.6                                      sha256:de0dc8097023716099f7c8a6ffc751511b90de7f5694f401b59f2d071db910be
rubygems           3.4.1
td                 3.4.1
td-agent           3.4.1
td-agent-cleanup   3.4.1
td-agent-files     3.4.1
util-macros        1.18.0                                     md5:fd0ba21b3179703c071bbb4c3e5fb0f4
version-manifest   0.0.1
xproto             7.0.25                                     md5:a47db46cb117805bd6947aa5928a7436
zlib               1.2.11                                     sha256:c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1

Reproduce:

// setup fluent hook
if *fluent {
    hook, err := logrus_fluent.NewWithConfig(logrus_fluent.Config{
        Host: *fluenthost,
        Port: *fluentport,
    })
    if err != nil {
        logrus.Panic(err)
    }
    hook.SetLevels([]logrus.Level{
        logrus.InfoLevel,
    })
    hook.SetTag(*fluenttag)
    logrus.AddHook(hook)
}
// logging
logrus.WithFields(logrus.Fields{
    "clienthost":  r.stream.net.Src(),
    "clientport":  r.stream.transport.Src(),
    "serverhost":  r.stream.net.Dst(),
    "serverport":  r.stream.transport.Dst(),
    "username":    r.session.username,
    "database":    r.session.database,
    "query":       r.session.query,
    "startTime":   r.session.queryStartTime,
    "elapsedTime": r.session.queryElapsedTime.Nanoseconds() / 1000000, // convert to milliseconds
    "endTime":     r.session.queryEndTime,
}).Infof("Query")

td-agenet configuration:

## mysql.sniffer
<source>
  @type tcp
  tag mysql.sniffer # required
  <parse>
    @type none
  </parse>
  port 24224
</source>

<match mysql.sniffer**>
  @type copy
  <store> # debug purpose only
    @type stdout
  </store>
</match>

I captured traffic flow, and saw data had arrived, but there is nothing new in fluent log.
I use echo 'msg:QUIT' | nc 172.31.90.57 24224 to test, there is something in td-agent log.

"tag" field is removed from passed entry.

I'm trying to use preset *logrus.Entry with a "rag" field, so I don't have to specify tag on each log message.
Example:

    hook := logrus_fluent.NewHook("fluent-es-dev.cybertonica.aws", 24224)
    myLogger := logrus.New()
    myLogger.Hooks.Add(hook)
    myLogger.Formatter = new(logrus.JSONFormatter)
    myLogger.Level = logrus.DebugLevel
    myLog := myLogger.WithFields(logrus.Fields{"tag": "debug.scratch.test", "host": "iMac-eric"})

    myLog.Debug("First message")
    myLog.Debug("Second message")

First message is correctly tagged and sent to Fluentd, but second one is un-tagged.

I suggest to make a copy of entry's data in Fire call work with it, but not with original entry passed

Why is NewHook deprecated?

I understand that it would mean that by default the hook would open a new connection every time and it's not desired. What I specifically miss is the ability to open an Async connection to fluent which I can only do if I manually set the fluent instance in the hook.

So can we depend on NewHook in the future or should I use New instead and fallback to an async connection when the first connection fails?

(Context: I use docker-compose where the order of startup is not guaranteed, so in theory fluentd might not start before my app, but I don't want my app to fail because of missing fluentd connection).

Any help or thoughts would be appreciated!

Thanks in advance!

Question: Fluentd input plugin?

Hi,

With which Fluentd input plugin does this hook work? in_forward or in_tcp?

Pieter

Support for RequestAck config

I created a PR for this at #30

That is the only way I was able to configure the logs to be sent when the connection is lost and re-established.

invalid JSON message

my code

package main

import (
	"context"
	"errors"

	"github.com/evalphobia/logrus_fluent"
	"github.com/sirupsen/logrus"
)

func main() {
	hook, err := logrus_fluent.NewWithConfig(logrus_fluent.Config{
		Host: "localhost",
		Port: 5170,
	})
	if err != nil {
		panic(err)
	}

	// set custom fire level
	hook.SetLevels([]logrus.Level{
		logrus.PanicLevel,
		logrus.ErrorLevel,
		logrus.InfoLevel,
	})

	// set static tag
	hook.SetTag("original.tag")
	// ignore field
	hook.AddIgnore("context")

	// filter func
	hook.AddFilter("error", logrus_fluent.FilterError)
	logrus.SetFormatter(&logrus.JSONFormatter{})
	logrus.AddHook(hook)
	for i := 1; i <= 1000; i++ {
		logrus.Info("Hello World")
	}
}

func logging(ctx context.Context) {
	logrus.WithFields(logrus.Fields{
		"value":   "some content...",
		"error":   errors.New("unknown error"), // this field will be applied filter function in the hook.
		"context": ctx,                         // this field will be ignored in the hook.
	}).Error("error message")
}

my log

{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}
{"level":"info","msg":"Hello World","time":"2018-09-26T11:32:44+08:00"}

fluent bit config and warn log

[jukay@mac-pro]~/Code/Go/src/github.com/hellojukay/fluent-bit-0.14.1/build/bin$ ./fluent-bit -i tcp -o stdout
[2018/09/26 14:31:22] [ warn] [in_tcp] invalid JSON message, skipping
[2018/09/26 14:31:22] [ warn] [in_tcp] invalid JSON message, skipping
[2018/09/26 14:31:22] [ warn] [in_tcp] invalid JSON message, skipping
[2018/09/26 14:31:22] [ warn] [in_tcp] invalid JSON message, skipping
[2018/09/26 14:31:22] [ warn] [in_tcp] invalid JSON message, skipping
[2018/09/26 14:31:22] [ warn] [in_tcp] invalid JSON message, skipping
[2018/09/26 14:31:22] [ warn] [in_tcp] invalid JSON message, skipping
[2018/09/26 14:31:22] [ warn] [in_tcp] invalid JSON message, skipping
[2018/09/26 14:31:22] [ warn] [in_tcp] invalid JSON message, skipping
[2018/09/26 14:31:22] [ warn] [in_tcp] invalid JSON message, skipping

my question

Why log json error,how to fix it

No Entry.Message in output

 logrus.WithFields(logrus.Fields{
    "tag": "walrus.app",
    "dscr": "description",
 }).Info("something happened")

Output:

 2015-08-20T14:56:51+03:00  walrus.app  {"dscr":"description","level":"info"}

Error logging not working

If an error is logged, the field is always empty.

I've create #32 with a test to demonstrate this, but I'm not sure about the fix as I am not too familiar with reflection.

Please let me know.