brocode / fblog Goto Github PK

View Code? Open in Web Editor NEW

396.0 5.0 26.0 1.28 MB

Small command-line JSON Log viewer

License: Do What The F*ck You Want To Public License

Rust 99.81% Dockerfile 0.19%

json log viewer command-line command-line-tool

fblog's Introduction

fblog

A small tool to view json log files.

Print specific fields

fblog -a message -a "status > a" sample_nested.json.log

Prefix Logs

If your query docker or kubectl for multiple pods it will prefix the log lines: PODNAME | {"message": "test"}. fblog can parse this and add it to the message. Just use -p.

Filter

To filter log messages it is possible to use lua. If you are unsure which variables are available you can use --print-lua to see the code generated by fblog.

fblog -f 'level ~= "info"' # will print all message where the level is not info
fblog -f 'process == "play"' # will print all message where the process is play
fblog -f 'string.find(fu, "bow.*") ~= nil' # will print all messages where fu starts with bow
fblog -f 'process == "play"' # will print all message where the process is play
fblog -f 'process == "rust" and fu == "bower"'
fblog --no-implicit-filter-return-statement -f 'if 3 > 2 then return true else return false end'

# not valid lua identifiers like log.level gets converted to log_level.
# Every character that is not _ or a letter will be converted to _
fblog -d -f 'log_level == "WARN"' sample_elastic.log

# nested fields are converted to lua records
fblog  -d -f 'status.a == 100' sample_nested.json.log

# array fields are converted to lua tables (index starts with 1)
fblog  -d -f 'status.d[2] == "a"' sample_nested.json.log

Customize

fblog tries to detect the message, severity and timestamp of a log entry. This behavior can be customized. See --help for more information.

You can customize fblog messages: Format output:

fblog -p --main-line-format "{{#if short_message}}{{ red short_message }}{{/if}}" sample.json.log

The following sanitized variables are provided by fblog:

fblog_timestamp
fblog_level
fblog_message
fblog_prefix

For the default formatting see --help

Nested values are registered as objects. So you can use nested.value to access nested values.

handlebar helpers:

bold
yellow
red
blue
purple
cyan
green
color_rgb 0 0 0
uppercase
level_style
fixed_size 10

NO_COLOR

fblog disables color output if the NO_COLOR environment variable is present.

no-color

Message placeholder substitution

Placeholders in the message (fblog_message) can be substituted with their corresponding values in a context object or array. To enable substitutions, pass the -s flag or either set context key (-c context) or placeholder format (-F {key}).

Note that the placeholder format should be written like <PREFIX>key<SUFFIX>, where it would match a placeholder with the key key.

Example

Given the following log (referred to as example.log):

{"message": "Found #{count} new items.", "extra_data": {"count": 556}, "level": "info"}

Running with the following arguments:

fblog -c extra_data -F '#{key}' example.log

Result:

Installation

cargo install fblog

Available in package managers: AUR, brew

Log tailing

fblog does not support native log tailing but this is easily achiveable.

tail -f file | fblog

Or with kubernetes tooling for example

kubectl logs -f ... | fblog

In general you can pipe any endless stream to fblog.

shell completions

fblog --generate-completions <shell>

configuration file

fblog reads its configuration from a file called fblog.toml, which is located

Linux: $XDG_CONFIG_HOME or $HOME/.config

Windows: {FOLDERID_RoamingAppData}

macOS: $HOME/Library/Application Support

If the file does not exist or is empty this is the default config.

log levels

These levels are colorized by fblog:

trace
debug
info
warn
error
fatal

You can map additional level values (used for output and color):

[level_map]
10 = "trace"
20 = "debug"
30 = "info"
40 = "warn"
50 = "error"
60 = "fatal"
# these values for example are used by https://www.npmjs.com/package/bunyan#levels

k9s

In the file ~/.config/k9s/plugin.yml add this:

plugin:
  fblog-pod:
    shortCut: Shift-L
    confirm: false
    description: "fblog"
    scopes:
      - pods
    command: sh
    background: false
    args:
      - -c
      - "kubectl logs --follow -n $NAMESPACE $NAME | fblog"
  fblog-container:
    shortCut: Shift-L
    confirm: false
    description: "fblog"
    scopes:
      - containers
    command: sh
    background: false
    args:
      - -c
      - "kubectl logs  --follow -n $NAMESPACE $POD -c $NAME | fblog"
  fblog-pod-all:
    shortCut: Shift-A
    confirm: false
    description: "fblog -d"
    scopes:
      - pods
    command: sh
    background: false
    args:
      - -c
      - "kubectl logs --follow -n $NAMESPACE $NAME | fblog -d"
  fblog-container-all:
    shortCut: Shift-A
    confirm: false
    description: "fblog -d"
    scopes:
      - containers
    command: sh
    background: false
    args:
      - -c
      - "kubectl logs  --follow -n $NAMESPACE $POD -c $NAME | fblog -d"

fblog's People

Contributors

Stargazers

Watchers

fblog's Issues

Failed to install `fblog` on Mac

Command output:

> cargo install fblog

    Updating crates.io index
  Installing fblog v4.4.0
error: failed to compile `fblog v4.4.0`, intermediate artifacts can be found at `/var/folders/vl/dvf2lmqj6_12yrwmy9sqt4h00000gn/T/cargo-install8KhI94`

Caused by:
  failed to select a version for the requirement `clap = "^4"`
  candidate versions found which didn't match: 3.2.25, 3.2.24, 3.2.23, ...
  location searched: crates.io index
  required by package `fblog v4.4.0`

> cargo search clap
clap = "4.3.2"                             # A simple to use, efficient, and full-featured Command Line Argument Parser
clap-v3 = "3.0.0-beta.1"                   # A simple to use, efficient, and full-featured Command Line Argument Parser
nameless-clap = "3.0.0-beta.2.2"           # A simple to use, efficient, and full-featured Command Line Argument Parser
clap-serde = "0.5.1"                       # Provides a wrapper to deserialize clap app using serde.
clap-serde-derive = "0.2.0"                # Merge results from clap and serde into struct with derive
clap-digest = "0.3.0"                      # clap and digest integration
clap-nested = "0.4.0"                      # Convenient `clap` for CLI apps with multi-level subcommands.
clap_derive-v3 = "3.0.0-beta.1"            # Parse command line argument by defining a struct, derive crate.
nameless-clap_derive = "3.0.0-beta.2.2"    # Parse command line argument by defining a struct, derive crate.
clap-markdown = "0.1.3"                    # Autogenerate Markdown documentation for clap command-line tools
... and 522 crates more (use --limit N to see more)

Environment:
macOs 13.3.1 with m1

Help needed. Thanks

Replace ansi_term

messages are HTML escaped.

Whe I use fblog - https://github.com/brocode/fblog/releases/download/v2.1.0/fblog.deb deb packaged version, It works greatly.

But if I install with cargo install fblog, fblog print HTML escaped message.

What's the difference?

how filter fields separated by dashes

fblog -f 'method=="POST" and header.X-Customer-Id[1]=="123456"' -a body -a res_body req.log

{
  "header":{
      "X-Tenant-Id": ["123456"]
  }
}

"fblog -a" not allow multipy params

Version: fblog 1.3.1

Good application, but this feature doesn't work on multiple values

Maybe i do something wrong or this is a bug?

Add support for epoch timestamps

Hi,

It seems fblog cannot detect and convert epoch timestamps (in seconds / microseconds etc.) to a human-readable format automatically, which could be a very helpful feature to have.

Example:

{"time":"1705105985", ...}

Support nested fields

Currently only toplevel keys are processed.

no-color compliant

Would you consider implementing no-color?

I could create a PR if you did.

I ran into this while trying to pipe fblog output through less and getting something not as friendly

ESC[1m2020-03-02T22:45:22ESC[0m ESC[1;31mERROR:ESC[0m Failed to send sms
ESC[1;38;2;150;150;150m                     body:ESC[0m ID: 19169 Status: 1
ESC[1;38;2;150;150;150m                     file:ESC[0m my/path/file.go:194
ESC[1;38;2;150;150;150m                     func:ESC[0m other/path.monitorMsgID
ESC[1;38;2;150;150;150m                    level:ESC[0m error
ESC[1;38;2;150;150;150m                      msg:ESC[0m Failed to send sms
ESC[1;38;2;150;150;150m                  request:ESC[0m http://localhost/query
ESC[1;38;2;150;150;150m                     time:ESC[0m 2020-03-02T22:45:22Z
ESC[1m2020-03-02T22:45:22ESC[0m ESC[1;31mERROR:ESC[0m Failed to send sms
ESC[1;38;2;150;150;150m                    error:ESC[0m [{other/path/file.go:103: } {other/path/file.go:190: Failed to send sms}]
ESC[1;38;2;150;150;150m                     file:ESC[0m my/path/file.go:106
ESC[1;38;2;150;150;150m                     func:ESC[0m other/path.(*smsClient).Send
ESC[1;38;2;150;150;150m                    level:ESC[0m error
ESC[1;38;2;150;150;150m                      msg:ESC[0m Failed to send sms```

RFE: Customizable core fields & layout

I like the idea that for each log event fblog displays a main line and optional additional lines per field. However, succinctness is also a virtue. Three lines to display message + logger name + thread name is a significant waste of vertical space. Thus, I had like to be able to cram more info into the main line.

jaslog, a similar project, includes logger and thread name, https://github.com/jbruggem/jaslog/blob/master/src/line_formats.rs#L76. It seems a reasonable decision, but I imagine that essential fields are context dependent and every user has its personal opinion about what to display in the main line.

Would it make sense to:

Add more fields to the main line
Provide a configuration mechanism to decide what, and perhaps how, to display in the main line

Support fields with other types than string.

Sometime fblog does not parse number values(but sometime do).

$ cat > x.json
{"timestamp":"2020-09-17 07:11:54.936","@version":"1","message":"Done Good!. exitCode: 0","thread":"main","level":"INFO","caller_file_name":"MyApplication.java","caller_line_number":12345}

$ cat x.json | .cargo/bin/fblog --main-line-format "{{fblog_timestamp}} {{level_style fblog_level}} [{{thread}}] ({{caller_file_name}}#{{caller_line_number}}) {{fblog_message}}"

# there is no number 12345
2020-09-17 07:11:54.936 INFO [main] (MyApplication.java#) Done Good!. exitCode: 0

# the result I expected
2020-09-17 07:11:54.936 INFO [main] (MyApplication.java#12345) Done Good!. exitCode: 0

Keys with a dot cannot be filtered

I'm evaluating fblog to display log files implementing elastic common schema. Almost all ECS fields contains one or several dots. It is unclear how such keys are mapped to a valid LUA identifier.

$ cat /tmp/test2
{"@timestamp":"2020-02-24T08:34:47.886Z", "log.level": "INFO", "message":"Hello, world!", "process.thread.name":"main","log.logger":"com.greencomnetworks.log4j2.Removeme","error.type":"java.lang.RuntimeException","error.stack_trace":"java.lang.RuntimeException\n\tat com.greencomnetworks.poc.Removeme.main(Removeme.java:10)\n"}

$ fblog /tmp/test2  -f 'log.level == "INFO"'
Failed to apply filter expression: 'ExecutionError("[string \"chunk\"]:1: attempt to index global \'log\' (a nil value)")'

support log tailing

Please

support tailing logs (reading forever)
support starting from the end (for tailing)

Support for defaults from a config file

First of all, thanks for this tool! I had an idea to make this for myself and was pleasantly surprised to find this already existed. 😄

I was wondering if it would be possible to add support to pull default configuration from a config file somewhere like ~/.config/fblog/fblog.toml, or somewhere similar, and maybe the ability to provide a config path on the command line? Our logs have a few fields that don't make sense to add as auto-detected fields but are still helpful to show when they come up. Including those by default from a config file would allow me to set up some defaults without needing to remember (or type) them every time.

Thanks again!

bump formula to newest version

@ChristianLohmann

Provide precompiled binary

thread 'main' panicked at 'Expect to be able to write to out stream.'

Sorry to bother you again. This happens when you pipe to less and then quit, like so (bash):

> fblog -d huge-log.json | less -r
# now press `:q` to quit less
thread 'main' panicked at 'Expect to be able to write to out stream.: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', src/log.rs:100:7
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I don't think that it's a problem, but it doesn't look nice.

Missing pod name prefix

Hi!

Thanks for creating this! I've been using it daily for a few weeks, and it's great 🥇

Just noticed the prefix-logs section in the readme mention that fblog should prefix logs with pod names when multiple pods are queries for logs. Unfortunately, it doesn't seem to work for me:

When I run

kubectl logs -l app=foo -n bar --follow | ./target/release/fblog -p

with and without -p, it prints regular logs with no pod identifier prefix. Am I missing something, or is the readme not quite right?

Print all fields of a nested json object

Hi, thank you for the useful tool! I have an issue with using the flag "--additional-value".

I'm using fblog v3.0.2 built by myself using Cargo and source code.

The output of execute $ fblog --help:

fblog 3.0.2
Brocode inc <[email protected]>
json log viewer

USAGE:
    fblog [FLAGS] [OPTIONS] [--] [INPUT]

...

The problem is that the output does not contain the field I specified using the flag -a.

I have the log message:

{"time": "2021-11-08T13:55:45.924593+00:00", "level": "INFO", "caller": "/app/service/runner.py:58", "message": "Some message here", "process": "1", "thread": "MainThread", "function": "run", "line_no": "58", "module": "runner", "logger": "service.runner", "extra": {"a": 1}}

I want to see a value of the key "extra", so i'm using fblog -a extra and it doesn't work. At the same time, if I use the command fblog -a module, the value of the key module is displayed correctly.

Please help me figure out what I'm doing wrong, or confirm that this is a bug.

replace hlua

unmaintained library

pretty printed json seems to be an issue

I have a json file that I have to pre-process via jq after that run I end up in pretty printed (multiline) json.
Feeding this into fblog causes some strange errors, example below:

the json fed into fblog looks like (excerpt):

{
  "@message": "[object Object]\nResponse Headers have already been sent",
  "@timestamp": "2021-05-05T11:00:42.627Z",
  "@fields": {
    "level": "error"
  }
}
{
  "@message": "[object Object]\nResponse Headers have already been sent",
  "@timestamp": "2021-05-05T11:01:13.650Z",
  "@fields": {
    "level": "error"
  }
}

This very json ends up like this:

$ cat *json* | grep -v 'info' | grep -v 'debug' | jq '.log | fromjson ' | fblog

??? > {
??? >   "@message": "[object Object]\nResponse Headers have already been sent",
??? >   "@timestamp": "2021-05-05T11:00:42.627Z",
??? >   "@fields": {
??? >     "level": "error"
??? >   }
??? > }
??? > {
??? >   "@message": "[object Object]\nResponse Headers have already been sent",
??? >   "@timestamp": "2021-05-05T11:01:13.650Z",
??? >   "@fields": {
??? >     "level": "error"
??? >   }
??? > }

besides the json keys not fitting the default of fblog (that can be fixed via -l | -m | -t) the whole multiline json topic seems to be the root cause.

Does fblog expect single line json?