reubano / pygogo Goto Github PK

View Code? Open in Web Editor NEW

281.0 13.0 20.0 464 KB

A Python logging library with superpowers

License: MIT License

Makefile 1.25% Python 94.20% Shell 4.56%

logging logger structured-logging shell cli library featured

pygogo's Introduction

pygogo: a Python logger with superpowers

Index

Introduction

pygogo is a Python logging library and command-line interface with super powers. pygogo leverages the standard Python logging module under the hood, so there's no need to learn yet-another logging library. The default implementation sends all messages to stdout, and any messages at level WARNING or above also to stderr.

With pygogo, you can

Log via different handlers depending on the event severity
Format log messages as plain text, csv, json, and more..
Send logs to stdout, stderr, file, email, sockets, and more..
Inter-operate with the standard python logging module
and much more...

Requirements

pygogo has been tested and is known to work on Python 3.7, 3.8, and 3.9; and PyPy3.7.

Motivation

The standard logging module is great, but requires a ton of boilerplate before you can do anything really interesting with it. I designed pygogo to provide many useful logging use-cases out of the box. A reimplementation of Using LoggerAdapters to impart contextual information is shown below:

import pygogo as gogo

logger = gogo.Gogo(__name__).get_structured_logger(connid='1234')
logger.info('log message')

# Prints the following to stdout

{"message": "log message", "connid": "1234"}

Usage

pygogo is intended to be used either directly as a Python library or from the terminal via the command-line interface.

Library

Examples

Hello World

from pygogo import logger

logger.debug('hello world')
logger.error('hello error')

# Prints the following to `stdout`

hello world
hello error

# Prints the following to `stderr`

hello error

Log based debugging

import pygogo as gogo

def main(verbose=False):
    logger = gogo.Gogo(__name__, verbose=verbose).logger
    logger.debug('I will log to `stdout` only if `verbose` is True')
    logger.info('I will log to `stdout` always')
    logger.warning('I will log to both `stdout` and `stderr` always')

Disabled dual logging

import pygogo as gogo

logger = gogo.Gogo(monolog=True).logger
logger.debug('debug message')
logger.info('info message')
logger.warning('warning message')
logger.error('error message')
logger.critical('critical message')

# Prints the following to `stdout.log` (all messages at level `INFO` or below):

debug message
info message

# Prints the following to `stderr` (messages at level `WARNING` or above):

warning message
error message
critical message

Custom formatter¹

import logging
import pygogo as gogo

log_format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
formatter = logging.Formatter(log_format)

logger = gogo.Gogo(
    'examples.fmt',
    low_hdlr=gogo.handlers.file_hdlr('custom_fmt.log'),
    low_formatter=formatter,
    high_level='error',
    high_formatter=formatter).logger

# Now let's log something!

logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')

# Prints the following to `custom_fmt.log` (all messages):

2015-12-18 18:51:30,416 - examples.fmt.base - DEBUG - debug message
2015-12-18 18:51:30,416 - examples.fmt.base - INFO - info message
2015-12-18 18:51:30,416 - examples.fmt.base - WARNING - warn message
2015-12-18 18:51:30,416 - examples.fmt.base - ERROR - error message
2015-12-18 18:51:30,416 - examples.fmt.base - CRITICAL - critical message

# Prints the following to `stderr` (messages at level `ERROR` or above):

2015-12-18 18:51:30,416 - examples.fmt.base - ERROR - error message
2015-12-18 18:51:30,416 - examples.fmt.base - CRITICAL - critical message

Structured logging²

import pygogo as gogo

formatter = gogo.formatters.structured_formatter
kwargs = {'low_level': 'info', 'low_formatter': formatter}
logger = gogo.Gogo('examples.structured', **kwargs).logger
extra = {'set_value': set([1, 2, 3]), 'snowman': '☃'}
logger.info('log message', extra=extra)  # doctest: +ELLIPSIS

# Prints the following to `stdout`:

{"snowman": "\u2603", "name": "examples.structured.base", "level": "INFO", "message": "log message", "time": "2015-12-18 18:52:39", "msecs": 58.973073959350586, "set_value": [1, 2, 3]}

Using Filters to impart contextual information³

import logging
import pygogo as gogo

levels = ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL')
log_frmt = (
   '%(asctime)-4s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: '
   '%(user)-8s %(message)s')

formatter = logging.Formatter(log_frmt)
going = gogo.Gogo('a', low_formatter=formatter)
a1 = going.get_logger('b.c', ip='123.231.231.123', user='fred')
a2 = going.get_logger('e.f', ip='192.168.0.1', user='sheila')

# Now let's log something!

a1.debug('A debug message')
a1.info('An info %s', 'message')

for level in [getattr(logging, l) for l in levels]:
   name = logging.getLevelName(level)
   a2.log(level, 'A %s msg', name)

# Prints the following to `stdout` (all messages):

2015-12-19 10:12:24,479 a.b.c DEBUG    IP: 123.231.231.123 User: fred     A debug message
2015-12-19 10:12:24,479 a.b.c INFO     IP: 123.231.231.123 User: fred     An info message
2015-12-19 10:12:24,479 a.e.f DEBUG    IP: 192.168.0.1     User: sheila   A DEBUG msg
2015-12-19 10:12:24,479 a.e.f INFO     IP: 192.168.0.1     User: sheila   AN INFO msg
2015-12-19 10:12:24,479 a.e.f WARNING  IP: 192.168.0.1     User: sheila   A WARNING msg
2015-12-19 10:12:24,479 a.e.f ERROR    IP: 192.168.0.1     User: sheila   AN ERROR msg
2015-12-19 10:12:24,479 a.e.f CRITICAL IP: 192.168.0.1     User: sheila   A CRITICAL msg

# Prints the following to `stderr` (messages at level `WARNING` or above):

2015-12-19 10:12:24,479 a.e.f WARNING  IP: 192.168.0.1     User: sheila   A WARNING msg
2015-12-19 10:12:24,479 a.e.f ERROR    IP: 192.168.0.1     User: sheila   AN ERROR msg
2015-12-19 10:12:24,479 a.e.f CRITICAL IP: 192.168.0.1     User: sheila   A CRITICAL msg

Multiple loggers⁴

import pygogo as gogo

going = gogo.Gogo(
    'examples.lggrs',
    low_hdlr=gogo.handlers.file_hdlr('multi_lggrs.log'),
    low_formatter=gogo.formatters.fixed_formatter,
    high_level='info',
    high_formatter=gogo.formatters.console_formatter)

root = going.logger
logger1 = going.get_logger('area1')
logger2 = going.get_logger('area2')

# Now let's log something!

root.info('Jackdaws love my big sphinx.')
logger1.debug('Quick zephyrs blow, daft Jim.')
logger1.info('How daft jumping zebras vex.')
logger2.warning('Jail zesty vixen who grabbed pay.')
logger2.error('The five boxing wizards jump.')

# Prints the following to `multi_lggrs.log` (all messages):

2015-12-18 17:21:37.417 examples.lggrs.base INFO     Jackdaws love my big sphinx.
2015-12-18 17:21:37.417 examples.lggrs.area1 DEBUG    Quick zephyrs blow, daft Jim.
2015-12-18 17:21:37.417 examples.lggrs.area1 INFO     How daft jumping zebras vex.
2015-12-18 17:21:37.417 examples.lggrs.area2 WARNING  Jail zesty vixen who grabbed pay.
2015-12-18 17:21:37.417 examples.lggrs.area2 ERROR    The five boxing wizards jump.

# Prints the following to `stderr` (messages at level `INFO` or above):

examples.lggrs.base: INFO     Jackdaws love my big sphinx.
examples.lggrs.area1: INFO     How daft jumping zebras vex.
examples.lggrs.area2: WARNING  Jail zesty vixen who grabbed pay.
examples.lggrs.area2: ERROR    The five boxing wizards jump.

Notes

Command-line Interface

Examples

Basic Usage

gogo [options] <message>

show help

gogo -h

CLI usage

usage: gogo [options] <message>

description: Logs a given message

positional arguments:

message The message to log (defaults to reading from stdin).

optional arguments:

-h, --help show this help message and exit -l LEVEL, --msg-level LEVEL The level to log the message (default: info). Must be one of: critical, error, warning, info, debug.

-n NAME, --name NAME The logger name (default: pygogo) -D HANDLER, --high-hdlr HANDLER The high pass log handler (default: stderr). Must be one of: buffered, email, file, fileobj, socket, stderr, stdout, syslog, webhook.

-d HANDLER, --low-hdlr HANDLER

The low pass log handler (default: stdout). Must be one of: buffered, email, file, fileobj, socket, stderr, stdout, syslog, webhook.

-L LEVEL, --high-level LEVEL

Min level to log to the high pass handler (default: warning). Must be one of: buffered, email, file, fileobj, socket, stderr, stdout, syslog, webhook.

-e LEVEL, --low-level LEVEL

Min level to log to the low pass handler (default: debug). Must be one of: buffered, email, file, fileobj, socket, stderr, stdout, syslog, webhook.

-F FORMAT, --high-format FORMAT

High pass handler log format (default: basic). Must be one of: basic, bom, console, csv, fixed, json, structured.

-o FORMAT, --low-format FORMAT

Low pass handler log format (default: basic). Must be one of: basic, bom, console, csv, fixed, json, structured.

-m, --monolog Log high level events only to high pass handler. -f FILENAME, --filename FILENAME The filename to log to. Required for the follow handlers: file.

-s SUBJECT, --subject SUBJECT

The log subject (default: You've got mail). Used in the follow handlers: email.

-u URL, --url URL The log url. Required for the follow handlers: webhook. -H HOST, --host HOST The host. Used in the follow handlers: socket and syslog.

-p NUM, --port NUM The port number.

Used in the follow handlers: socket and syslog.

-t, --tcp Use TCP instead of UDP.

Used in the follow handlers: socket and syslog.

-g, --get Use a GET request instead of POST.

Used in the follow handlers: webhook.

-v, --version Show version and exit. -V, --verbose Increase output verbosity.

Hello World

gogo 'hello world'

Log based debugging

gogo 'default info level will log to `stdout`'
gogo --level=debug "debug won't log"
gogo --level=debug -V 'verbose will log to `stdout`'
gogo --level=info 'info will log to `stdout`'
gogo --level=warning 'warning will log to both `stdout` and `stderr`'

# Prints the following to `stdout`:

default info level will log to `stdout`
verbose will log to `stdout`
info will log to `stdout`
warning will log to both `stdout` and `stderr`

# Prints the following to `stderr`:

warning will log to both `stdout` and `stderr`

Disable dual logging

gogo --level=debug -V 'debug message'
gogo --level=info 'info message'
gogo --level=warning -m 'warning message'
gogo --level=error -m 'error message'
gogo --level=critical -m 'critical message'

# Prints the following to `stdout.log` (all messages at level `INFO` or below):

debug message
info message

# Prints the following to `stderr` (messages at level `WARNING` or above):

warning message
error message
critical message

Structured logging

gogo --low-format=json 'log message'

# Prints the following to `stdout`:

{"time": "2015-12-19 11:26:53.776", "name": "pygogo.runner", "level": "INFO", "message": "log message"}

Alternate handler

gogo --low-hdlr=file 'log message'

# Prints the following to `pygogo.log` in the current dir (assuming the current dir is named `pygogo`):

{"time": "2015-12-19 11:26:53.776", "name": "pygogo.runner", "level": "INFO", "message": "log message"}

Installation

(You are using a virtualenv, right?)

At the command line, install pygogo using either pip (recommended)

pip install pygogo

or easy_install

easy_install pygogo

Please see the installation doc for more details.

Project Structure

┌── bin
│   └── gogo
├── docs
│   ├── AUTHORS.rst
│   ├── CHANGES.rst
│   ├── INSTALLATION.rst
│   └── TODO.rst
├── helpers
│   ├── check-stage
│   ├── clean
│   ├── pippy
│   ├── srcdist
│   └── wheel
├── pygogo
│   ├── __init__.py
│   ├── formatters.py
│   ├── handlers.py
│   ├── main.py
│   └── utils.py
├── tests
│   ├── __init__.py
│   ├── standard.rc
│   ├── test.py
│   └── test_main.py
├── CONTRIBUTING.rst
├── LICENSE
├── MANIFEST.in
├── Makefile
├── README.rst
├── dev-requirements.txt
├── examples.py
├── manage.py
├── py2-requirements.txt
├── setup.cfg
├── setup.py
└── tox.ini

Design Principles

the built-in logging module isn't broken so don't reinvent the wheel
prefer functions over objects
keep the API as simple as possible

Structured Logging

There are severals ways to get structured (machine readable) log messages using pygogo. Each method makes a different customization/complexity trade-off which is outlined below:

Setup

The following methods make use of these variables.

import pygogo as gogo

kwargs = {'contextual': True}
extra = {'additional': True}

Methods

basic structured logger

The simplest to use. Useful if you don’t need message metadata, i.e., log level, log name, and log time.

logger = gogo.Gogo('basic').get_structured_logger('base', **kwargs)
logger.debug('message', extra=extra)

# Prints the following to `stdout`:

{"additional": true, "contextual": true, "message": "message"}

structured formatter

Requires an additional step of specifying a formatter. Useful if you need message metadata, i.e., log level, log name, and log time.

formatter = gogo.formatters.structured_formatter
logger = gogo.Gogo('struct', low_formatter=formatter).get_logger(**kwargs)
logger.debug('message', extra=extra)

# Prints the following to `stdout`:

 {"additional": true, "contextual": true, "level": "DEBUG", "message": "message", "msecs": 760.5140209197998, "name": "struct.base", "time": "2015-12-19 14:25:58"}

JSON formatter

Requires an additional step of specifying a formatter. Useful if you require millisecond precision in the date. If you are ok with having the milliseconds in a separate field, consider the structured formatter since it supports the extra keyword and contextual information.

formatter = gogo.formatters.json_formatter
logger = gogo.Gogo('json', low_formatter=formatter).get_logger(**kwargs)
logger.debug('message', extra=extra)

# Prints the following to `stdout`:

{"level": "DEBUG", "message": "message", "name": "json.base", "time": "2015-12-19 14:25:58.760"}

# Note that both `extra` and `kwargs` were ignored

custom logger

The most complex and customizable. Useful if you need a custom log or date format not provided by the above methods. However, even though this method supports the extra keyword when logging, it is static (unlike the structured logger or structured formatter). This is because the log format must be specified at the time of the log's creation and therefore can't adapt to log messages with differing extra parameters.

logfmt = (
    '{"time": "%(asctime)s.%(msecs)d", "name": "%(name)s", "level":'
    ' "%(levelname)s", "message": "%(message)s", '
    '"contextual": "%(contextual)s", "additional": "%(additional)s"}')

fmtr = logging.Formatter(logfmt, datefmt=gogo.formatters.DATEFMT)
logger = gogo.Gogo('custom', low_formatter=fmtr).get_logger(**kwargs)
logger.debug('message', extra=extra)

# Prints the following to `stdout`:

{"additional": "True", "contextual": "True", "level": "DEBUG", "message": "message", "name": "custom.logger", "time": "2015-12-19 14:25:58.760"}

Summary

The following table can help make sense of the different methods:

	structured logger	structured formatter	json formatter	custom logger
contextual information	✔	✔		✔
`extra` param support	✔	✔		✔
dynamic `extra` support	✔	✔
message metadata		✔	✔	✔
available via the command line		✔	✔
`msecs` field		✔
milliseconds in time field			✔	✔
custom date format				✔
custom log format				✔

Formatters

pygogo has several builtin formatters and also supports any logging.Formatter instance.

Examples

builtin CSV format in python

import pygogo as gogo

formatter = gogo.formatters.csv_formatter
gogo.Gogo('csv', low_formatter=formatter).logger.debug('message')

# Prints the following to `stdout`:

2015-12-19 17:03:48.99,csv.base,DEBUG,"message"

`logging.Formatter` instance in python

import logging
import pygogo as gogo

datefmt = gogo.formatters.DATEFMT
formatter = logging.Formatter(gogo.formatters.CSV_FORMAT, datefmt=datefmt)
gogo.Gogo('csv', low_formatter=formatter).get_logger('custom').debug('message')

# Prints the following to `stdout`:

2015-12-19 17:03:48.99,csv.custom,DEBUG,"message"

builtin CSV format via CLI

gogo --low-format=csv 'message'

# Prints the following to `stdout`:

2015-12-19 15:51:32.16,pygogo.runner,INFO,"message"

Summary

The following table can help make sense of the different builtin formatters:

name	message
basic	message
bom	message
console	name: INFO message
csv	2015-12-19 15:51:32.16,name,INFO,"message"
fixed	2015-12-19 15:51:32.16 name INFO message
json	{"level": "INFO", "message": "message", "name": "name", "time": "2015-12-19 15:51:32.16"}
structured	{"level": "INFO", "message": "message", "msecs": 16.5140209197998, "name": "name", "time": "2015-12-19 15:51:32"}

Handlers

pygogo has several builtin handlers and also supports any instance from the logging.handlers module.

Examples

builtin stdout handler in python

import pygogo as gogo

hdlr = gogo.handlers.stdout_hdlr()
gogo.Gogo('stdout', low_hdlr=hdlr).logger.debug('message')

# Prints 'message' to `stdout`

`logging.StreamHandler` instance in python

import logging
import sys
import pygogo as gogo

hdlr = logging.StreamHandler(sys.stdout)
gogo.Gogo('stdout', low_hdlr=hdlr).get_logger('custom').debug('message')

# Prints 'message' to `stdout`

builtin CSV format via CLI

gogo --low-hdlr=stdout 'message'

# Prints 'message' to `stdout`

Summary

The following table can help make sense of the different builtin handlers:

name	description
buffered	Holds log in memory until it reaches its capacity, or it logs a message with a level at or above the flush level
email	Emails log to a given email address
file	Writes log to a given filename
fileobj	Writes log to a given file-like object
socket	Writes log to a given network socket
stderr	Writes log to standard error
stdout	Writes log to standard output
syslog	Writes log to syslog
webhook	POSTs log to a url

Scripts

pygogo comes with a built in task manager manage.py

Setup

pip install -r dev-requirements.txt

Examples

Run python linter and nose tests

manage lint
manage test

Contributing

Please mimic the coding style/conventions used in this repo. If you add new classes or functions, please add the appropriate doc blocks with examples. Also, make sure the python linter and nose tests pass.

Please see the contributing doc for more details.

License

pygogo is distributed under the MIT License.

pygogo's People

Contributors

Stargazers

Watchers

Forkers

mypan liutaihua tricosmo foobacca bcaashok luiscape mrmuli rkx-forks vault-the gyp niranjansingh nicewook lewisacidic bpx-energy huangchenqipapegames tjni

pygogo's Issues

Initial Update

Hi 👊

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create seperate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! 🤖

tests/test.py fails with invalid syntax

Testing 1.3.0, running tests fails like this:

$ PYTHONPATH=/sw/build.build/pygogo-py38-1.3.0-1/pygogo-1.3.0/build/lib /sw/bin/python3.8 tests/test.py
Script result: /sw/build.build/pygogo-py38-1.3.0-1/pygogo-1.3.0/bin/gogo --help
  return code: 1
-- stderr: --------------------
Traceback (most recent call last):
  File "/sw/build.build/pygogo-py38-1.3.0-1/pygogo-1.3.0/bin/gogo", line 13, in <module>
    from pygogo import main
  File "/sw/build.build/pygogo-py38-1.3.0-1/pygogo-1.3.0/build/lib/pygogo/__init__.py", line 49, in <module>
    from . import formatters, handlers, utils
  File "/sw/build.build/pygogo-py38-1.3.0-1/pygogo-1.3.0/build/lib/pygogo/formatters.py", line 405
    logging.DEBUG: f"{debug_color} {self._fmt} {RESET}",
                                                      ^
SyntaxError: invalid syntax

Traceback (most recent call last):
  File "tests/test.py", line 98, in <module>
    main(script, tests)
  File "tests/test.py", line 44, in main
    result = env.run(command, cwd=p.abspath(p.dirname(p.dirname(__file__))))
  File "/sw/lib/python3.8/site-packages/scripttest.py", line 273, in run
    result.assert_no_error(quiet)
  File "/sw/lib/python3.8/site-packages/scripttest.py", line 426, in assert_no_error
    raise AssertionError(
AssertionError: Script returned code: 1

/sw/bin/nosetests3.8 -xv finished fine with no errors in 57 tests. This failure happens with py3.8, 3.9, and 3.10. This is macOS 10.14.5, but I don't think this would matter given the error type.

Request: Add log rotation for file_hdlr

A welcome feature would be built-in log rotation for the file handler, enabled via kwarg for max file size, and maybe another kwarg for the number of old/rotated logs to keep around.

Would you consider that in scope for this project?

README.md - misspell

I think below stdout.log should changed to stdout
Thanks for the package

Disabled dual logging

import pygogo as gogo

logger = gogo.Gogo(monolog=True).logger
logger.debug('debug message')
logger.info('info message')
logger.warning('warning message')
logger.error('error message')
logger.critical('critical message')

Prints the following to `stdout.log` (all messages at level `INFO` or below):

csv formatter output invalid

>>> gogo.Gogo(name="root", low_formatter=gogo.formatters.csv_formatter).info('..."hmm"')
# stdout:
# 2019-02-23 10:00:02.825,root.base,INFO,"..."hmm""

It should use a real csv formatter

Logger function to file logging multiple times

Hi, i'm using Pygogo to log my project but in the wild the pygogo have some weird behavior, like this:
{"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"} {"time": "2018-01-22 10:59:47.279", "name": "json.base", "level": "INFO", "message": "Connected to:PRDSLMAI"}

It all has the same time. So I think this is the wrong behavior, and this is my function that calls pygogo:
def get_logger(): file = utils.open_file_from_project_root('resources\\log_file.json') log_json = json.loads(file) log_path = os.path.join(log_json['path'], log_json['filename']) return gogo.Gogo('json', low_hdlr=gogo.handlers.file_hdlr(log_path), low_formatter=gogo.formatters.json_formatter, high_level='error', high_formatter=gogo.formatters.json_formatter)\ .get_logger()

What should I do?

Add support for logfmt

logfmt (second article) is a format meant to be between JSON and normal logs for (human) readability and (machine) parse-ability.

Any chance it could be added to pygogo? I might have a go myself, but thought I'd ask at least.

ImportError: No module named builtins

Traceback (most recent call last):
  File "testscript.py", line 9, in <module>
    import pygogo as gogo
  File ".../virtualenv/lib/python2.7/site-packages/pygogo/__init__.py", line 52, in <module>
    from builtins import *
ImportError: No module named builtins

Say what?

Fix broken test

9dccca6

Add async support

factor out the builtin logging calls to create a pluggable backend (via #1)
add twisted backend
add tornado backend
add asyncio backend

Loggly support?

Does this library have loggly (https://www.loggly.com/) support ?

Is `pygogo` support log-rotation?

Is pygogo support log-rotation?
One single file is huge for my case.

Agree upon a common JSON format with other structured logging frameworks

Hi! I've recently become more interested in structured logging, and have looked into a few structured logging libraries.

You get amazing power when you dump the logs from all of your different systems and sources into a centralized log store, and can then view and analyze them as one whole.

What I've noticed though is that the various structured logging frameworks all save JSON log entries in similar, but slightly different schemas.

For example, for storing timestamps, this library uses the JSON key "time", while other libraries use "timestamp" or "at". Another area where libraries differ is in how they store log levels.

These small differences cause friction when analyzing the central logstore, which contains structured logs that have been collected from multiple systems/microservices.

For example, one service might tag warnings with the string "WARN" while another with the string "warning". So if I want to view only warnings, I need to take this difference into account and write a tricky "OR" filter expression. This may seem minor, but these small inconsistencies cause great pain.

I believe that these small differences between the various structured logging libraries exist not because of any strongly held opinions, but simply because mainstream structured and centralized logging is still relatively young, and so there is no standard or common consensus.

I think it would be very beneficial to everyone if we could all unite around a common format.

To get things started, I have created a GitHub repository to centralize discussions here: https://github.com/bitc/structured-logging-schema

It contains a comparison between several structured logging libraries, summarizing the differences between them all. This can hopefully be a start to help us arrive at common ground.

I encourage the authors of this library (and anyone else who has an opinion) to participate in the discussion!

Discussion takes place in the issues for this repo: https://github.com/bitc/structured-logging-schema

Add optimzation benchmarks

Include some benchmarks to show the improvement gained by different implementations of #1

bump version and push to PyPI

The current version in PyPI (0.10.0) lacks some important patches such as 0feab9e, 910fa6e or the not-yet-merged 530d82d which makes a simple 'pip install' of this package or anything else (such as https://github.com/reubano/csv2ofx) impossible. A manual workaround is possible but it's a hassle.

Add optimization option

the built-in logging module isn't broken so don't reinvent the wheel

Hmm.. Do you have any performance benchmarks that prove that logging is really isn't broken in terms of performance?

It logging wasn't broken then I suppose there won't be such things as https://twistedmatrix.com/documents/15.2.1/core/howto/logger.html

Which also says that:

logging is a blocking API, and logging can be configured to block for long periods (eg. it may write to the network). No protection is provided to prevent blocking

As for don't reinvent the wheel here is a good example of "logging with superpowers". https://eliot.readthedocs.org/en/0.11.0/index.html =)

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

reubano / pygogo Goto Github PK

pygogo's Introduction

pygogo: a Python logger with superpowers

Index

Introduction

Requirements

Motivation

Usage

Library

Examples

Notes

Command-line Interface

Examples

Installation

Project Structure

Design Principles

Structured Logging

Setup

Methods

basic structured logger

structured formatter

JSON formatter

custom logger

Summary

Formatters

Examples

builtin CSV format in python

logging.Formatter instance in python

builtin CSV format via CLI

Summary

Handlers

Examples

builtin stdout handler in python

logging.StreamHandler instance in python

builtin CSV format via CLI

Summary

Scripts

Setup

Examples

Contributing

License

pygogo's People

Contributors

Stargazers

Watchers

Forkers

pygogo's Issues

I think below stdout.log should changed to stdout Thanks for the package

Prints the following to stdout.log (all messages at level INFO or below):

Recommend Projects

Recommend Topics

Recommend Org

`logging.Formatter` instance in python

`logging.StreamHandler` instance in python

I think below stdout.log should changed to stdout
Thanks for the package

Prints the following to `stdout.log` (all messages at level `INFO` or below):