Code Monkey home page Code Monkey logo

bes's Introduction

Getting Started With bes

bes is a flexible bulk uploader for Elastic Search. At the moment, it only supports index uploads over UDP. You can retrieve the logged data from Elastic Search and analyze it dynamically using Kibana (source). This package just makes the initial upload to Elastic Search as painless as possible.

The connection parameters and other configuration options are stored in bes.DEFAULT. Override them as you see fit (e.g. to connect to a remote Elastic Search server):

>>> import bes
>>> bes.DEFAULT['host'] = 'my-es-server.example.com'

Then log away:

>>> bes.log(type='record', user='jdoe', action='swims')

If you are so inclined, you can override the DEFAULT index on a per-call basis:

>>> bes.log(index='my-index', type='my-type', user='jdoe', action='bikes')

The log generated by the above will look like:

{
  "@timestamp": "2013-09-26T16:34:09.179048",
  "@version": 1,
  "action": "bikes",
  "user": "jdoe"
}

following Jordan Sissel's new format for Logstash:

You should specify a unique type for each record you create, because Elastic Search doesn't recalculate its mapping if you post a new event that uses an old key with a new data type. If you are only logging a single record type, you can configure a global default:

>>> bes.DEFAULT['type'] = 'record'
>>> bes.log(user='jdoe', action='runs')

Django

Although the core of the library is framework-agnostic, we'll be using this to log events in a Django website. There are a few helpers for extracting information from HttpRequest to make this as easy as possible. Drop it into your views with somthing like:

import django.http
imort django.shortcuts
import bes.django
from polls.models import Poll

def detail(request, poll_id):
    try:
        p = Poll.objects.get(pk=poll_id)
    except Poll.DoesNotExist:
        bes.django.log_user_request_path(
            request=request, type='404', poll_id=poll_id)
        raise django.http.Http404
    bes.django.log_user_request_path(
        request=request, type='poll-detail',
        poll_id=poll_id, poll_name=p.name)
    return django.shortcutsrender_to_response(
        'polls/detail.html', {'poll': p})

You can also override bes' defaults in your Django config:

BULK_ELASTIC_SEARCH_LOGGING_HOST = 'my-es-server.example.com'

The Django config names match the uppercased DEFAULT keys with a BULK_ELASTIC_SEARCH_LOGGING_ prefix (for example, host โ†’ BULK_ELASTIC_SEARCH_LOGGING_HOST).

Tracing

To log activity for critical functions, bes has tracing decorators:

import bes.trace

@bes.trace.trace()
def my_function():
    return 1 + 2 / 3

By default this logs with bes.log using the decorated function's name (e.g. my_function) as the log type. You can override this if you want something different:

def my_logger(*args, **kwargs):
    print((args, kwargs))

@bes.trace.trace(type='my-type', logger=my_logger)
def my_function():
    return 1 + 2 / 3

Additional trace arguments are passed straight through to the logger:

@bes.trace.trace(index='my-index', type='my-type', sport='triathalon')
def my_function():
    return 1 + 2 / 3

By default, you'll get separate logged messages before the decorated function starts, after it completes, or if it raises an exception. An action argument is passed to the logger in each case, with values of start, complete, and error respectively. In the error case, the logger is also passed the string-ified exception (error=str(e)) to help with debugging the problem. If you don't want one or more of these logged messages, you can turn them off:

@bes.trace.trace(start=False, error=False, complete=False):
def my_function():
    return 1 + 2 / 3

although if you turn them all off, you're not going to get any messages at all ;).

The trace decorator uses wrapt (if it's installed) to create introspection-preserving decorators. If wrapt is not installed, we fallback to the builtin functools.wraps which handles the basics like docstrings, but doesn't do as well for more involved introspection (e.g. inspect.getsource).

Testing

Testing uses unittest's automatic test discovery. Run the test suite with:

$ python -m unittest discover

For Python <3.3, the test suite requires the external mock package, which is bundled as unittest.mock in Python 3.3.

bes's People

Contributors

shofetim avatar wking avatar

Watchers

 avatar  avatar  avatar  avatar

bes's Issues

@timestamp and index time stamp differ

I don't appear to have permissions to the repository to create a branch and pull request

ERROR: Permission to wking/bes.git denied to shofetim.
fatal: The remote end hung up unexpectedly

and

ERROR: Permission to wking/bes.git denied to azurestandard.
fatal: The remote end hung up unexpectedly

The time stamp used in @timestamp is in UTC, while the time stamped index name is in local time.

bes/bes/init.py:76 should be
kwargs['@timestamp'] = _datetime.datetime.now().isoformat()

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.