Code Monkey home page Code Monkey logo

python-dtrace's Introduction

Python as a DTrace consumer

This is a first shot into looking at getting something up and running like https://github.com/bcantrill/node-libdtrace for Python.

Note: This is far from ready - documentation on writing own DTrace consumers is very rare :-/

The codes in the examples folder should give an overview of how to use Python as a DTrace consumer.

Currently, this package provides two modules: one wraps libdtrace using ctypes; The other one uses cython. Should you not have cython installed, the ctypes wrapper can still be used.

The Python DTrace consumer can be installed via source here on GitHub, or using pypi.

Cython based wrapper

The Cython wrapper is more sophisticated and generally easier to use. Initializing the Python based DTrace consumer is as simple as:

import dtrace

SCRIPT = 'dtrace:::BEGIN {trace("Hello World");}'

consumer = dtrace.DTraceConsumer(walk_func=my_walk [...])
consumer.run(SCRIPT, runtime=3)

The simple DTraceConsumer can be initialized with self written callbacks which will allow you to get the output from DTrace, the chewing of the probes as well as walking the aggregate at the end of the DTrace run. The signatures looks like this:

def simple_chew(cpu):
    pass

def simple_chewrec(action):
    pass

def simple_out(value):
    pass

def simple_walk(action, identifier, keys, value):
    pass

Those functions can also be part of a class. Simply add the self parameter as the first one:

def simple_chew(self, cpu):
    pass

The DTraceConsumer has a run_script function which will run a provided DTrace script for some time or till it is finished. The time on how long it is run can be provided as parameter just like the script. During the DTrace chew, chewrec and the out callbacks are called. When the run is finished the aggregation will be walked - Thus you can aggregate for 3 seconds and then see the results.

There also exists a DTraceConsumerThread which can be used to continuously run DTrace in the background. This might come in handy when you want to continuously aggregate data for e.g. an GUI. The chew, chewrec, out and walk callbacks are now called as the snapshot function of DTrace is called.

The DTraceConsumerThread has a parameter sleep which defaults to 0. This means that the Thread will wait for DTrace for new aggregation data to arrive. This has a major drawback since during the wait the Python GIL is acquired and your program will stop if it needs to wait for DTrace to get new data. Setting the parameter to 1 (or higher) will let the Thread snapshot the DTrace aggregate every second instead of waiting for new data. Both usages might make sense - set the sleep parameter if you know data will arrive sporadically or simply let it default to 0 if you know data comes in all the time - so nothing will be blocked.

thr = DTraceConsumerThread(SCRIPT)
thr.start()

# we will stop the thread after 5 sec...
time.sleep(5)

# stop and wait for join...
thr.stop()
thr.join()

Examples for both ways of handling the Python DTrace consumer can be found in the examples folder.

Ctypes based wrapper

Both the DTraceConsumer and the DTraceConsumerThread can be initialized with self written chew, chewrec, buffered out and walk functions just like in the cython approach. See the examples on how to do this. Also note that the out, walk, chew and chewrec are currently limited in their functionality and require you to understand how ctypes work. For examples on how to implement those functions please review the consumer module in the dtrace_ctypes package. The functions are named: simple_chew, simple_chewrec, simple_walk and simple_buffered_out_writer function.

The examples for using this libdtrace wrapper can be found in the examples/ctypes folder.

(c) 2011-2015 Thijs Metsch (c) 2013 engjoy UG (haftungsbeschraenkt)

python-dtrace's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

python-dtrace's Issues

Ctypes on Mac OS X

$ python syscall_count_continuous.py
Segmentation fault: 11

$ python -V
Python 2.7.6

xception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000718

VM Regions Near 0x718:
-->
__TEXT 00000001045b4000-00000001045b5000 [ 4K] r-x/rwx SM=COW /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libdtrace.dylib 0x00000001049626d5 dtrace_setopt + 253
1 libffi.dylib 0x00007fff90f98f44 ffi_call_unix64 + 76
2 libffi.dylib 0x00007fff90f99781 ffi_call + 853
3 _ctypes.so 0x00000001048c3762 _ctypes_callproc + 874
4 _ctypes.so 0x00000001048bdbad 0x1048bb000 + 11181
5 org.python.python 0x00000001045c92ac PyObject_Call + 99
6 org.python.python 0x0000000104645c00 PyEval_EvalFrameEx + 11370
7 org.python.python 0x0000000104642d62 PyEval_EvalCodeEx + 1413
8 org.python.python 0x00000001045e730a 0x1045bf000 + 164618
9 org.python.python 0x00000001045c92ac PyObject_Call + 99
10 org.python.python 0x00000001045d40cf 0x1045bf000 + 86223
11 org.python.python 0x00000001045c92ac PyObject_Call + 99
12 org.python.python 0x00000001046103f4 0x1045bf000 + 332788
13 org.python.python 0x000000010460bcfb 0x1045bf000 + 314619
14 org.python.python 0x00000001045c92ac PyObject_Call + 99
15 org.python.python 0x0000000104645c00 PyEval_EvalFrameEx + 11370
16 org.python.python 0x000000010464960e 0x1045bf000 + 566798
17 org.python.python 0x00000001046463e3 PyEval_EvalFrameEx + 13389
18 org.python.python 0x0000000104642d62 PyEval_EvalCodeEx + 1413
19 org.python.python 0x00000001046427d7 PyEval_EvalCode + 54
20 org.python.python 0x00000001046627bd 0x1045bf000 + 669629
21 org.python.python 0x0000000104662860 PyRun_FileExFlags + 133
22 org.python.python 0x00000001046623fd PyRun_SimpleFileExFlags + 769
23 org.python.python 0x0000000104673b23 Py_Main + 3051
24 libdyld.dylib 0x00007fff89bd35c9 start + 1

Cython version leaks all handles

In the Cython version, the handles are closed in __del__ both for DTraceConsumer and DTraceContinuousConsumer (used by DTraceConsumerThread). However, these are extension types, and Cython does not use __del__ for extension types (for whatever reason), so they will never be called (the explicit del self.consumer in DTraceConsumerThread's destructor, which is called as it's a normal Python object, just ensures the DTraceContinuousConsumer gets deallocated, but the consumer's __del__ still won't be called). I believe using __dealloc__ for the two extension classes would work (though handle must be checked for being non-null first).

This has also exposed a (minor) potential issue: with Cython, __init__ can end up being called multiple times, so technically handle could be initialised twice, leaking the old return from dtrace_open. To be truly correct, either __init__ should guard against this, or __cinit__ should be used instead, which will only ever be called once.

Aggregates with integer keys

Does python-dtrace support using integer keys in aggregates?

Here is a simple example:

from dtrace import DTraceConsumerThread
from subprocess import check_output, Popen, PIPE, STDOUT
import time

script = """syscall::read:entry /execname == "dd"/
{
    @values[arg2] = count();
}"""
command = ["dd", "if=/dev/zero", "of=/dev/null", "status=none", "bs=1M", "count=10"]

print("Running with dtrace-python")
dtrace_thread = DTraceConsumerThread(script, sleep=1)
dtrace_thread.start()
check_output(command)
dtrace_thread.stop()
dtrace_thread.join()

print("\nRunning with command line dtrace")
with Popen(["dtrace", "-n", script], stderr=STDOUT, stdout=PIPE) as proc:
    time.sleep(0.5) # let dtrace start
    check_output(command)
    proc.terminate()
    print(proc.stdout.read().decode())

python-dtrace prints:

1793 1 [b'/'] 1
1793 1 [b'\x80'] 1
1793 1 [b''] 10

The dtrace command prints

               47                1
              128                1
          1048576               10

Is there some way to recover the integer aggregate keys from the byte strings returned (b'/', b'\x80', b''), or is this functionality not yet supported?
Thanks in advance.

(I'm using version 0.0.12 installed from master, on FreeBSD 13)

Can't install on Mac OS X 10.10

Version info and build log below:

$ uname -a
Darwin mbp.local 14.0.0 Darwin Kernel Version 14.0.0: Fri Sep 19 00:26:44 PDT 2014; root:xnu-2782.1.97~2/RELEASE_X86_64 x86_64

$ sudo python setup.py install
running install
running build
running build_py
creating build
creating build/lib.macosx-10.10-intel-2.7
creating build/lib.macosx-10.10-intel-2.7/dtrace_ctypes
copying dtrace_ctypes/__init__.py -> build/lib.macosx-10.10-intel-2.7/dtrace_ctypes
copying dtrace_ctypes/consumer.py -> build/lib.macosx-10.10-intel-2.7/dtrace_ctypes
copying dtrace_ctypes/dtrace_structs.py -> build/lib.macosx-10.10-intel-2.7/dtrace_ctypes
running build_ext
cythoning dtrace/dtrace.pyx to dtrace/dtrace.c

Error compiling Cython file:
------------------------------------------------------------
...
    def __init__(self, chew_func=None, chewrec_func=None, out_func=None,
                 walk_func=None):
        '''
        Constructor. Gets a DTrace handle and sets some options.
        '''
        self.chew_func = chew_func or simple_chew
                                                ^
------------------------------------------------------------

dtrace/dtrace.pyx:239:49: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
    def __init__(self, chew_func=None, chewrec_func=None, out_func=None,
                 walk_func=None):
        '''
        Constructor. Gets a DTrace handle and sets some options.
        '''
        self.chew_func = chew_func or simple_chew
                                                ^
------------------------------------------------------------

dtrace/dtrace.pyx:239:49: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
                 walk_func=None):
        '''
        Constructor. Gets a DTrace handle and sets some options.
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
                                                         ^
------------------------------------------------------------

dtrace/dtrace.pyx:240:58: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
                 walk_func=None):
        '''
        Constructor. Gets a DTrace handle and sets some options.
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
                                                         ^
------------------------------------------------------------

dtrace/dtrace.pyx:240:58: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
        '''
        Constructor. Gets a DTrace handle and sets some options.
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
        self.out_func = out_func or simple_out
                                             ^
------------------------------------------------------------

dtrace/dtrace.pyx:241:46: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
        '''
        Constructor. Gets a DTrace handle and sets some options.
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
        self.out_func = out_func or simple_out
                                             ^
------------------------------------------------------------

dtrace/dtrace.pyx:241:46: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
        Constructor. Gets a DTrace handle and sets some options.
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
        self.out_func = out_func or simple_out
        self.walk_func = walk_func or simple_walk
                                                ^
------------------------------------------------------------

dtrace/dtrace.pyx:242:49: Cannot convert 'object (object, object, object, object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
        Constructor. Gets a DTrace handle and sets some options.
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
        self.out_func = out_func or simple_out
        self.walk_func = walk_func or simple_walk
                                                ^
------------------------------------------------------------

dtrace/dtrace.pyx:242:49: Cannot convert 'object (object, object, object, object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
    def __init__(self, script, chew_func=None, chewrec_func=None,
                 out_func=None, walk_func=None):
        '''
        Constructor. will get the DTrace handle
        '''
        self.chew_func = chew_func or simple_chew
                                                ^
------------------------------------------------------------

dtrace/dtrace.pyx:354:49: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
    def __init__(self, script, chew_func=None, chewrec_func=None,
                 out_func=None, walk_func=None):
        '''
        Constructor. will get the DTrace handle
        '''
        self.chew_func = chew_func or simple_chew
                                                ^
------------------------------------------------------------

dtrace/dtrace.pyx:354:49: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
                 out_func=None, walk_func=None):
        '''
        Constructor. will get the DTrace handle
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
                                                         ^
------------------------------------------------------------

dtrace/dtrace.pyx:355:58: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
                 out_func=None, walk_func=None):
        '''
        Constructor. will get the DTrace handle
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
                                                         ^
------------------------------------------------------------

dtrace/dtrace.pyx:355:58: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
        '''
        Constructor. will get the DTrace handle
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
        self.out_func = out_func or simple_out
                                             ^
------------------------------------------------------------

dtrace/dtrace.pyx:356:46: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
        '''
        Constructor. will get the DTrace handle
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
        self.out_func = out_func or simple_out
                                             ^
------------------------------------------------------------

dtrace/dtrace.pyx:356:46: Cannot convert 'object (object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
        Constructor. will get the DTrace handle
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
        self.out_func = out_func or simple_out
        self.walk_func = walk_func or simple_walk
                                                ^
------------------------------------------------------------

dtrace/dtrace.pyx:357:49: Cannot convert 'object (object, object, object, object, int __pyx_skip_dispatch)' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
        Constructor. will get the DTrace handle
        '''
        self.chew_func = chew_func or simple_chew
        self.chewrec_func = chewrec_func or simple_chewrec
        self.out_func = out_func or simple_out
        self.walk_func = walk_func or simple_walk
                                                ^
------------------------------------------------------------

dtrace/dtrace.pyx:357:49: Cannot convert 'object (object, object, object, object, int __pyx_skip_dispatch)' to Python object
building 'dtrace' extension
creating build/temp.macosx-10.10-intel-2.7
creating build/temp.macosx-10.10-intel-2.7/dtrace
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c dtrace/dtrace.c -o build/temp.macosx-10.10-intel-2.7/dtrace/dtrace.o
dtrace/dtrace.c:1:2: error: Do not use this file, it is the result of a failed Cython compilation.
#error Do not use this file, it is the result of a failed Cython compilation.
 ^
1 error generated.
error: command 'cc' failed with exit status 1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.