Code Monkey home page Code Monkey logo

whisper-to-graphite's Introduction

whisper-to-graphite Build Status

Read and send metrics from whisper files to graphite - Used to migrate to different graphite backends

Basically this little helper calculated the metric name from the filename of a whisper file and sends all timestamp/value tuples using the graphite protocol. TCP and UDP are supported. Additionally there is a NOP protocol which logs all data instead of sending it.

Usage

% ./whisper-to-graphite -h
Usage of ./whisper-to-graphite:
  -basedirectory string
    	Base directory where whisper files are located. Used to retrieve the metric name from the filename. (default "/var/lib/graphite/whisper")
  -directory string
    	Directory containing the whisper files you want to send to graphite again (default "/var/lib/graphite/whisper/collectd")
  -from int
    	Starting timestamp to dump data from
  -host string
    	Hostname/IP of the graphite server (default "127.0.0.1")
  -port int
    	graphite Port (default 2003)
  -pps int
    	Number of maximum points per second to send (0 means rate limiter is disabled)
  -protocol string
    	Protocol to use to transfer graphite data (tcp/udp/nop) (default "tcp")
  -retries int
    	How many connection retries worker will make before failure. It is progressive and each next pause will be equal to 'retry * 1s' (default 3)
  -to int
    	Ending timestamp to dump data up to (default 2147483647)
  -workers int
    	Workers to run in parallel (default 5)

Assuming you don't want to use this as testcase for your IO subsystem, you might want to ensure that you don't send the data to the host you are reading it from.

whisper-to-graphite's People

Contributors

bzed avatar felixoid avatar github-vincent-miszczak avatar vicpopov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

whisper-to-graphite's Issues

rate for metric

may be set rate for send metric?
1 worker does it too fast :)

The program produces points for every archive automatically

Hello. Here's the reference file for an issue
counter.wsp.gz

When you are trying to dump it with command ./whisper-to-graphite -basedirectory . -directory . -from 1581019200 -to 1581022800 -host localhost -port 42003 -workers 1, it produces the next data
dump.txt

There are 3 1581019200 and 1581022800 timestamps with aggregated values. I'm not entirely sure how is it possible, but because of that, the graph looks quite scary.
Screenshot_2020-02-10 New dashboard - Grafana
The data from 21:00 to 14:00 was reuploaded.

For me, it looks like the whisper dumps not only existing data but as well it applies the aggregation for some reason.

Maybe, you have a quick idea of how to fix it?

Empty points are converted into zero with current timestamp

Hello!
There some strange behavior on few whisper files. For example:

$ ./whisper-to-graphite -basedirectory /tmp/ -directory /tmp/TimeSite200.wsp -workers 1 -port 2004
2017/10/06 11:42:39 OK: /tmp/TimeSite200.wsp
# from another terminal
$ netcat -l 2004 | wc -l
30480
$ whisper-fetch.py /tmp/TimeSite200.wsp --from=0 | wc -l
17520

This is the file for example: TimeSite200.wsp.gz

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.