Code Monkey home page Code Monkey logo

librato-metrics's Introduction

Librato Metrics

Gem Version Build Status Code Climate

A convenient Ruby wrapper for the Librato Metrics API.


Important note on breaking change

NOTE: Starting with version 2.1.0 librato-metrics requires a Librato account that supports tagged metrics. If your Librato account doesn't yet support tagged metrics please use the 1.6.1 version.


This gem provides granular control for scripting interactions with the Metrics core API. It is well suited for integrations, scripts, workers & background jobs. If you want to submit from a web app, take at look at librato-rails and/or librato-rack.

Installation

In your shell:

gem install librato-metrics

Then, in your application or script:

require 'librato/metrics'

Optional steps

For best performance we recommend installing yajl-ruby:

gem install yajl-ruby

If you are using jruby, you need to ensure jruby-openssl is available:

gem install jruby-openssl

Quick Start

If you are looking for the quickest possible route to getting a data into Metrics, you only need two lines:

Librato::Metrics.authenticate 'email', 'api_key'
Librato::Metrics.submit my_metric: { value: 42, tags: { host: 'localhost' } }

While this is all you need to get started, if you are sending a number of metrics regularly a queue may be easier/more performant so read on...

Authentication

Make sure you have an account for Librato and then authenticate with your email and API key (on your account page):

Librato::Metrics.authenticate 'email', 'api_key'

Sending Measurements

A measurement includes a metric name, value, and one or more tags. Tags include a name/value pair that describe a particular data stream. Each unique tag set creates an individual metric stream which can later be filtered and aggregated along.

Queue up a simple metric named temperature:

queue = Librato::Metrics::Queue.new
queue.add temperature: {value: 77, tags: { city: 'oakland' }}
queue.submit

Top-Level Tags

You can initialize Queue and/or Aggregator with top-level tags that will be applied to every measurement:

queue = Librato::Metrics::Queue.new(tags: { service: 'auth', environment: 'prod', host: 'auth-prod-1' })
queue.add my_metric: 10
queue.submit

Per-Measurement Tags

Optionally, you can submit per-measurement tags by passing a tags Hash when adding measurements:

queue.add my_other_metric: { value: 25, tags: { db: 'rr1' } }
queue.submit

For more information, visit the API documentation.

Querying Metrics

Get name and properties for all metrics you have in the system:

metrics = Librato::Metrics.metrics

Get only metrics whose name includes time:

metrics = Librato::Metrics.metrics name: 'time'

Retrieving Measurements

Get the series for exceptions in production grouped by sum within the last hour:

query = {
  resolution: 1,
  duration: 3600,
  group_by: "environment",
  group_by_function: "sum",
  tags_search: "environment=prod*"
}
Librato::Metrics.get_series :exceptions, query

For more information, visit the API documentation.

Aggregate Measurements

If you are measuring something very frequently e.g. per-request in a web application (order mS) you may not want to send each individual measurement, but rather periodically send a single aggregate measurement, spanning multiple seconds or even minutes. Use an Aggregator for this.

Aggregate a simple gauge metric named response_latency:

aggregator = Librato::Metrics::Aggregator.new
aggregator.add response_latency: 85.0
aggregator.add response_latency: 100.5
aggregator.add response_latency: 150.2
aggregator.add response_latency: 90.1
aggregator.add response_latency: 92.0

Which would result in a gauge measurement like:

{name: "response_latency", count: 5, sum: 517.8, min: 85.0, max: 150.2}

You can specify a source during aggregate construction:

aggregator = Librato::Metrics::Aggregator.new(tags: { service: 'auth', environment: 'prod', host: 'auth-prod-1' })

You can aggregate multiple metrics at once:

aggregator.add app_latency: 35.2, db_latency: 120.7

Send the currently aggregated metrics to Metrics:

aggregator.submit

Benchmarking

If you have operations in your application you want to record execution time for, both Queue and Aggregator support the #time method:

aggregator.time :my_measurement do
  # do work...
end

The difference between the two is that Queue submits each timing measurement individually, while Aggregator submits a single timing measurement spanning all executions.

If you need extra attributes for a Queue timing measurement, simply add them on:

queue.time :my_measurement do
  # do work...
end

Annotations

Annotation streams are a great way to track events like deploys, backups or anything else that might affect your system. They can be overlaid on any other metric stream so you can easily see the impact of changes.

At a minimum each annotation needs to be assigned to a stream and to have a title. Let's add an annotation for deploying v45 of our app to the deployments stream:

Librato::Metrics.annotate :deployments, 'deployed v45'

There are a number of optional fields which can make annotations even more powerful:

Librato::Metrics.annotate :deployments, 'deployed v46', source: 'frontend',
    start_time: 1354662596, end_time: 1354662608,
    description: 'Deployed 6f3bc6e67682: fix lotsa bugs…'

You can also automatically annotate the start and end time of an action by using annotate's block form:

Librato::Metrics.annotate :deployments, 'deployed v46' do
  # do work..
end

More fine-grained control of annotations is available via the Annotator object:

annotator = Librato::Metrics::Annotator.new

# list annotation streams
streams = annotator.list

# fetch a list of events in the last hour from a stream
annotator.fetch :deployments, start_time: (Time.now.to_i-3600)

# delete an event
annotator.delete_event 'deployments', 23

See the documentation of Annotator for more details and examples of use.

Auto-Submitting Metrics

Both Queue and Aggregator support automatically submitting measurements on a given time interval:

# submit once per minute
timed_queue = Librato::Metrics::Queue.new(autosubmit_interval: 60)

# submit every 5 minutes
timed_aggregator = Librato::Metrics::Aggregator.new(autosubmit_interval: 300)

Queue also supports auto-submission based on measurement volume:

# submit when the 400th measurement is queued
volume_queue = Librato::Metrics::Queue.new(autosubmit_count: 400)

These options can also be combined for more flexible behavior.

Both options are driven by the addition of measurements. If you are adding measurements irregularly (less than once per second), time-based submission may lag past your specified interval until the next measurement is added.

If your goal is to collect metrics every x seconds and submit them, check out this code example.

Setting Metric Properties

Setting custom properties on your metrics is easy:

# assign a period and default color
Librato::Metrics.update_metric :temperature, period: 15, attributes: { color: 'F00' }

Deleting Metrics

If you ever need to remove a metric and all of its measurements, doing so is easy:

# delete the metrics 'temperature' and 'humidity'
Librato::Metrics.delete_metrics :temperature, :humidity

You can also delete using wildcards:

# delete metrics that start with cpu. except for cpu.free
Librato::Metrics.delete_metrics names: 'cpu.*', exclude: ['cpu.free']

Note that deleted metrics and their measurements are unrecoverable, so use with care.

Using Multiple Accounts Simultaneously

If you need to use metrics with multiple sets of authentication credentials simultaneously, you can do it with Client:

joe = Librato::Metrics::Client.new
joe.authenticate 'email1', 'api_key1'

mike = Librato::Metrics::Client.new
mike.authenticate 'email2', 'api_key2'

All of the same operations you can call directly from Librato::Metrics are available per-client:

# list Joe's metrics
joe.metrics

There are two ways to associate a new queue with a client:

# these are functionally equivalent
joe_queue = Librato::Metrics::Queue.new(client: joe)
joe_queue = joe.new_queue

Once the queue is associated you can use it normally:

joe_queue.add temperature: { value: 65.2, tags: { city: 'san francisco' } }
joe_queue.submit

Thread Safety

The librato-metrics gem currently does not do internal locking for thread safety. When used in multi-threaded applications, please add your own mutexes for sensitive operations.

More Information

librato-metrics is sufficiently complex that not everything can be documented in the README. Additional options are documented regularly in the codebase. You are encouraged to take a quick look through the source for more.

We also maintain a set of examples of common uses and appreciate contributions if you have them.

Contribution

  • Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
  • Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
  • Fork the project and submit a pull request from a feature or bugfix branch.
  • Please review our code conventions.
  • Please include specs. This is important so we don't break your changes unintentionally in a future version.
  • Please don't modify the gemspec, Rakefile, version, or changelog. If you do change these files, please isolate a separate commit so we can cherry-pick around it.

Copyright

Copyright (c) 2011-2017 Solarwinds, Inc. See LICENSE for details.

librato-metrics's People

Contributors

chancefeick avatar cyx avatar dipth avatar gmckeever avatar gorsuch avatar jderrett avatar josephruscio avatar mheffner avatar neilmock avatar nextmat avatar nuvan avatar oggy avatar paul avatar portertech avatar techdubb avatar ys avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

librato-metrics's Issues

Segfaults

I'm consistently getting seg faults coming from this gem in my testing process. My testing environment is slightly odd, but here's the info -- configured to use the eventmachine adapter, as it's used within a goliath server:

-- Control frame information -----------------------------------------------
c:0030 p:---- s:0141 b:0141 l:000140 d:000140 CFUNC :connect
c:0029 p:0011 s:0138 b:0138 l:0024c0 d:000137 BLOCK /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:799
c:0028 p:0031 s:0136 b:0136 l:000135 d:000135 METHOD /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/timeout.rb:54
c:0027 p:0026 s:0124 b:0124 l:000123 d:000123 METHOD /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/timeout.rb:99
c:0026 p:0485 s:0118 b:0118 l:0024c0 d:0024c0 METHOD /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:799
c:0025 p:0011 s:0110 b:0110 l:000109 d:000109 METHOD /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:755
c:0024 p:0048 s:0107 b:0107 l:000106 d:000106 METHOD /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:744
c:0023 p:0025 s:0104 b:0104 l:000103 d:000103 METHOD /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1284
c:0022 p:0075 s:0097 b:0097 l:000096 d:000096 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/faraday-0.8.4/lib/faraday/adapter/net_http.rb:74
c:0021 p:0157 s:0092 b:0092 l:000091 d:000091 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/faraday-0.8.4/lib/faraday/adapter/net_http.rb:37
c:0020 p:0015 s:0084 b:0084 l:000083 d:000083 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/faraday-0.8.4/lib/faraday/response.rb:8
c:0019 p:0031 s:0080 b:0080 l:000079 d:000079 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/middleware/count_requests.rb:22
c:0018 p:0024 s:0076 b:0076 l:000075 d:000075 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/middleware/retry.rb:16
c:0017 p:0034 s:0071 b:0071 l:000070 d:000070 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/middleware/request_body.rb:11
c:0016 p:0090 s:0067 b:0067 l:000066 d:000066 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/faraday-0.8.4/lib/faraday/connection.rb:226
c:0015 p:0030 s:0058 b:0058 l:000057 d:000057 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/faraday-0.8.4/lib/faraday/connection.rb:99
c:0014 p:0025 s:0051 b:0051 l:000050 d:000050 METHOD /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/forwardable.rb:201
c:0013 p:0043 s:0046 b:0046 l:000036 d:000045 BLOCK /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/persistence/direct.rb:22
c:0012 p:---- s:0042 b:0042 l:000041 d:000041 FINISH
c:0011 p:---- s:0040 b:0040 l:000039 d:000039 CFUNC :each
c:0010 p:0057 s:0037 b:0037 l:000036 d:000036 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/persistence/direct.rb:19
c:0009 p:0078 s:0029 b:0029 l:000028 d:000028 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/processor.rb:31
c:0008 p:0068 s:0025 b:0025 l:000024 d:000024 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/processor.rb:94
c:0007 p:0022 s:0021 b:0021 l:000020 d:000020 METHOD /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/aggregator.rb:37
c:0006 p:0031 s:0017 b:0017 l:0003a8 d:000016 BLOCK /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/bundler/gems/rtb_common-dc1ef59fdc57/lib/reporting.rb:24
c:0005 p:---- s:0014 b:0014 l:000013 d:000013 FINISH
c:0004 p:---- s:0012 b:0012 l:000011 d:000011 CFUNC :call
c:0003 p:0063 s:0009 b:0009 l:001ea0 d:000008 BLOCK /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/eventmachine-1.0.0/lib/eventmachine.rb:1037
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:---- s:0002 b:0002 l:000001 d:000001 TOP

-- Ruby level backtrace information ----------------------------------------
/Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/eventmachine-1.0.0/lib/eventmachine.rb:1037:in block in spawn_threadpool' /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/eventmachine-1.0.0/lib/eventmachine.rb:1037:incall'
/Users/joshuah/.rvm/gems/ruby-1.9.3-p125/bundler/gems/rtb_common-dc1ef59fdc57/lib/reporting.rb:24:in block in gauge' /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/aggregator.rb:37:inadd'
/Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/processor.rb:94:in autosubmit_check' /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/processor.rb:31:insubmit'
/Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/persistence/direct.rb:19:in persist' /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/persistence/direct.rb:19:ineach'
/Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/persistence/direct.rb:22:in block in persist' /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/forwardable.rb:201:inpost'
/Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/faraday-0.8.4/lib/faraday/connection.rb:99:in post' /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/faraday-0.8.4/lib/faraday/connection.rb:226:inrun_request'
/Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/middleware/request_body.rb:11:in call' /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/middleware/retry.rb:16:incall'
/Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/librato-metrics-0.7.1/lib/librato/metrics/middleware/count_requests.rb:22:in call' /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/faraday-0.8.4/lib/faraday/response.rb:8:incall'
/Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/faraday-0.8.4/lib/faraday/adapter/net_http.rb:37:in call' /Users/joshuah/.rvm/gems/ruby-1.9.3-p125/gems/faraday-0.8.4/lib/faraday/adapter/net_http.rb:74:inperform_request'
/Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1284:in request' /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:744:instart'
/Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:755:in do_start' /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:799:inconnect'
/Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/timeout.rb:99:in timeout' /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/timeout.rb:54:intimeout'
/Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:799:in block in connect' /Users/joshuah/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:799:inconnect'

Empty measurement payload can be submitted

This was occurring before I added code to force submit a queue over a given size to address #81. Confirmed with logs on server that the payload is sent with only: {:measure_time => 1377612480} and no measurements.

Exception:

..../gems/librato-metrics-1.1.1/lib/librato/metrics/middleware/expects_status.rb:19:in `on_complete': {:method=>:post, :body=>"{"errors":{"request":["Must contain a measurement"]}}", :url=>#<URI::HTTPS:0x00000002d1e050 URL:https://metrics-api.librato.com/v1/metrics>, :request_headers=>{"User-Agent"=>"librato-metrics/1.1.1 (ruby; 1.9.3p194; x86_64-linux) direct-faraday/0.8.8", "Content-Type"=>"application/json", "Authorization"=>"..."}, :parallel_manager=>nil, :request=>{:open_timeout=>20, :timeout=>30, :proxy=>nil}, :ssl=>{}, :request_body=>"{"measure_time":1377612480}", :status=>400, :response_headers=>{"content-type"=>"application/json;charset=utf-8", "date"=>"Tue, 27 Aug 2013 14:09:22 GMT", "server"=>"nginx/1.0.15", "status"=>"400 Bad Request", "content-length"=>"53", "connection"=>"Close"}, :response=>
#<Faraday::Response:0x00000002d020a8 @env={...}, @on_complete_callbacks=[]>} (Librato::Metrics::ClientError)

uninitialized constant Faraday::Response::Middleware (NameError)

I get this error:

[snip path]gems/librato-metrics-2.1.2/lib/librato/metrics/middleware/count_requests.rb:5:in `module:Middleware': uninitialized constant Faraday::Response::Middleware (NameError)
Did you mean? Faraday::Middleware

Gemfile:

source 'https://rubygems.org'

gem 'librato-metrics'
gem 'mysql2'
$ bundle show
Gems included by the bundle:
  * aggregate (0.2.3)
  * bundler (1.17.2)
  * faraday (2.3.0)
  * faraday-net_http (2.0.3)
  * librato-metrics (2.1.2)
  * mysql2 (0.5.4)
  * ruby2_keywords (0.0.5)

Ruby code:

require 'librato/metrics'

(That is the entire file. Simply requiring the module throws the exception.)

Test mode

I was integrating our app with Librato Metrics and noticed that I needed to override some methods to avoid networking, since all I wanted on my specs was a should_receive check.

Although we use VCR for most of out network related specs, it would be nice to have a test mode. Is that of any value to anyone else? I was thinking about implementing this and sending a PR.

Any feedback is welcome, thanks!

Queue should aggregate metrics and flush on a given period

Add capability to queue abstraction to aggregate measurements and automatically flush the aggregate on a given period. Example use case is measuring something per-request and flushing a single aggregate measurement every 60 seconds.

Drop support for ruby 1.8

Trying to balance dependencies between newer and older versions of ruby along w/ jruby, ree etc has become increasingly difficult. I'm inclined to cut a v2.0 which drops support for anything before the 1.9 syntax (REE, MRI 1.8, jruby 1.8 mode etc).

Any thoughts / strong opinions? @obfuscurity @jderrett @mheffner @josephruscio

#fetch auto-paginates when retrieving large groups of measurements

We currently auto-paginate metric listing, but don't auto-paginate when retrieving large measurement sets.

fetch should take care of this transparently. We should consider also some kind of block-version which allows streaming operation since in some cases the retrieve measurement set may be very large and require a large number of network ops.

Related: #78.

Authenticating multiple clients in one process

As I was building some more tooling for our infrastructure I had the requirement to dump metrics into multiple Librato accounts. The way the client works right now is that there's a global authentication per process, so that's not really possible right now. To work around it I patched the Processor module to allow setting the client via an accessor. I'm curious (hence the issue) if there's any interest on e.g. making the Queue and other classes configurable in the way that you can pass in a client instead of relying on the global one?

Allow proxy configuration

Related to a handful of user requests, librato-metrics should support proxy configuration of some sort. Since there is not currently a configuration file, the intial feature might just be added to the API vs requiring a config.

"1.1 is not a valid source name"

Hi,

i use metrics to report some data about our iphone clients. We collect information like app version, hardware version, ios version. I report the total amount of this data with the different version as "source". But for some reason the "1.1" is invalid. In contrast "2.0" is ok. "1.1.1" is ok too. Is there a reason for this?

Can you have a look? This is the request body:

request_body{"gauges":[{"source":"iPod2_1","measure_time":1363857180,"value":5,"name":"device_version"},{"source":"iPod1_1","measure_time":1363857180,"value":3,"name":"device_version"},{"source":"iPod5_1","measure_time":1363857180,"value":6,"name":"device_version"},{"source":"iPod4_1","measure_time":1363857180,"value":16,"name":"device_version"},{"source":"iPod3_1","measure_time":1363857180,"value":4,"name":"device_version"},{"source":"iPhone1_2","measure_time":1363857180,"value":11,"name":"device_version"},{"source":"iPhone5_2","measure_time":1363857180,"value":179,"name":"device_version"},{"source":"iPhone5_1","measure_time":1363857180,"value":15,"name":"device_version"},{"source":"iPhone4_1","measure_time":1363857180,"value":240,"name":"device_version"},{"source":"iPhone3_3","measure_time":1363857180,"value":2,"name":"device_version"},{"source":"iPhone3_2","measure_time":1363857180,"value":2,"name":"device_version"},{"source":"iPhone3_1","measure_time":1363857180,"value":218,"name":"device_version"},{"source":"iPhone2_1","measure_time":1363857180,"value":52,"name":"device_version"},{"source":"x86_64","measure_time":1363857180,"value":127,"name":"device_version"},{"source":"iPad1_1","measure_time":1363857180,"value":15,"name":"device_version"},{"source":"iPad3_6","measure_time":1363857180,"value":8,"name":"device_version"},{"source":"iPad3_4","measure_time":1363857180,"value":11,"name":"device_version"},{"source":"iPad3_3","measure_time":1363857180,"value":18,"name":"device_version"},{"source":"iPad3_1","measure_time":1363857180,"value":19,"name":"device_version"},{"source":"iPad2_7","measure_time":1363857180,"value":5,"name":"device_version"},{"source":"iPad2_5","measure_time":1363857180,"value":7,"name":"device_version"},{"source":"iPad2_4","measure_time":1363857180,"value":4,"name":"device_version"},{"source":"iPad2_2","measure_time":1363857180,"value":39,"name":"device_version"},{"source":"iPad2_1","measure_time":1363857180,"value":32,"name":"device_version"},{"source":"6.1","measure_time":1363857180,"value":251,"name":"os_version"},{"source":"6.0","measure_time":1363857180,"value":29,"name":"os_version"},{"source":"5.1","measure_time":1363857180,"value":1,"name":"os_version"},{"source":"4.3","measure_time":1363857180,"value":1,"name":"os_version"},{"source":"5.1.1","measure_time":1363857180,"value":51,"name":"os_version"},{"source":"5.0.1","measure_time":1363857180,"value":1,"name":"os_version"},{"source":"6.1.3","measure_time":1363857180,"value":11,"name":"os_version"},{"source":"6.1.2","measure_time":1363857180,"value":552,"name":"os_version"},{"source":"6.1.1","measure_time":1363857180,"value":27,"name":"os_version"},{"source":"6.0.2","measure_time":1363857180,"value":11,"name":"os_version"},{"source":"6.0.1","measure_time":1363857180,"value":72,"name":"os_version"},{"source":"3.1.3","measure_time":1363857180,"value":2,"name":"os_version"},{"source":"4.3.5","measure_time":1363857180,"value":5,"name":"os_version"},{"source":"4.3.4","measure_time":1363857180,"value":1,"name":"os_version"},{"source":"4.3.3","measure_time":1363857180,"value":4,"name":"os_version"},{"source":"4.3.1","measure_time":1363857180,"value":1,"name":"os_version"},{"source":"4.2.1","measure_time":1363857180,"value":18,"name":"os_version"},{"source":"2.0","measure_time":1363857180,"value":1,"name":"app_version"},{"source":"34","measure_time":1363857180,"value":36,"name":"app_version"},{"source":"32","measure_time":1363857180,"value":4,"name":"app_version"},{"source":"31","measure_time":1363857180,"value":53,"name":"app_version"},{"source":"30","measure_time":1363857180,"value":7,"name":"app_version"},{"source":"1.1 ","measure_time":1363857180,"value":1,"name":"app_version"},{"source":"28","measure_time":1363857180,"value":15,"name":"app_version"},{"source":"26","measure_time":1363857180,"value":2,"name":"app_version"},{"source":"25","measure_time":1363857180,"value":4,"name":"app_version"},{"source":"24","measure_time":1363857180,"value":7,"name":"app_version"},{"source":"23","measure_time":1363857180,"value":10,"name":"app_version"},{"source":"21","measure_time":1363857180,"value":4,"name":"app_version"},{"source":"20","measure_time":1363857180,"value":2,"name":"app_version"},{"source":"1.2.4","measure_time":1363857180,"value":879,"name":"app_version"},{"source":"19","measure_time":1363857180,"value":1,"name":"app_version"},{"source":"18","measure_time":1363857180,"value":13,"name":"app_version"}]}response#Faraday::Response:0x7332cd0status400body{"errors":{"params":{"source":["1.1 is not a valid source name"],"name":"app_version"}}}

Autosubmit Interval failling...

Hi, I have a service to send info to Librato in interval of each 5 minutes, yesterday I become to have the error {"errors":{"params":{"measure_time":["is too far in the past"]}},"request_time":1475920599}", everytime a long of one entire day.
My doubt is... How will the Librato gem send a value too old if my autosubmit_interval is equal to 5 minutes?

This is my service:

require 'librato/metrics'
require 'singleton'
class LibratoService
  include Singleton

  AUTOSUBMIT_COUNT=5
  AUTOSUBMIT_INTERVAL=5.minutes

  def initialize
    Librato::Metrics.authenticate(ENV['LIBRATO_EMAIL'], ENV['LIBRATO_API_KEY'])
    @queue = Librato::Metrics::Queue.new(autosubmit_interval: AUTOSUBMIT_INTERVAL)
    @aggregator = Librato::Metrics::Aggregator.new(autosubmit_interval: AUTOSUBMIT_INTERVAL)
  end

  def aggregator_watcher(metric, value)
    @aggregator.add("#{prefix_metric_name(metric)}" => value)
  end

  def latency_watcher(metric)
    @queue.time prefix_metric_name(metric) do
      yield
    end
  end

  def send_gauge(metric, value)
    Librato::Metrics.submit "#{prefix_metric_name(metric)}" => value
  end

  def send_metrics
    @queue.submit
    @aggregator.submit
  rescue Librato::Metrics::CredentialsMissing => e
    LoggingService.log(e.message)
  end

  private

  def prefix_metric_name(metric_name)
    "Notification.#{metric_name}"
  end
end

Annotate gets a start_time before the end_time

Hello,

I'm using the client.annotate method annotate the beginning and end of a method call.

The problem I'm having is that, when the process takes too little time to complete, less then a second to be more precise, the Librato API responds with a 400 error saying that start_time is greater then end_time. Here is an example response:

#<struct Faraday::Env method=:put, body="{\"errors\":{\"params\":{\"start_time\":[\"must be before end_time\"]}}}", url=#<URI::HTTPS:0x00000001351910 URL:https://metrics-api.librato.com/v1/annotations/system.timed_action/68919280>, request=#<Faraday::RequestOptions timeout=30, open_timeout=20>, request_headers={"User-Agent"=>"librato-metrics/1.5.0 (ruby; 2.0.0p384; x86_64-linux-gnu) direct-faraday/0.9.1", "Content-Type"=>"application/json", "Authorization"=>"xxxxxxxxx"}, ssl=#<Faraday::SSLOptions verify=true>, parallel_manager=nil, params=nil, response=nil, response_headers={"content-type"=>"application/json;charset=utf-8", "date"=>"Mon, 01 Jun 2015 18:11:13 GMT", "server"=>"nginx", "status"=>"400 Bad Request", "content-length"=>"64", "connection"=>"Close"}, status=400>
//Extracted body
{ 
  "errors": {
    "params": {
      "start_time": [ 
        "must be before end_time"
      ]
    }
  }
}"

Sample call:

client = Librato::Metrics.Client.new('[email protected]', 'MUCH_SECRET_SUCH_KEY') 
client.annotate('system.timed_action', 'sample time action') do
  # do stuff...
end

Fake Metrics Client

Is there already or the possibility of a fake metrics client?

I want to be able to use librato on development the same way I use it in production without having the hassle of checking the environment condition on each request.

If there's not, how are people working around this?

Would this be a desirable addition to this gem? For example:

if ENV['RACK_ENV'] != 'production'
  Librato::Metrics.client = Librato::FakeClient.new(logger: ENV['RACK_ENV'] == 'test' ? nil : $stdout)
end

In the example above we define a fake client that logs the data submissions on development and nothing on test, replacing the 'real' client outside the production environment.

Add warning message before test runs - production data pruned

Hi,

I've just checked out the repo so I could contribute, ran the tests and had 53 failures without touching the code. The rspec output showed always the same error, saying that there was a missing TEST_API_USER and TEST_API_KEY ENV variable so I went to the librato UI of my production account and created a new token for testing and passed this token to the rspec run. The tests started to run and I've noticed a decline in my production metrics in librato UI.

Unfortunately, the test run deleted all my production metrics within seconds due to this line:

def delete_all_metrics

This is really frustrating and it just doesn't look right to me that this can happen at all and that no TEST environment has been evaluated prior to deleting the metrics.

Shouldn't there be at least a big warning sign on the public github repo warning devs about running this in their non-test accounts ? Or even stating that there's a need for an extra test account in order to run the tests would have saved me. One could also include a warning box in spec_helper that runs in a before(:suite) hook with a sleep of 10 seconds so people have a chance to ctrl-c the run ? I wasn't aware of the consequences at all and I would like others to not nuke their production data like I did by running the simple command:

bundle exec rspec

IMHO, this is really not how it should work.

Allow Queue to be submitted asynchronously

We ran into trouble when using Delayed Jobs / Sidekiq to #submit the Queue asynchronously. We want to do this so we could submit custom metrics from Rails controllers without blocking the web request. The Librato::Metrics::Queue object caused an exception when it was deserialized, so, we built a small wrapper to handle this use case:

class MetricsQueueSubmitter
  attr_reader :queued
  def initialize(queue)
    @queued = queue.queued
  end

  def submit
    Librato::Metrics::Queue.new.merge!(queued).submit
  end

  class << self
    def submit_async(queue)
      new(queue).delay.submit
    end
  end
end

# Usage
queue = Librato::Metrics::Queue.new
queue.add 'key' => 2
MetricsQueueSubmitter.submit_async(queue)

It would be nice if this could be done without having to use the wrapper to retrieve .queued and merge.

Also, if there is there a way to do this currently that I missed, please let me know :)

Sporadic "Must contain a measurement" errors

This has now been reported by a handful of users (IIRC e.g. @roidrage, @eric, @ doctorjnupe, etc). Sporadically the gem will return a 400 indicating no measurement was included in the request, even though the request does contain a measurement.

broken with ruby 1.8.7?

root@ip-10-2-17-207:# irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'librato/metrics'
NameError: uninitialized constant Librato::Metrics::Client::Forwardable
from /var/lib/gems/1.8/gems/librato-metrics-1.0.0/lib/librato/metrics/client.rb:5
from /usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:36:in gem_original_require' from /usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:36:inrequire'
from /var/lib/gems/1.8/gems/librato-metrics-1.0.0/lib/librato/metrics.rb:9
from /usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:59:in gem_original_require' from /usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:59:inrequire'
from (irb):2
from :0
irb(main):003:0> quit
root@ip-10-2-17-207:
# ruby -v
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

Am I doing something wrong?

Client should test for invalid data

In the spirit of #69 the client should already test if some data is invalid to prevent it from being sent to the server in the first place. I stumbled across a coding error where I sent much more data to the server as intended, and the librato server correctly returned an error ("metrics is configured as a counter, but..".

I guess this error could have been raised from the gem already, without the roundtrip.

Not able to define 'source' in Timed Aggregator

I am using Librato Timed Aggregator to send metrics to Librato. I use the following code :

timed_performance_aggregator = Librato::Metrics::Aggregator.new(:autosubmit_interval => 60)

But i also want to define source for the each metric sent to Librato. I am using the following command for that :

timed_performance_aggregator.add 'view_runtime' => {:source=>'My source',:value=>654.87}

The above command does not work in the sense that the metric is submitted to Librato but source is always showing 'undefined'.

Although when i define source when defining timed aggregator like :

timed_performance_aggregator = Librato::Metrics::Aggregator.new(:source=>'My source')

it works.

Please tell me if I am doing anything wrong.

Deprecate autosubmit support

We currently support autosubmit_interval for Queue & `Aggregator, as described here.

Unfortunately there isn't a way to make this automatically submit at regular intervals without introducing a running loop or background thread (and handling related thread safety), both of which can introduce unexpected behavior in the host's code. For this reason the current implementation only checks if it should submit when it receives a measurement - when this happens rarely submission period can run well over the specified submit interval.

We've had a number of people reasonably confused by this behavior (likely the cause of #125) so I suggest we just drop it entirely since rolling your own version of this behavior is pretty trivial and much less confusing.

Autosubmitting based on measurement count autosubmit_count doesn't have this issue and should probably remain.

JSON, wrong number of arguments (given 0, expected 1+)

#<ArgumentError: wrong number of arguments (given 0, expected 1+)> 
  /app/vendor/ruby-2.3.1/lib/ruby/2.3.0/json/common.rb:467:in `JSON' 
  /app/vendor/bundle/ruby/2.3.0/gems/librato-metrics-2.0.0/lib/librato/metrics/persistence/direct.rb:21:in `block in persist' 
  /app/vendor/bundle/ruby/2.3.0/gems/librato-metrics-2.0.0/lib/librato/metrics/persistence/direct.rb:20:in `each' 
  /app/vendor/bundle/ruby/2.3.0/gems/librato-metrics-2.0.0/lib/librato/metrics/persistence/direct.rb:20:in `persist' 
  /app/vendor/bundle/ruby/2.3.0/gems/librato-metrics-2.0.0/lib/librato/metrics/processor.rb:34:in `submit'

started happening when upgrading to 2.0.0

Road to 2.0

Our dependencies are getting creaky and some parts of this gem are long in the tooth so I wanted to go ahead and put together a list of what I think we need to do to transition to a solid 2.0

  • Remove support for ruby 1.8 / REE
  • Ensure support for ruby 2.2
  • Remove support for deprecated versions of methods
  • Unlock faraday version dependency (possible once we don't need to support 1.8)
  • Don't require multijson as a hard dependency, only use if present (otherwise use stdlib json)
  • Upgrade rspec stack to modern versions / conventions (expect, etc)

Make sure proper values are passed for metrics

Because of an error in my code I ended up passing a bad hash as a value:

data = { :type => :counter, :value => { :value => 35 } }
Librato::Metrics.submit :my_metric => data

That resulted in sending this:

{
    "gauges": [
        {
            "name": "my_metric",
            "source": null,
            "value": {
                "value": 35
            }
        }
    ]
}

It would be great if the library recognized my error before sending it.

Map metrics "scope" parameter to "source", instead of hard-coding "source"

At least in our use-case, it looks like it makes more sense to map the "Scope" parameter of yammer metrics into a "source".

Scope in yammer metrics is:
Scope
An optional name describing the metric’s scope. Useful for when you have multiple instances of a class.

Source in librato documentation is:
source
(optional) A string which describes the originating source of a measurement when that measurement is tracked across multiple members of a population. Examples: foo.bar.com, user-123, 77025.

Right now scopes in this yammer metrics connector are being mapped to the metric name, creating an awful lot of metrics if you have a big number of scopes.

Optionally being able to map scope<=>source would be great, I think.

Fetch response is case insensitive

If possible, it would be nice to treat the key strings in the fetch response as case insensitive.

The following query will return data for the source "bLaH", but given the API downcases the source names, the hash lookup would then break:

Librato::Metrics.fetch(metric_name, :count => count, :source => "bLaH")["bLaH"]

Maybe a Hash override that downcased key values before lookup?

uninitialized constant Class::RUBY_ENGINE (NameError)

ip-10-66-83-30% ruby -v
ruby 1.8.7 (2010-06-23 patchlevel 299) [i686-linux]

ip-10-66-83-30% gem -v
1.3.7

Adding metric to queue: qa.i-ca5e3eaa.rabbitmq.website_worker_search_worker.consumers 1 1325281816
Attempting to flush metrics to Librato
/usr/lib/ruby/gems/1.8/gems/librato-metrics-0.2.2/lib/librato/metrics/simple.rb:87:in user_agent': uninitialized constant Class::RUBY_ENGINE (NameError) from /usr/lib/ruby/gems/1.8/gems/librato-metrics-0.2.2/lib/librato/metrics/simple.rb:100:incommon_headers'
from /usr/lib/ruby/gems/1.8/gems/librato-metrics-0.2.2/lib/librato/metrics/simple.rb:49:in connection' from /usr/lib/ruby/gems/1.8/gems/librato-metrics-0.2.2/lib/librato/metrics/persistence/direct.rb:14:inpersist'
from /usr/lib/ruby/gems/1.8/gems/librato-metrics-0.2.2/lib/librato/metrics/queue.rb:75:in submit' from /usr/lib/ruby/gems/1.8/gems/recognizer-0.0.2/lib/recognizer/librato.rb:24:ininitialize'
from /usr/lib/ruby/gems/1.8/gems/recognizer-0.0.2/lib/recognizer/librato.rb:23:in synchronize' from /usr/lib/ruby/gems/1.8/gems/recognizer-0.0.2/lib/recognizer/librato.rb:23:ininitialize'
from /usr/lib/ruby/gems/1.8/gems/recognizer-0.0.2/lib/recognizer/librato.rb:19:in loop' from /usr/lib/ruby/gems/1.8/gems/recognizer-0.0.2/lib/recognizer/librato.rb:19:ininitialize'
from /usr/lib/ruby/gems/1.8/gems/recognizer-0.0.2/lib/recognizer/librato.rb:18:in new' from /usr/lib/ruby/gems/1.8/gems/recognizer-0.0.2/lib/recognizer/librato.rb:18:ininitialize'
from /usr/lib/ruby/gems/1.8/gems/recognizer-0.0.2/lib/recognizer.rb:14:in new' from /usr/lib/ruby/gems/1.8/gems/recognizer-0.0.2/lib/recognizer.rb:14:inrun'
from /usr/lib/ruby/gems/1.8/gems/recognizer-0.0.2/bin/recognizer:8
from /usr/bin/recognizer:19:in `load'
from /usr/bin/recognizer:19

Incrementing Counters (similar to Librato Rails gem)

Hello, the quick-start in librato-rails (https://github.com/librato/librato-rails#quick-start)

Shows an example of incrementing a counter:

# keep counts of key events
Librato.increment 'user.signup'

In README in this repo shows counters in this format:

Librato::Metrics.submit :my_metric => {:type => :counter, :value => 1002, :source => 'myapp'}

How does this translate? Would this be the equiv?

Librato::Metrics.submit 'user.signup' => { :type => :counter, :value => 1 }

Setting measure_time on queue should enable skip_measurement_times

If a measure_time is set on a queue, it would seem reasonable that measurements within the queue would use that queue-level measure_time. Right now you also have to explicitly set skip_measurement_times=>true to avoid each individual measurement getting its own measure_time set to the current epoch.

If a measure_time is set on the queue, can we set skip_measurement_times=true automatically?

Support a call-back block context for timed queues

For a timed queue it would be nice to have a method that could invoke a callback to determine the current value of a metric to submit. For example, the callback might poll an external system to determine the current value of a metric:

queue.add :total_items do
   read_total_item_count()
end

/cc @erichmond

V2 references Faraday::Error::ConnectionFailed, which doesn't exist in newer versions of Faraday

A long time ago, Faraday::Error::ConnectionFailed was moved to Faraday::ConnectionFailed. This mismatch results in the following error:

Check failed to run: uninitialized constant Faraday::Error::ConnectionFailed
Did you mean?  Faraday::ConnectionFailed

Here's the line that raises the exception: https://github.com/librato/librato-metrics/blob/v2.1.2/lib/librato/metrics/middleware/retry.rb#L19

The error class was updated on the 1.x branch here, but the 2.x branch still uses the old class.

"list indices must be integers, not unicode" in error response handling

I triggered an error response from a script using the librato metrics lib and rather than displaying a sane error message I got a stack trace (that didn't name the error). The server I'm on is using Python 2.6.6.

Digging into exceptions.py, I found the response code is 400 and the response body looks like this:

{u'errors': {u'request': [u'Must contain a measurement']}}

Here is a copy of the stack trace, minus the lines from my own script.

  File "/usr/local/lib/python2.6/dist-packages/librato_metrics-0.4.3-py2.6.egg/librato/queue.py", line 58, in submit
    self.connection._mexe("metrics", method="POST", query_props=c)
  File "/usr/local/lib/python2.6/dist-packages/librato_metrics-0.4.3-py2.6.egg/librato/__init__.py", line 144, in _mexe
    resp_data, success, backoff = self._process_response(resp, backoff)
  File "/usr/local/lib/python2.6/dist-packages/librato_metrics-0.4.3-py2.6.egg/librato/__init__.py", line 125, in _process_response
    raise exceptions.get(resp.status, resp_data)
  File "/usr/local/lib/python2.6/dist-packages/librato_metrics-0.4.3-py2.6.egg/librato/exceptions.py", line 67, in get
    for m in resp_data['errors'][key][v]:
TypeError: list indices must be integers, not unicode
Exception caught at 341: list indices must be integers, not unicode

Multidimensional check is not always correct

Take the following code:

measure_time = (Time.now.tv_sec / 300) * 300

Librato::Metrics.authenticate 'token', opts[:appoptics_token]
queue = Librato::Metrics::Queue.new(:prefix => "foo",
                                    :time => measure_time)
  queue.add "mymetric": {
    value: 1.34,
    tags: {"foo" => "bar"}
  }

This leads to a payload that includes a top-level :measure_time => <> key which is not valid for multidimensional (and fails to submit). This appears to be because we use a has_tags? check on the Queue, however the queue could legitimately not have any tags because they are added on the individual measurements. Therefore, it seems like the check must be performed on the individual measurements and not on the queue itself.

Support a default source for measurements

We received a request to "Have an option to set default source in the Librato::Metrics.authenticate call".

Maybe this could look something like:
Librato::Metrics.default_source = 'foo'

somewhat similar to the default source behavior in aggregator = Librato::Metrics::Aggregator.new(:source => 'foobar')

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.