Comments (10)
Hi! I just started using this package for some custom metrics and was very disappointed to find that counters are incorrectly created in stackdriver as gauges.
To be clear, there is a SetGauge
method which should be used for gauges. IncrCounter
should be creating a metric of type cumulative
. If it does the same thing as SetGauge
then why does it have a different name, right?
from go-metrics-stackdriver.
^ @tam7t
from go-metrics-stackdriver.
So I'm understanding correctly, you're wanting to create an alert on rate-of-change of a counter metric and it looks like that is not possible when the counter is actually a GAUGE metric?
from go-metrics-stackdriver.
Not only alert on deltas but primarily draw charts with deltas. As far as I can see one can use delta as an Aligner but only delta and cumulative are supported.
Imagine I want to count the number of function calls or the number of requests. I can either calculate request rate or function calls rate in code and expose it as a Gauge or I can use a cumulative counter and compute deltas in query time. The latter is a very common scenario. Many open source services expose cumulative values which are later converted to deltas by either metrics agents or monitoring tools by themselves. As far as I see Google Cloud Monitoring tool also gives me ability to calculate deltas but for sure it doesn't allow me doing it for gauges. Both ALIGN_DELTA and ALIGN_RATE mention this in the docs: This aligner is valid for CUMULATIVE and DELTA metrics with numeric values.
I mean more or less this: #19
from go-metrics-stackdriver.
@tam7t any thoughts here?
from go-metrics-stackdriver.
A cumulative metric measures a value that constantly increases, such as “sent bytes count” for Firebase. Cumulative metrics are never drawn directly in practice; you always use aligners (discussed below) to turn them into gauge or delta metrics first. If you could draw the raw data for “sent bytes count,” you would see an ever-increasing line going up as the total number of sent bytes grows without bound.
What this library does for the go-metrics
counter type is report the actual raw data (ever-increasing line, except where I turned off and restarted the program):
Running the example/main.go
and plotting the counter metric using MQL:
fetch generic_task
| metric 'custom.googleapis.com/go-metrics/baz_counter'
| group_by [], [value_baz_counter_mean: aggregate(value.baz_counter)]
| window 1m
Produces a graph like this:
If we want to plot (or alert) on the rate of change of the counter we can use the following MQL:
fetch generic_task
| metric 'custom.googleapis.com/go-metrics/baz_counter'
| align delta_gauge()
| group_by [], [value_baz_counter_mean: aggregate(value.baz_counter)]
| window 1m
Here we see a stead 1200/minute rate (which is consistent with running 2 instances of example/main.go
each incrementing the counter once every 100ms).
I'm honestly not sure why the GUI does not provide the DELTA
aligner on gauge metrics, but it is possible to create an alert from these counters using MQL:
fetch generic_task
| metric 'custom.googleapis.com/go-metrics/baz_counter'
| align delta_gauge()
| group_by [], [value_baz_counter_mean: aggregate(value.baz_counter)]
| window 1m
| condition val() > 605
I can see the desire to switch the type to CUMULATIVE, as it appears that is how opentelemetry and other stackdriver built-in-counter metrics are reported - but I just want to demonstrate how to build alerts on the existing implementation.
from go-metrics-stackdriver.
@tam7t Wow. Thank you. I think this solves most of the problems in fact. I ot an impression that one cannot use delta aligner for gauges. It seems I have to invest time to get familiar with MQL and use it instead of clicking through UI.
from go-metrics-stackdriver.
Glad to hear that!
Be aware that in my tests | align delta_gauge()
was needed, I tried a few times with | align delta()
and while the resulting graphs looked the same I could not get alerts to fire.
My strategy to find those queries was to use the UI to build the initial aggregation, then click the MQL button to see what the MQL looks like and edit from there.
I think switching to CUMULATIVE
may also have behavior on overflows - I don't think I've actually tested out what happens when a counter rolls-over, but that may be another thing look at to ensure it doesn't cause an extra or missing alert.
from go-metrics-stackdriver.
@tam7t Thank you for all your support. Do you also have suggestions regarding time metrics sent by calling MeasureSinceWithLabels
. I try to visualize max
but I get an error that max cannot be used for distributions. I have to use high percentile as a workaround. You can see what I do in the example below. I'll appreciate all the suggestions if there is a better way (without a workaround) to find min/max values. Thanks
fetch generic_task
| metric custom.googleapis.com/blabla/refresh_time
| align delta()
| group_by
[metric.source],
[refresh_time: percentile(value.refresh_time, 99.99)]
| window 1m
from go-metrics-stackdriver.
When my process restarts I end up with huge negative spikes in the delta, as the value goes from some high number to some very low number. Is there anything I can do about that?
from go-metrics-stackdriver.
Related Issues (9)
- Error: Only one point can be written per TimeSeries HOT 4
- Graceful shutdown of Sink HOT 3
- Use protocmp for rpc testing
- NewSink constructor should accept an interface
- Sampling Results Inaccurate HOT 2
- Sink no longer implements metrics.MetricSink
- Support different metric types with the same key
- Duplicate TimeSeries encountered HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go-metrics-stackdriver.