Comments (9)
We use exometer_report_opentsdb
exclusively.
from elixometer.
I'm assuming that you're talking about the pobox stuff; Elixometer also has a genserver that serves to serialize config changes. None of the metric updates go through that process.
The reason Elixometer has pobox is to shed load if metrics are updated en masse. We experienced a pretty significant performance bottleneck with exometer when we threw many metrics updates at it at once. Adding pobox completely fixes this by dropping updates after the process mailbox reaches 1000, which is a large amount of backlog. If you're under the limit, you won't be affected, and if you're over this limit, I think discarding metrics is better than slowing your app down.
from elixometer.
Hey @scohen, thanks for the reply! ❤️
The reason Elixometer has pobox is to shed load if metrics are updated en masse. We experienced a pretty significant performance bottleneck with exometer when we threw many metrics updates at it at once.
Interesting! Have you found out which of exometer's processes was the culprit? Was it the reporter itself, that is only capable of receiving and handling one metric/datapoint message at a time? Or was it the underlying gen_tcp
setup?
after the process mailbox reaches 1000
Do you recall what's the flush interval I could use to replicate that?
The main question origin is: I'm very new to exometer and I don't really grok it fully yet. I'm looking for a drop-in replacement for my folsom/folsomite setup, where it simply blows up with around 3k total unique metrics (due to ETS leak). I'm dumping the aggregated metrics to hostedgraphite every 40 seconds. I know I can fix this by just bumping ERL_ETS_MAX_TABLES
, but that won't scale in the long run, and I feel there should be a way to do better, without dropping metrics overflow on the floor. So I guess what I'm looking for is a test case that'll allow me to benchmark this properly, before I make the production switch (staging is a lie).
from elixometer.
@aerosol, The problem wasn't with the flush interval, but because exometer uses :gen_server.call
somewhere in its stack. We were flooding the exometer process with tens of thousands of messages at once, and the calls were synchronous, and it would take 30ms or so for the process to clear its queue.
I don't think having tens of thousands of messages will be a problem, but I'm not extremely familiar with exometer
's internals. You might want to ask Ulf about this. The problem we fixed was due to many, many messages being created in a very short time.
from elixometer.
@scohen awesome.
We were flooding the exometer process with tens of thousands of messages at once, and the calls were synchronous
How many unique metrics (as in "metric name") did you have?
from elixometer.
BTW, do you think it'd be worthwhile to extend elixometer with a metric for the number of messages dropped?
from elixometer.
How many unique metrics (as in "metric name") did you have?
Per app? Maybe 10 - 50.
BTW, do you think it'd be worthwhile to extend elixometer with a metric for the number of messages dropped?
In our world, dropping doesn't happen often, and I didn't think it was that important in the grand scheme of things. Exometer can process tens of thousands of messages per second, and we've only ever had a single application cross this threshold to where exometer became a bottleneck.
The pobox
library handles dropping of messages, so we'd have to instrument that somehow.
from elixometer.
Thank you, this helps a lot! Last final question and we can close this: which reporter plugin did you use?
from elixometer.
@aerosol I'm going to close this, feel free to ask more questions if any come up.
from elixometer.
Related Issues (20)
- @timed macro reports 1 microsecond
- Dialyzer reports "Function has no local return" HOT 4
- Production environment causing naming inconsistency HOT 2
- README should explain why I'd want to use this over plain-ol' exometer HOT 1
- All types are gauges HOT 4
- While runing "mix ecto.create" or "mix phoenix.server" getting below error HOT 1
- Elixometer.clear_counter/1 is supposed to use configured Formatter HOT 2
- Allow :pobox 1.x minor revision upgrade HOT 2
- [Question] Why is there two different function names in `DeclarativeTest` module for testing `@timed` with a guarded function? HOT 1
- i have some issue while runing the phoenix server HOT 2
- Timeout on registering a metric HOT 3
- Lager is not really used, but it is in dependencies HOT 1
- CRASH REPORT Process exometer_report_tty with 0 neighbours crashed HOT 4
- Unable to install dependencies when adding exometer to deps HOT 7
- Release version 1.3.0 HOT 5
- Get metric value by wildcard key HOT 1
- Question: is there any way to set tags? HOT 6
- "Enumerable not implemented for nil" on subscription creation HOT 6
- Could not compile dependency :setup, "/Users/alan/.mix/rebar3 bare compile --paths HOT 11
- @timed functions return [do: result] HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elixometer.