life-research / datomic-tx-metrics Goto Github PK

We

...should use base units (e.g. seconds, bytes, meters - not milliseconds, megabytes, kilometers). See below for a list of base units.
https://prometheus.io/docs/practices/naming/

Instead of using multiple alarm metric just use one and make use of labels to enable Prometheus' multidimensional model.

As of version 0.2.0 there are other metrics that are not emitted using a base unit although they should.
The following metrics are affected:

• available-ram-megabytes
• memory-index-consumed-megabytes

This is related to #10

The build artifact of this project is expected to be placed within the lib directory of the datomic transactor. A first step to achieve this is to offer a pre-built JAR, leveraging Travis CI to do so.

Additionally, this would make it fairly easy to integrate it within a docker container.

Besides JVM metrics replacing an additional JMX exporter, metrics related to Datomic's transactor also have to be added.

Have a look below to see an example output of metrics handed over to the registered callback handler by the transactor:

{
:TransactionDatoms {:lo 43, :hi 18734, :sum 526412, :count 61},
:GarbageSegments {:lo 1, :hi 116, :sum 406, :count 122},
:ObjectCacheCount 4486,
:CreateEntireIndexMsec {:lo 12600, :hi 13400, :sum 26000, :count 2},
:MemoryIndexMB {:lo 39, :hi 39, :sum 39, :count 1},
:StoragePutMsec {:lo 1, :hi 129, :sum 3529, :count 940},
:TransactionBatch {:lo 1, :hi 1, :sum 61, :count 61},
:TransactionBytes {:lo 755, :hi 335483, :sum 9254428, :count 61},
:AvailableMB 2060.0,
:IndexWriteMsec {:lo 1, :hi 130, :sum 10963, :count 760},
:FulltextSegments {:lo 12, :hi 12, :sum 24, :count 2},
:IndexWrites {:lo 370, :hi 390, :sum 760, :count 2},
:RemotePeers {:lo 2, :hi 2, :sum 2, :count 1},
:HeartbeatMsec {:lo 5000, :hi 5070, :sum 60110, :count 12},
:StorageGetMsec {:lo 0, :hi 63, :sum 436, :count 129},
:IndexDatoms {:lo 18430302, :hi 19215353, :sum 37645655, :count 2},
:TransactionMsec {:lo 1350, :hi 6680, :sum 229690, :count 61},
:StoragePutBytes {:lo 84, :hi 91533, :sum 17253404, :count 940},
:LogWriteMsec {:lo 8, :hi 103, :sum 2978, :count 61},
:ObjectCache {:lo 0, :hi 1, :sum 42797, :count 42926},
:MetricsReport {:lo 1, :hi 1, :sum 1, :count 1},
:MemoryIndexFillMsec {:lo 505589, :hi 505589, :sum 505589, :count 1},
:PodUpdateMsec {:lo 5, :hi 82, :sum 754, :count 61},
:DbAddFulltextMsec {:lo 0, :hi 3, :sum 67, :count 61},
:PodGetMsec {:lo 2, :hi 2, :sum 4, :count 2},
:StorageGetBytes {:lo 84, :hi 25527, :sum 383696, :count 129},
:IndexSegments {:lo 3532, :hi 3677, :sum 7209, :count 2},
:Datoms {:lo 6812456, :hi 7104076, :sum 13916532, :count 2},
:CreateFulltextIndexMsec {:lo 0, :hi 0, :sum 0, :count 2}
}

Descriptions of what these metrics describe are summarized on Datomic's official documentation site.
Watch out that some descriptions are hidden within the change log section of the documentation, while others are not mentioned there at all.

The metrics overview for both JVM metrics as well as Datomic transactor metrics do not provide a clear image at first glance of what is actually going to be emitted as a metric. This is especially true for Datomic metrics since we are converting units.

This should be improved using an additional tabular layout.

As of version 0.1.0 metrics being emitted as gauges are not always reset although they should be. Potential metrics affected by this are timing gauges. Since they are only set if the metric is present in the metrics blob handed over by the transactor, wrong values could be emitted.

Example:
The transactor emits metrics containing 'StoragePutMsec'. Now, if the transactor does not emit this metric in any subsequent metric reports (due to the fact that nothing is written to storage) the gauge will stay at the last reported value and this does not necessarily resemble reality.

Thus, gauges that may be affected by this issue should be cleared if the associated transactor metric is missing from the metric report.