dalmatinerdb / ddb_client Goto Github PK

License: MIT License

Makefile 6.35% Erlang 93.65%

ddb_client's Introduction

Read more at the official site and the documentation.

DalmatinerDB

DalmatinerDB is a metric database written in pure Erlang. It takes advantage of some special properties of metrics to make some tradeoffs. The goal is to make a store for metric data (time, value of a metric) that is fast, has a low overhead, and is easy to query and manage.

Tradeoffs

I try here to be explicit about the tradeoffs we made, so people can decide if they are happy with them (costs vs gains). The acceptable tradeoffs differ from case to case, but I hope the choices we made fit a metric store quite well. If you are comparing DalmatinerDB with X, please don't assume that just because X does not list the tradeoffs they made, they have none; be inquisitive and make a decision based on facts, not marketing.

A detailed comparison between databases can be found here:

https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/edit#gid=0

Let the Filesystem handle it

A lot of work is handed down to the file system, ZFS is exceptionally smart and can do things like checksums, compressions and caching very well. Handing down these tasks to the filesystem simplifies the codebase, and builds on very well tested and highly performant code, instead of trying to reimplement it.

Prioritise the overall writes over individual ones

DalmatinerDB offers a 'best effort' on storing the metrics, there is no log for writes (if enabled in ZFS, the ZIL (ZFS Intent Log) can log write operations) or forced sync after each write. This means that if your network fails, packets can get lost, and if your server crashes, unwritten data can be lost.

The point is that losing one or two metric points in a huge series is a non-problem, the importance of a metric is often seen in aggregates, and DalmatinerDB fills in the blanks with the last written value. However there is explicitly no guarantee that data is written, so this can be an issue if every single point of metric is of importance!

Flat files

Data is stored in a flat binary format, this means that reads and writes can be calculated to a filename+offset by simple math, there is no need for traversing data-structures. This means however that if a metric stops unwritten, points can 'linger' around for a while depending on how the file size was picked.

As an example: if metrics are stored with a precision down to the second, and 1 week of data is stored per file, up to one week of unused data can be stored, but it should be taken into account that with compression this data will be compressed quite well.

ddb_client's People

Contributors

Stargazers

Watchers

Forkers

j-tomahawk davecromberge erszcz jacktang weaver-viii ltamas97

ddb_client's Issues

compile fails

(I'm new to erlang and rebar)

in the README.md there are information how to build the client using rebar3
but there is no rebar3 app as is in other repos.

make fails with rebar installed on my debian jessie box.

/Download/ddb_client$ make
rebar get-deps
==> ddb_client (get-deps)
rebar compile
==> ddb_client (compile)
Dependency not available: dproto->0.2.0 (undefined)
Dependency not available: mmath->0.2.0 (undefined)
Dependency not available: snappy->1.1.1 (undefined)
Makefile:9: recipe for target 'compile' failed
make: *** [compile] Error 1

Best regards, Marinó

How do I set dimension while pushing metrics in batch mode

It is not clear from the documentation, how dimensions are associated while sending out metrics in batches.

wrong socket usage

https://github.com/dalmatinerdb/ddb_client/blob/master/src/ddb_tcp.erl#L452
Socket in the line above should be from Con1, not Con.

Unable to send metrics to ddb

I can only imagine this being an error on my side, but I am unable to get any metrics in DalmatinerDB using this client (or any other). Below are the files I am using in my setup.

docker-compose.yml

version: '3' 
services:
  dfe:
    image: projectfifo/dfe
    depends_on:
      - ddb
      - postgres
    ports:
      - "8080:8080"
    environment:
      DB_NODE: ddb
      PG_NODE: postgres
    volumes:
      - dfe-data:/data/dalmatinerfe/db
  ddb:
    image: projectfifo/ddb
    volumes:
      - ddb-data:/data/dalmatinerdb/db
    ports:
      - "5555:5555"
  postgres:
    image: projectfifo/metric_metadata
    volumes:
      - postgres-data:/var/lib/postgresql/data
  erlang:
    image: erlang:19.2.3
    working_dir: /opt/project
    volumes:
      - .:/opt/project
    depends_on:
      - ddb

volumes:
  ddb-data:
  dfe-data:
  postgres-data:

send-data.erl

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -pa _build/default/lib/ddb_client/ebin _build/default/lib/dproto/ebin _build/default/lib/mmath/ebin _build/default/lib/snappiest/ebin

main(_) ->
  {ok, Con} = ddb_tcp:connect("ddb", 5555),
  Bucket = <<"my_bucket1">>,
  Delay = 1,
  {ok, SCon} = ddb_tcp:stream_mode(Bucket, Delay, Con),

  Timestamp = 1466072419,
  Metric = dproto:metric_from_list([<<"abc">>, <<"def">>]),
  Value  = mmath_bin:from_list([10.0, 20.5, 30.9]),
  {ok, SCon1} = ddb_tcp:send(Metric, Timestamp, Value, SCon),

  ddb_tcp:close(SCon1).

get-data.erl

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -pa _build/default/lib/ddb_client/ebin _build/default/lib/dproto/ebin _build/default/lib/mmath/ebin _build/default/lib/snappiest/ebin

main(_) ->
  {ok, Con} = ddb_tcp:connect("ddb", 5555),

  Timestamp = 1466072419,
  Bucket = <<"my_bucket">>,
  Metric = dproto:metric_from_list([<<"abc">>, <<"def">>]),
  Count = 3,
  Opts = [{rr, off}, {r, 1}],
  {ok, Data, Con1} = ddb_tcp:get(Bucket, Metric, Timestamp, Count, Opts, Con),

  io:format(Data),

  ddb_tcp:close(Con1).

The get-data.erl script produces no output (in the Data var), nor is the metric visible in dfe. The ddb console output after executing get-data.erl:

[warning] <0.898.0>@metric_io:do_read:739 [IO] Unknown metric: <<"my_bucket">>/<<3,97,98,99,3,100,101,102>>

I've also tried the node and python clients, so not sure if this is the right place to create this issue, as that would point to this being related to something like the docker setup.

dalmatinerdb / ddb_client Goto Github PK

ddb_client's Introduction

DalmatinerDB

Tradeoffs

Let the Filesystem handle it

Prioritise the overall writes over individual ones

Flat files

ddb_client's People

Contributors

Stargazers

Watchers

Forkers

ddb_client's Issues

compile fails

How do I set dimension while pushing metrics in batch mode

wrong socket usage

Unable to send metrics to ddb

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent