Code Monkey home page Code Monkey logo

netdata-timescale-relay's People

Contributors

mahlonsmith avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

netdata-timescale-relay's Issues

Netdata-TSRelay broken?

Hi @mahlonsmith

i've just come to try this tool, however i cannot get it to work.

I'm using ArchLinux x64, which currently features nim v1.2.0. At the time of the latest release, nim v0.19.0 was the most recent.

Compilation gives some warnings but it builds fine:

[mpkg@zink netdata-timescale-relay]$ make
nim -d:release --opt:speed --parallelBuild:0 --nimcache:.cache c netdata_tsrelay.nim
Hint: used config file '/etc/nim/nim.cfg' [Conf]
Hint: used config file '/etc/nim/config.nims' [Conf]
Hint: system [Processing]
Hint: widestrs [Processing]
Hint: io [Processing]
Hint: netdata_tsrelay [Processing]
Hint: db_postgres [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: math [Processing]
Hint: bitops [Processing]
Hint: macros [Processing]
Hint: algorithm [Processing]
Hint: unicode [Processing]
Hint: postgres [Processing]
Hint: db_common [Processing]
Hint: json [Processing]
Hint: hashes [Processing]
Hint: tables [Processing]
Hint: lexbase [Processing]
Hint: streams [Processing]
Hint: parsejson [Processing]
Hint: options [Processing]
Hint: typetraits [Processing]
Hint: nativesockets [Processing]
Hint: os [Processing]
Hint: pathnorm [Processing]
Hint: osseps [Processing]
Hint: posix [Processing]
Hint: times [Processing]
Hint: net [Processing]
Hint: sets [Processing]
Hint: monotimes [Processing]
Hint: ssl_certs [Processing]
Hint: ospaths [Processing]
Hint: parseopt [Processing]
Hint: terminal [Processing]
Hint: strformat [Processing]
Hint: colors [Processing]
Hint: termios [Processing]
/home/mpkg/netdata-timescale-relay/netdata_tsrelay.nim(201, 27) Warning: use strformat module instead; round is deprecated [Deprecated]
/home/mpkg/netdata-timescale-relay/netdata_tsrelay.nim(37, 5) Warning: imported and not used: 'os' [UnusedImport]
CC: stdlib_assertions.nim
CC: stdlib_dollars.nim
CC: stdlib_formatfloat.nim
CC: stdlib_io.nim
CC: stdlib_system.nim
CC: stdlib_parseutils.nim
CC: stdlib_math.nim
CC: stdlib_unicode.nim
CC: stdlib_strutils.nim
CC: stdlib_postgres.nim
CC: stdlib_db_common.nim
CC: stdlib_db_postgres.nim
CC: stdlib_hashes.nim
CC: stdlib_tables.nim
CC: stdlib_streams.nim
CC: stdlib_lexbase.nim
CC: stdlib_parsejson.nim
CC: stdlib_options.nim
CC: stdlib_json.nim
CC: stdlib_times.nim
CC: stdlib_os.nim
CC: stdlib_nativesockets.nim
CC: stdlib_monotimes.nim
CC: stdlib_net.nim
CC: stdlib_parseopt.nim
CC: stdlib_terminal.nim
CC: netdata_tsrelay.nim
Hint:  [Link]
Hint: 71950 LOC; 2.961 sec; 73.398MiB peakmem; Release build; proj: /home/mpkg/netdata-timescale-relay/netdata_tsrelay.nim; out: /home/mpkg/netdata-timescale-relay/netdata_tsrelay [SuccessX]

However when setup, i'm not getting my timescaledb populated. The relay logs the following:

sudo -u netdata /usr/local/bin/netdata_tsrelay -d --listen-addr=127.0.0.1                                                      ✔  0.008s  /etc/netdata  23:35:35
(dbopts: "host=localhost dbname=netdata application_name=netdata-tsrelay", dbtable: "netdata", listen_port: 14866, listen_addr: "127.0.0.1", verbose: true, debug: true, insertsql: "    INSERT INTO netdata\n        ( time, host, metrics )\n    VALUES\n        ( \'epoch\'::timestamptz + ? * \'1 second\'::interval, ?, ? )\n    ", timeout: 500)
Successfully tested connection to the backend database.
Listening for incoming connections on 127.0.0.1:14866

1594503365 0 sample(s) parsed from 127.0.0.1 in 0.0 seconds.
1594503425 0 sample(s) parsed from 127.0.0.1 in 0.0 seconds.
1594503485 0 sample(s) parsed from 127.0.0.1 in 0.0 seconds.
1594503545 0 sample(s) parsed from 127.0.0.1 in 0.0 seconds.
1594503605 0 sample(s) parsed from 127.0.0.1 in 0.0 seconds.
1594503665 0 sample(s) parsed from 127.0.0.1 in 0.0 seconds.
1594503725 0 sample(s) parsed from 127.0.0.1 in 0.0 seconds.
1594503785 0 sample(s) parsed from 127.0.0.1 in 0.0 seconds.
1594503845 0 sample(s) parsed from 127.0.0.1 in 0.0 seconds.
...

Depending on how much metrics i send, i'm getting different error messages in netdata's error.log:

"Many metrics":

netdata ERROR : MAIN : EXPORTING: failed to write data to '127.0.0.1:14866'. Willing to write 685507 bytes, wrote 524288 bytes. Will re-connect.

"Little metrics":

netdata ERROR : MAIN : EXPORTING: '127.0.0.1:14866' closed the socket

My exporting.conf (netdata v1.23.1) contains the following:

[exporting:global]
    enabled = yes

[json:timescaledb]
    hostname           = zink
    enabled            = yes
    type               = json
    data source        = average
    destination        = 127.0.0.1:14866
    prefix             = netdata
    update every       = 60
    buffer on failures = 10
    send charts matching = *
    send hosts matching = *

Any suggestions appreciated!

What is the max data size which can be processed by netdata-timescale-relay?

Here is my problem.
When I export metrics from metrics at every 1s (as collected) from parent node and there is one child node, I get error from netdata like
2021-01-18 17:20:31: netdata ERROR : MAIN : EXPORTING: failed to write data to 'localhost:14866'. Willing to write 2092079 bytes, wrote 1766662 bytes. Will re-connect. (errno 11, Resource temporarily unavailable).

When I add filter send charts matching = !cpu.cpu* !ipv6* !users.* nfs.rpc net.* net_drops.* net_packets.* !system.interrupts* system.* disk.* disk_space.* disk_ops.* mem.*, it works well.

So I guess the error is raised by data size. There is too much data exporting to timescaledb through netdata-timescale-relay.

Am I right?

If yes, what is the max data flow that netdata-timescale-relay can handle?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.