Code Monkey home page Code Monkey logo

dsc-datatool's Introduction

DSC DataTool

Total alerts Bugs Security Rating

Tool for converting, exporting, merging and transforming DSC data.

Please have a look at the man-page(s) dsc-datatool (1) on how to use or the wiki article on how to set this up using Influx DB and Grafana.

More information about DSC may be found here:

Issues should be reported here:

General support and discussion:

Dependencies

dsc-datatool requires the following Python libraries:

  • PyYAML
  • maxminddb

Python Development Environment

Using Ubuntu/Debian:

sudo apt-get install python3-maxminddb python3-yaml python3-venv
python3 -m venv venv --system-site-packages
. venv/bin/activate
pip install -e . --no-deps

dsc-datatool's People

Contributors

jelu avatar lgtm-migrator avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

dsc-datatool's Issues

dsc-datatool chokes on '...' tld

Occasionally, someone manages to send a query to a NS containing '...' as the zone being queries.

This ends up as the following in the xml files:

<array name="qtype_vs_tld" dimensions="2" start_time="1562012880" stop_time="1562012940">
   <dimension number="1" type="Qtype"/>
   <dimension number="2" type="TLD"/>
   <data>
     <Qtype val="1" count="374">
       <TLD val="..." count="15"/>
       <TLD val="africa." count="10"/>
       <TLD val="co.za." count="332"/>
       <TLD val="za." count="17"/>
     </Qtype>

dsc-datatool dies when encountering this :(

Similar to my previous issue reported in April?

Datatool wants to connect to 192.0.47.8 => www.iana.org

The dsc-datatool want to connect to 192.0.47.8 which points to `www.iana.org'.
The machine that needs to convert the data has absolutely no direct connection to the internet and will never have so.
How do I stop the datatool connecting to Icann?

XML parsing error

  • capture XML parsing errors and display an well-formed error message (with file path) instead of an stack trace

test-site dashboard query qps issue

when I visit the demo site(https://dev.dns-oarc.net/dsc-grafana2/dashboard/db/dsc), i found the qps issue.
eg. try
set Time Resolution to 1m, then check the bynode qps, got avg qps: 14pps
set Time Resolution to 5m, t got avg qps: 66pps
set Time Resolution to 10m, t got avg qps: 133pps
set Time Resolution to 1h, t got avg qps: 598pps

is the avg qps right or wrong?

but from the bynode graph, the avg qps in 3hours is less then 20pps
IMHO, in different Time Resolution, the avg qps should be in the same range.

IANA Assignments file changed causing exception

IANA has updated this file: https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.csv

In April, they added "Segment Routing (SRv6) SIDs" with a "status" of "ALLOCATED" but no "whois" which is causing an Exception in client_subnet_authority.py on line 68.

This file needs adjusted to handle this scenario. Anyone using the dsc-datatool with client_subnet_authority is broken at the moment due to this issue.

DSC to influxDB failing

On the collector side, when collection is done for:
dataset second_ld_vs_rcode dns Rcode:rcode SecondLD:second_ld replies-only max-cells=50;

Sometimes queries are received for null names:

<array name="second_ld_vs_rcode" dimensions="2" start_time="1556173500" stop_time="1556173560">
  <dimension number="1" type="Rcode"/>
  <dimension number="2" type="SecondLD"/>
  <data>
    <Rcode val="0">
      <SecondLD val="org.za" count="2"/>
      <SecondLD val="" count="2"/>
      <SecondLD val="amazonaws.com" count="1"/

which ends up after DSC-datatool as

second_ld_vs_rcode,server=servername,node=ns.coza.net.za,rcode=5,secondld= value=2 1556174460000000000

which chokes when fed into 'influx -import' because of the missing value @ "secondld="

dsc-datatool needs to add a null character when encountered, to prevent the influxdb import from breaking.

Logging

Add proper logging and honor --verbose

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.