Code Monkey home page Code Monkey logo

jtimon's Introduction

Build Status

jtimon

Junos Telemetry Interface client

Setup

$ git clone https://github.com/nileshsimaria/jtimon.git
$ cd jtimon
$ make linux or make darwin
$ ./jtimon-linux-amd64 --help or jtimon-darwin-amd64 --help

Please note that if you use make to build source, it will produce binary with GOOS and GOARCH names e.g. jtimon-darwin-amd64, jtimon-linux-amd64 etc. Building the source using make is recommended as it will insert git-revision, build-time info in the binary.

To understand what targets are available in make, run the make help command as follows:

$ make help

Note

If you are cloning the source and building it, please make sure you have environment variable GOPATH is set correctly. https://golang.org/doc/code.html#GOPATH

Docker container

Alternatively to building jtimon native, one can build a jtimon Docker container and run it dockerized while passing the local directory to the container to access the json file.

To build the container:

make docker

Check the resulting image:

$ docker images jtimon
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
jtimon              latest              f1a794609339        7 minutes ago       24.5MB

Run it:

docker run -ti --rm -v ${PWD}:/u:ro jtimon --help

Or simply by calling ./jtimon, which is a symlink to launch-docker-container.sh, capable of launching the container by name with the current directory mounted into /u:

$ ./jtimon
Enter config file name: bla.json
2018/03/02 13:53:44 File error: open bla.json: no such file or directory

CLI Options

$ ./jtimon-linux-amd64 --help
Usage of ./jtimon-linux-amd64:
      --compression string         Enable HTTP/2 compression (gzip)
      --config strings             Config file name(s)
      --config-file-list string    List of Config files
      --consume-test-data          Consume test data
      --explore-config             Explore full config of JTIMON and exit
      --generate-test-data         Generate test data
      --json                       Convert telemetry packet into JSON
      --log-mux-stdout             All logs to stdout
      --max-run int                Max run time in seconds
      --no-per-packet-goroutines   Spawn per packet go routines
      --pprof                      Profile JTIMON
      --pprof-port int32           Profile port (default 6060)
      --prefix-check               Report missing __prefix__ in telemetry packet
      --print                      Print Telemetry data
      --prometheus                 Stats for prometheus monitoring system
      --prometheus-host string     IP to bind Prometheus service to (default "127.0.0.1")
      --prometheus-port int32      Prometheus port (default 8090)
      --stats-handler              Use GRPC statshandler
      --version                    Print version and build-time of the binary and exit
pflag: help requested

Config

To explore what can go in config, please use --explore-config option.

Except connection details like host, port, etc no other part of the config is mandatory e.g. do not use influx in your config if you dont want to insert data into it.

$ ./jtimon-linux-amd64 --explore-config                                                                                                                                   [8/1981]
2021/10/02 17:15:22 Version: v2.3.0-7bfd8fdf2fcae1d55079e3d9eceb761be0842eae-master BuildTime 2021-10-02T19:51:16-0400
2021/10/02 17:15:22
{
    "port": 0,
    "host": "",
    "user": "",
    "password": "",
    "cid": "",
    "meta": false,
    "eos": false,
    "grpc": {
        "ws": 0
    },
    "tls": {
        "clientcrt": "",
        "clientkey": "",
        "ca": "",
        "servername": ""
    },
    "influx": {
        "server": "",
        "port": 0,
        "dbname": "",
        "user": "",
        "password": "",
        "recreate": false,
        "measurement": "",
        "batchsize": 0,
        "batchfrequency": 0,
        "http-timeout": 0,
        "retention-policy": "",
        "accumulator-frequency": 0,
        "write-per-measurement": false
    },
    "kafka": null,
    "paths": [
        {
            "path": "",
            "freq": 0,
            "mode": ""
        }
    ],
    "log": {
        "file": "",
        "periodic-stats": 0,
        "verbose": false
    },
    "vendor": {
        "name": "",
        "remove-namespace": false,
        "schema": null,
        "gnmi": null
    },
    "alias": "",
    "password-decoder": "",
    "enable-uint": false
}    

I am explaining some config options which are not self-explanatory.

meta : send username and password over gRPC meta instead of invoking LoginCheck() RPC for authentication. 
Please use SSL/TLS for security. For more details on how to use SSL/TLS, please refer wiki
https://github.com/nileshsimaria/jtimon/wiki/SSL
cid : client id. Junos expects unique client ids if multiple clients are subscribing to telemetry streams.
eos : end of sync. Tell Junos to send end of sync for on-change subscriptions.
grpc/ws : window size of grpc for slower clients

Kafka Publish

To publish gRPC/Openconfig JTI data to Kafka, use the following json config.

$ cat kafka-test-1.json
{
    "host": "2.2.2.2",
    "port": 32767,
    "user": "username",
    "password": "password",
    "cid": "cid123",
    "kafka": {
        "brokers": ["1.1.1.1:9094"],
        "topic": "test",
        "client-id": "testjtimonmx86"
    },

    "paths": [
        {
            "path": "/interfaces",
            "freq": 10000
        }
    ]
}

Below are all possible Kafka config options.

type KafkaConfig struct {
	Version            string   `json:"version"`
	Brokers            []string `json:"brokers"`
	ClientID           string   `json:"client-id"`
	Topic              string   `json:"topic"`
	CompressionCodec   int      `json:"compression-codec"`
	RequiredAcks       int      `json:"required-acks"`
	MaxRetry           int      `json:"max-retry"`
	MaxMessageBytes    int      `json:"max-message-bytes"`
	SASLUser           string   `json:"sasl-username"`
	SASLPass           string   `json:"sasl-password"`
	TLSCA              string   `json:"tls-ca"`
	TLSCert            string   `json:"tls-cert"`
	TLSKey             string   `json:"tls-key"`
	InsecureSkipVerify bool     `json:"insecure-skip-verify"`
}

jtimon's People

Contributors

akshaygogeri avatar alrighttheresham avatar balasankarrajaguru avatar m-vivek avatar mwiget avatar nileshsimaria avatar nsimariaj avatar philodavies avatar weijiage avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jtimon's Issues

Running with prometheus parameter

If we run jtimon with the parameter --prometheus we get the following error:

root@opennti:~# gocode/bin/jtimon --config juniper.json --prometheus
Version: version-not-available BuildTime build-time-not-available
listen tcp: invalid port 842350541320

Also with prometheus-port:

root@opennti:~# gocode/bin/jtimon --config juniper.json --prometheus --prometheus-port 8090
Version: version-not-available BuildTime build-time-not-available
listen tcp: invalid port 842350541264

We see that the program is not listening on port 8090
If we run it without the prometheus parameter we see no port error:

root@opennti:~# gocode/bin/jtimon --config juniper.json
Version: version-not-available BuildTime build-time-not-available

Influx 2.0

Just wondering what the plan is regarding Influx2.0? Is it planned? Is there a pathway to it?

build issue

Hello

i can not succeed to build the latest jtimon image. I faced this issue :

root@muztc871:/home/david/telemetry/jtimon# go version
go version go1.10.3 linux/amd64

root@muztc871:/home/david/telemetry/jtimon# go build

_/home/david/telemetry/jtimon

./main.go:205:37: cannot use conn (type *"google.golang.org/grpc".ClientConn) as type *"github.com/nileshsimaria/jtimon/vendor/google.golang.org/grpc".ClientConn in argument to authentication.NewLoginClient
./subscribe.go:101:41: cannot use conn (type *"google.golang.org/grpc".ClientConn) as type *"github.com/nileshsimaria/jtimon/vendor/google.golang.org/grpc".ClientConn in argument to telemetry.NewOpenConfigTelemetryClient

Could you help me to fix this ? I need latest version of jtimon to have gnmi options.

BR
David

[Feature Request] convert string/enum values for prometheus exporter

We are currently using jtimon to monitor some juniper routers - with some caveats this is working fairly well.

For sensors that expose strings or enums, the exporter will omit them (see:

case *na_pb.KeyValue_StrValue:
floatVal, err := strconv.ParseFloat(v.GetStrValue(), 64)
if err != nil {
continue
}
fieldValue = floatVal
)

In certain cases, it would be great to be able to convert these values and expose them. For example the bgp session-state sensor at /network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/state/session-state exposes a ENUM (see screenshot below from explorer)

image

It would be great to expose those as metrics.

From the tip of my head, I see two viable options:

  1. Provide a mapping for certain paths to ENUM values, a approach was done at #46

  2. Move the value of that metric into a label and expose it with a constant value - POC here patrickjahns@039a7af

It would be great to receive quick feedback which the preferred option would be - I am happy to provide a pull request for this

ca-profile names are not matching

These 2 CLI commands should use the same ca-profile names

set security pki ca-profile ca1 ca-identity caid1
run request security pki ca-certificate load ca-profile caprof1 filename /var/tmp/RootCA.crt

Dialout Changes Broke Dockerfile

Not really an important issue but I thought I'd just let you know:

The changes to add in Dialout termination support (and subsequent clean up) have broken the Dockerfile

COPY failed: file not found in build context or excluded by .dockerignore: stat certs/self_signed/: file does not exist
make: *** [Makefile:33: docker] Error 1

As far as I understand it, the certs are only needed when acting as a dialout terminator (and when not skipping verification)? So maybe a quick fix would be just to have (pseudo)empty certs/self_signed directory in the repo? Maybe also with some extra documentation in the readme for when dialout is enabled that these certs need to be present.

compilation issue

To accomodate new client I did:
go get github.com/influxdata/influxdb1-client/v2 \

then, declaration section of your Go file, paste the following: import "github.com/influxdata/influxdb1-client/v2".

I am getting this error once trying to compile:

`tkaagent@ubuntu1:~/jtimon$ sudo go build

_/home/tkaagent/jtimon

./subscribe_cisco_iosxr.go:198:33: cannot use conn (type *"google.golang.org/grpc".ClientConn) as type *"github.com/nileshsimaria/jtimon/vendor/google.golang.org/grpc".ClientConn in argument to IOSXRExtensibleManagabilityService.NewGRPCConfigOperClient
./subscribe_juniper_junos.go:105:41: cannot use conn (type *"google.golang.org/grpc".ClientConn) as type *"github.com/nileshsimaria/jtimon/vendor/google.golang.org/grpc".ClientConn in argument to "github.com/nileshsimaria/jtimon/telemetry".NewOpenConfigTelemetryClient
./subscribe_juniper_junos.go:228:32: cannot use conn (type *"google.golang.org/grpc".ClientConn) as type *"github.com/nileshsimaria/jtimon/vendor/google.golang.org/grpc".ClientConn in argument to authentication.NewLoginClient
tkaagent@ubuntu1:~/jtimon$`

scalar array not displayed properly

please update the function handleOnePacket to display scalar array datatype as well.

also decimal64 datatype is being added (20.4 onwards), so please pick the latest telemetry.proto and add support for displaying it as well something like below:
case *na_pb.KeyValue_DecimalValue:
s += fmt.Sprintf(" decimal64_value: digits=%d precision=%d\n", value.DecimalValue.Digits, value.DecimalValue.Precision)

lint issues

Hi Nilesh, the following issues have been identified by gometalinter

config.go:103:1:warning: ValidateConfig is unused (deadcode)
errs.go:18:1:warning: NewErrs is unused (deadcode)
errs.go:39:1:warning: AppendErrs is unused (deadcode)
influx.go:46:1:warning: timeDiff is unused (deadcode)
main.go:130:21:warning: ineffectual assignment to err (ineffassign)
influx.go:200::warning: cyclomatic complexity 33 of function addIDB() is high (> 10) (gocyclo)
subscribe.go:17::warning: cyclomatic complexity 24 of function handleOnePacket() is high (> 10) (gocyclo)
main.go:77::warning: cyclomatic complexity 24 of function worker() is high (> 10) (gocyclo)
subscribe.go:81::warning: cyclomatic complexity 22 of function subSendAndReceive() is high (> 10) (gocyclo)
statshandler.go:46::warning: cyclomatic complexity 16 of function (*statshandler).HandleRPC() is high (> 10) (gocyclo)
main.go:223::warning: cyclomatic complexity 16 of function main() is high (> 10) (gocyclo)
gtrace.go:20::warning: Binds to all network interfaces,MEDIUM,HIGH (gas)
logs.go:36::warning: Expect file permissions to be 0600 or less,MEDIUM,HIGH (gas)
influx.go:47:2:warning: unused struct field github.com/nileshsimaria/jtimon.timeDiff.field (structcheck)
influx.go:48:2:warning: unused struct field github.com/nileshsimaria/jtimon.timeDiff.tags (structcheck)
influx.go:252:26:warning: unnecessary conversion (unconvert)
main.go:166:44:warning: unnecessary conversion (unconvert)
config.go:12:13:warning: struct of size 376 could be 368 (maligned)
config.go:29:16:warning: struct of size 56 could be 48 (maligned)
influx.go:32:19:warning: struct of size 136 could be 128 (maligned)
drops.go:30:8:warning: error return value not checked (f.Seek(0, 0)) (errcheck)
logs.go:12:14:warning: printf-style function with dynamic first argument and no further arguments should use print-style function instead (SA1006) (megacheck)
main.go:130:21:warning: this value of err is never used (SA4006) (megacheck)
main.go:167:28:warning: grpc.WithDecompressor is deprecated: use encoding.RegisterCompressor instead.  (SA1019) (megacheck)
config.go:127:2:warning: should use 'return <expr>' instead of 'if <expr> { return <bool> }; return <bool>' (S1008) (megacheck)
drops.go:54:5:warning: should omit comparison to bool constant, can be simplified to !ok (S1002) (megacheck)
drops.go:59:5:warning: should omit comparison to bool constant, can be simplified to !ok (S1002) (megacheck)
drops.go:64:5:warning: should omit comparison to bool constant, can be simplified to !ok (S1002) (megacheck)
gtrace.go:13:5:warning: should omit comparison to bool constant, can be simplified to gtrace (S1002) (megacheck)
influx.go:207:34:warning: should use make([]*client.Point, 0) instead (S1019) (megacheck)
influx.go:250:5:warning: redundant break statement (S1023) (megacheck)
influx.go:253:5:warning: redundant break statement (S1023) (megacheck)
influx.go:256:5:warning: redundant break statement (S1023) (megacheck)
influx.go:259:5:warning: redundant break statement (S1023) (megacheck)
influx.go:262:5:warning: redundant break statement (S1023) (megacheck)
influx.go:265:5:warning: redundant break statement (S1023) (megacheck)
influx.go:268:5:warning: redundant break statement (S1023) (megacheck)
main.go:33:61:warning: should use make([]string, 0) instead (S1019) (megacheck)
main.go:117:3:warning: should use for range instead of for { select {} } (S1000) (megacheck)
main.go:194:11:warning: should omit comparison to bool constant, can be simplified to !jctx.config.Meta (S1002) (megacheck)
main.go:201:12:warning: should omit comparison to bool constant, can be simplified to !dat.Result (S1002) (megacheck)
main.go:265:30:warning: should use make([]*workerCtx, n) instead (S1019) (megacheck)
statshandler.go:155:10:warning: the argument is already a string, there's no need to use fmt.Sprintf (S1025) (megacheck)
statshandler.go:156:10:warning: the argument is already a string, there's no need to use fmt.Sprintf (S1025) (megacheck)
statshandler.go:157:10:warning: the argument is already a string, there's no need to use fmt.Sprintf (S1025) (megacheck)
statshandler.go:159:10:warning: the argument is already a string, there's no need to use fmt.Sprintf (S1025) (megacheck)
statshandler.go:160:10:warning: the argument is already a string, there's no need to use fmt.Sprintf (S1025) (megacheck)
statshandler.go:161:10:warning: the argument is already a string, there's no need to use fmt.Sprintf (S1025) (megacheck)
statshandler.go:196:5:warning: should omit comparison to bool constant, can be simplified to jctx.config.Log.DropCheck (S1002) (megacheck)
subscribe.go:84:5:warning: should omit comparison to bool constant, can be simplified to jctx.config.Meta (S1002) (megacheck)
errs.go:4:6:warning: type Errors is unused (U1000) (megacheck)
errs.go:18:6:warning: func NewErrs is unused (U1000) (megacheck)
errs.go:27:6:warning: func AppendErr is unused (U1000) (megacheck)
errs.go:39:6:warning: func AppendErrs is unused (U1000) (megacheck)
errs.go:54:6:warning: func ToString is unused (U1000) (megacheck)
influx.go:46:6:warning: type timeDiff is unused (U1000) (megacheck)

Nilesh you should consider hooking up to travis ci (community edition) and have a build execute with the test and lint phases as part of the pull request process.

If you want I can create the config, let me know.

When running jtimon-linux-x86_64 without cvs-stats & logs option then its running double the time of max-run.

root@jvision-ubuntu3:/opt/go-collector/RLI-35537# ./jtimon-linux-x86_64 --config Brackla75_PFE_39048_GRPC.json --stats 2 --max-run 60
gRPC headers from Junos:
init-response: [response { subscription_id: 1 } path_list { path: "/interfaces/" sample_frequency: 2000 } ]
grpc-accept-encoding: [identity,deflate,gzip]
+------------------------------+--------------------+--------------------+--------------------+--------------------+
| Timestamp | KV | Packets | Bytes | Bytes(wire) |
+------------------------------+--------------------+--------------------+--------------------+--------------------+
| Wed Mar 6 01:43:38 PST 2019 | 4262 | 11 | 141632 | 141632 |
| Wed Mar 6 01:43:40 PST 2019 | 8524 | 22 | 283272 | 283272 |
| Wed Mar 6 01:43:42 PST 2019 | 12786 | 33 | 424915 | 424915 |
| Wed Mar 6 01:43:44 PST 2019 | 17048 | 44 | 566559 | 566559 |
| Wed Mar 6 01:43:46 PST 2019 | 21310 | 55 | 708203 | 708203 |
| Wed Mar 6 01:43:48 PST 2019 | 25572 | 66 | 849847 | 849847 |
| Wed Mar 6 01:43:50 PST 2019 | 29834 | 77 | 991491 | 991491 |
| Wed Mar 6 01:43:52 PST 2019 | 34096 | 88 | 1133135 | 1133135 |
| Wed Mar 6 01:43:54 PST 2019 | 38358 | 99 | 1274779 | 1274779 |
| Wed Mar 6 01:43:56 PST 2019 | 42620 | 110 | 1416423 | 1416423 |
| Wed Mar 6 01:43:58 PST 2019 | 46882 | 121 | 1558067 | 1558067 |
| Wed Mar 6 01:44:00 PST 2019 | 51144 | 132 | 1699711 | 1699711 |
| Wed Mar 6 01:44:02 PST 2019 | 55406 | 143 | 1841355 | 1841355 |
| Wed Mar 6 01:44:04 PST 2019 | 59668 | 154 | 1982999 | 1982999 |
| Wed Mar 6 01:44:06 PST 2019 | 63930 | 165 | 2124643 | 2124643 |
| Wed Mar 6 01:44:08 PST 2019 | 68192 | 176 | 2266287 | 2266287 |
| Wed Mar 6 01:44:10 PST 2019 | 72454 | 187 | 2407933 | 2407933 |
| Wed Mar 6 01:44:12 PST 2019 | 76716 | 198 | 2549579 | 2549579 |
| Wed Mar 6 01:44:14 PST 2019 | 80978 | 209 | 2691225 | 2691225 |
| Wed Mar 6 01:44:16 PST 2019 | 85240 | 220 | 2832871 | 2832871 |
| Wed Mar 6 01:44:18 PST 2019 | 89502 | 231 | 2974517 | 2974517 |
| Wed Mar 6 01:44:20 PST 2019 | 93764 | 242 | 3116163 | 3116163 |
| Wed Mar 6 01:44:22 PST 2019 | 98026 | 253 | 3257809 | 3257809 |
| Wed Mar 6 01:44:24 PST 2019 | 102288 | 264 | 3399455 | 3399455 |
| Wed Mar 6 01:44:26 PST 2019 | 106550 | 275 | 3541101 | 3541101 |
| Wed Mar 6 01:44:28 PST 2019 | 110812 | 286 | 3682749 | 3682749 |
| Wed Mar 6 01:44:30 PST 2019 | 115074 | 297 | 3824400 | 3824400 |

Collector Stats (Run time : 1m0.000702329s)
308 : in-packets
119336 : data points (KV pairs)
194 : in-header wirelength (bytes)
3966051 : in-payload length (bytes)
3966051 : in-payload wirelength (bytes)
66100 : throughput (bytes per seconds)

| Wed Mar 6 01:44:32 PST 2019 | 119336 | 308 | 3966051 | 3966051 |
| Wed Mar 6 01:44:34 PST 2019 | 123598 | 319 | 4107702 | 4107702 |
| Wed Mar 6 01:44:36 PST 2019 | 127860 | 330 | 4249353 | 4249353 |
| Wed Mar 6 01:44:38 PST 2019 | 132122 | 341 | 4391004 | 4391004 |
| Wed Mar 6 01:44:40 PST 2019 | 136384 | 352 | 4532655 | 4532655 |
| Wed Mar 6 01:44:42 PST 2019 | 140646 | 363 | 4674306 | 4674306 |
| Wed Mar 6 01:44:45 PST 2019 | 144908 | 374 | 4815957 | 4815957 |
| Wed Mar 6 01:44:47 PST 2019 | 149170 | 385 | 4957608 | 4957608 |
| Wed Mar 6 01:44:49 PST 2019 | 153432 | 396 | 5099259 | 5099259 |
| Wed Mar 6 01:44:51 PST 2019 | 157694 | 407 | 5240910 | 5240910 |
| Wed Mar 6 01:44:53 PST 2019 | 161956 | 418 | 5382561 | 5382561 |
| Wed Mar 6 01:44:55 PST 2019 | 166218 | 429 | 5524212 | 5524212 |
| Wed Mar 6 01:44:57 PST 2019 | 170480 | 440 | 5665863 | 5665863 |
| Wed Mar 6 01:44:59 PST 2019 | 174742 | 451 | 5807514 | 5807514 |
| Wed Mar 6 01:45:01 PST 2019 | 179004 | 462 | 5949165 | 5949165 |
| Wed Mar 6 01:45:03 PST 2019 | 183266 | 473 | 6090817 | 6090817 |
| Wed Mar 6 01:45:05 PST 2019 | 187528 | 484 | 6232471 | 6232471 |
| Wed Mar 6 01:45:07 PST 2019 | 191790 | 495 | 6374125 | 6374125 |
| Wed Mar 6 01:45:09 PST 2019 | 196052 | 506 | 6515779 | 6515779 |
| Wed Mar 6 01:45:11 PST 2019 | 200314 | 517 | 6657433 | 6657433 |
| Wed Mar 6 01:45:13 PST 2019 | 204576 | 528 | 6799087 | 6799087 |
| Wed Mar 6 01:45:15 PST 2019 | 208838 | 539 | 6940741 | 6940741 |
| Wed Mar 6 01:45:17 PST 2019 | 213100 | 550 | 7082395 | 7082395 |
| Wed Mar 6 01:45:19 PST 2019 | 217362 | 561 | 7224049 | 7224049 |
| Wed Mar 6 01:45:21 PST 2019 | 221624 | 572 | 7365703 | 7365703 |
| Wed Mar 6 01:45:23 PST 2019 | 225886 | 583 | 7507357 | 7507357 |
| Wed Mar 6 01:45:25 PST 2019 | 230148 | 594 | 7649011 | 7649011 |
| Wed Mar 6 01:45:27 PST 2019 | 234410 | 605 | 7790665 | 7790665 |
| Wed Mar 6 01:45:29 PST 2019 | 238672 | 616 | 7932319 | 7932319 |
| Wed Mar 6 01:45:31 PST 2019 | 242934 | 627 | 8073973 | 8073973 |
| Wed Mar 6 01:45:33 PST 2019 | 247196 | 638 | 8215627 | 8215627 |
| Wed Mar 6 01:45:35 PST 2019 | 251458 | 649 | 8357281 | 8357281 |
| Wed Mar 6 01:45:37 PST 2019 | 255720 | 660 | 8498935 | 8498935 |
| Wed Mar 6 01:45:39 PST 2019 | 259982 | 671 | 8640589 | 8640589 |
| Wed Mar 6 01:45:41 PST 2019 | 264244 | 682 | 8782243 | 8782243 |
| Wed Mar 6 01:45:43 PST 2019 | 268506 | 693 | 8923897 | 8923897 |
root@jvision-ubuntu3:/opt/go-collector/RLI-35537#

IPv6 Tests

JTIMON already works with IPv6 - this issue is to add tests to that effect.

Couple of optimization in influx.go

There is couple of optimization possibilities in influx.db.

  1. Instead of calling client.NewPoint for every point, points can be accumulated and client.Newpoint can be called in the end.
  2. Minor optimisation in prefix

Attached untested diff with these optimisation.

jtimon.txt

path_list not detected (thus, no telemetry data is shown)

Hello, I was trying to use jtimon with a Juniper VMX router. The router has a simple configuration (gRPC configured with no-authentication) that seems to work with Telegraf (telemetry data being received).

On the same server, I try to query with jtimon, the connection seems to appear but the "path_list" is not being returned (see below). I tried to change a bunch of stuff : different path, setting gRPC window size (0 by default), etc... Am I missing something ? Thanks for your help / advices !

[centos@telegraf01 ~]$ sudo /usr/local/go/bin/jtimon --config bibou --print
2020/11/27 04:11:38 Version: version-not-available BuildTime build-time-not-available
2020/11/27 04:11:38 logging in for 10.0.100.201:32767 [periodic stats every 0 seconds]
Running config of JTIMON:
{
"port": 32767,
"host": "10.0.100.201",
"user": "",
"password": "",
"cid": "",
"meta": false,
"eos": false,
"grpc": {
"ws": 524288
},
"tls": {
"clientcrt": "",
"clientkey": "",
"ca": "",
"servername": ""
},
"influx": {
"server": "",
"port": 0,
"dbname": "",
"user": "",
"password": "",
"recreate": false,
"measurement": "",
"batchsize": 102400,
"batchfrequency": 2000,
"http-timeout": 30,
"retention-policy": "",
"accumulator-frequency": 2000,
"write-per-measurement": false
},
"paths": [
{
"path": "/junos/system/linecard/cpu/memory",
"freq": 2000,
"mode": ""
}
],
"log": {
"file": "",
"periodic-stats": 0,
"verbose": false
},
"vendor": {
"name": "",
"remove-namespace": false,
"schema": null,
"gnmi": null
},
"alias": "",
"password-decoder": ""
}
invoking getInfluxClient for init
invoking getInfluxClient
compression = none
Connecting to 10.0.100.201:32767
Calling subscribe() ::: bibou
gRPC headers from host 10.0.100.201:32767
init-response: [response { subscription_id: 1 } ]
content-type: [application/grpc]
grpc-accept-encoding: [identity,deflate,gzip]
Receiving telemetry data from 10.0.100.201:32767

periodic stats not working?

I am looking for periodic stats from the collector, but I get nothing.

root@yangtze:~/hfzhang/D101# ./jtimon --config d102.json
Version: version-not-available BuildTime build-time-not-available
2018/08/24 11:03:00 logging in d102.test.log for 1.100.0.0:10162 [periodic stats every 2 seconds]

Here is my json file.

{
"host": "1.100.0.0",
"port": 10162,
"user": "regress",
"password": "MaRtInI",
"cid": "cid-yangtze-d100-3",
"influx" : {
"server" : "127.0.0.1",
"port" : 8086,
"dbname" : "tinyvale_kv",
"measurement" : "OC",
"user" : "influx",
"password" : "influxdb",
"recreate" : true
},
"tls" : {
"clientcrt" : "client/client.crt",
"clientkey" : "client/client.key",
"ca" : "CA/RootCA.crt",
"servername" : "tinyvale"
},
"grpc" : {
"ws" : 1048576
},
"paths": [
...
],
"log": {
"file": "d102.test.log",
"verbose": true,
"periodic-stats": 2,
"drop-check": true,
"latency-check": false,
"csv-stats": true,
"FileHandle": null,
"Logger": null
}
}

TelemetrySubscribe(_) = _, rpc error: code = Unknown desc = Authorization failed

Hi,
i am run the following command:
./jtimon --config test.json --gtrace --print

and the result is the following:

2018/09/20 04:41:36 Connecting to 10.49.169.179:50051
2018/09/20 04:41:36 gRPC headers from host 10.49.169.179:50051
2018/09/20 04:41:36 content-type: [application/grpc]
2018/09/20 04:41:36 grpc-accept-encoding: [identity,deflate,gzip]
2018/09/20 04:41:36 Receiving telemetry data from 10.49.169.179:50051
2018/09/20 04:41:36 &{0xc8200152c0 0x6a7340 10.49.169.179:50051 {passthrough 10.49.169.179:50051} 10.49.169.179:50051 { {120000000000 1000000000 1.6 0.2} false true 0 {grpc-go/1.11.3 0x5556f0 false [] {0 0 false} 524288 0 0 0} [] 0xecd8a8 false} 0xc82015ebe0 { 0x5556f0} 0xc82014dc80 0xc820015280 {{0 0} 0 0 0 0} { map[]} map[0xc820160b40:{}] {0 0 false} pick_first [{10.49.169.179:50051 0 }] 0xc820015580}.TelemetrySubscribe(_) = _, rpc error: code = Unknown desc = Authorization failed
2018/09/20 04:41:36 Restarting the connection
2018/09/20 04:41:36 Retrying the connection

the user correctly login into the node but it seems there is something wrong with authorization.

Regards,
Max

about data format

Hi,

I am thinking of using this tool for collecting MX telemetry info in a multi-vendor environment.
My question is, is there any kind of feature available in this tool to change the format to JSON?

MX, my primary target for checking telemetry, supports GPB format for encoding only (GPB-SDM too?).
On the other hand i need the data to be in JSON format.
So if there is any kind of feature available please let me know.

Regards,
hz2102

jtimon not working

This was working earlier. Not sure what went wrong. Any Clue? Tried changing cid, still see same issue.

/root/bin/jtimon --config srx4600_components_jtimon.cfg --print

2018/10/30 10:15:45 Connecting to 10.157.68.230:50052
2018/10/30 10:15:45 [0] Could not login: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing failed to do connect handshake, response: "HTTP/1.1 500 Domain Not Found\r\nContent-Length: 246\r\nAccept-Ranges: bytes\r\nCache-Control: private, no-cache\r\nConnection: keep-alive\r\nConnection: keep-alive\r\nContent-Type: text/html\r\nDate: Tue, 30 Oct 2018 17:15:42 GMT\r\nRetry-After: 0\r\nServer: Varnish\r\nVia: 1.1 varnish\r\nX-Served-By: cache-sjc3139-SJC\r\n\r\n\n\n\n<title>Fastly error: unknown domain 10.157.68.230</title>\n\n\n

Fastly error: unknown domain: 10.157.68.230. Please check that this domain has been added to a service.

\n

Details: cache-sjc3139-SJC

""
All done ... exiting!

About Kafka and/or logstash support

Hi,

Do you have plan to support other tools like kafka, logstash, elastic search etc.

If you have any kind of plans for supporting please let me know.

Regards,
HZ2102

grpc connection mux

For a given device, Dial() only once and send subscribe() on the same connection to avoid multiple gRPC connections to the device.

Alias not working

Hi,

We have some metrics that doesn't convert to the alias short name:

# HELP _lldp_state_counters_frame_in JTIMON Metric
# TYPE _lldp_state_counters_frame_in untyped
_lldp_state_counters_frame_in{device="XXXX"} 1086
# HELP _lldp_state_counters_frame_out JTIMON Metric
# TYPE _lldp_state_counters_frame_out untyped
_lldp_state_counters_frame_out{device="XXXX"} 1174
# HELP _lldp_state_counters_tlv_accepted JTIMON Metric
# TYPE _lldp_state_counters_tlv_accepted untyped
_lldp_state_counters_tlv_accepted{device="XXXX"} 2.7879268e+07
# HELP _lldp_state_enabled JTIMON Metric
# TYPE _lldp_state_enabled untyped
_lldp_state_enabled{device="XXXX"} 1
# HELP _lldp_state_hello_timer JTIMON Metric
# TYPE _lldp_state_hello_timer untyped
_lldp_state_hello_timer{device="XXXX"} 30

Here's our alias.txt:

lldp-frame-in:/lldp/state/counters/frame-in
lldp-frame-out:/lldp/state/counters/frame-out
lldp-hello-timer:/lldp/state/hello-timer
lldp-system-name:/lldp/state/system-name
lldp-enabled:/lldp/state/enabled

Cannot make docker image 'cannot load protobuf descriptor'

Hi,
I no longer seem to be able to make docker image and getting the error below:
'build github.com/nileshsimaria/jtimon: cannot load github.com/golang/protobuf/protoc-gen-go/descriptor: open /go/src/app/vendor/github.com/golang/protobuf/protoc-gen-go/descriptor: no such file or directory'
Any guidance what could be causing this?
Thank you.

Error - 2018/05/31 15:53:36 [0] Could not login: rpc error: code = InvalidArgument desc = JGrpcServer: Client Id jti already exists

It was working before and it stopped, getting below error. Any ideas?

2018/05/31 15:53:24 batch size: 1048576 batch frequency: 2000
2018/05/31 15:53:24 Successfully initialized InfluxDB Client
2018/05/31 15:53:24 Connecting to 192.168.1.139:50051
2018/05/31 15:53:26 gRPC headers from host 192.168.1.139:50051
2018/05/31 15:53:26 content-type: [application/grpc]
2018/05/31 15:53:26 grpc-accept-encoding: [identity,deflate,gzip]
2018/05/31 15:53:26 Receiving telemetry data from 192.168.1.139:50051
2018/05/31 15:53:26 &{0xc420012a80 0x55f9e46d95e0 192.168.1.139:50051 {passthrough 192.168.1.139:50051} 192.168.1.139:50051 { {120000000000 1000000000 1.6 0.2} false true 0 {grpc-go/1.11.3 0x55f9e495bb80 false [] {0 0 false} 0xc42000e288 524288 0 0 0} [] 0x55f9e50f2778 false} 0xc420133a40 { 0x55f9e495bb80} 0xc420141f20 0xc420012a40 {{0 0} 0 0 0 0} { map[]} map[0xc4200738c0:{}] {0 0 false} pick_first [{192.168.1.139:50051 0 }] 0xc420012bc0}.TelemetrySubscribe(_) = _, rpc error: code = Unknown desc = Authorization failed
2018/05/31 15:53:36 Reconnecting to 192.168.1.139:50051
2018/05/31 15:53:36 [0] Could not login: rpc error: code = InvalidArgument desc = JGrpcServer: Client Id jti already exists
^C2018/05/31 15:53:44

Collector Stats for 192.168.1.139:50051 (Run time : 19.962453879s)
0 : in-packets

Docker image size

Hi,

after latest commits docker image size grew almost 25 times.
Image build before e8782ca

REPOSITORY        TAG                 IMAGE ID            CREATED             SIZE
jtimon            latest              88df549101f1        5 days ago          17.4MB

Image build today after 1171e80

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
jtimon              latest              91672f0036b0        8 minutes ago       437MB

17.4MB vs 437MB size

Was a such huge difference in image size planned or it is bug? Thanks!

Regards,
Lukas

question about the Juniper api

The Juniper api used mentions https://github.com/openconfig/public/blob/master/release/models/rpc/openconfig-rpc-api.yang, which went away in May 2016 (openconfig/public@e6a2f26), and the whole .../release/models/rpc subtree went away a year after that (June 2017, openconfig/public@9dcdf78).

I've tested jtimon against a switch running junos 18.3R1.9, and of course it works perfectly. (Some of my own code, using the new(er) OpenConfig api, does not work nearly as well, due (so far as I can tell) to bugs on the Juniper side.)

Would you happen to know when or if Juniper will be upgrading to the new API (or, I guess I should say, fixing it, since you can actually make those rpc calls), or where I might go to get more information on that?

Thank you for your time and this code. As mentioned above, my own code using the new api is broken in severe ways, but this definitely provides a roadmap forwards. Thanks again and I appreciate it!

New Release possible?

Hey, first of all thank you so much for the project, great work! ๐ŸŽ‰

I was just wondering if we could see a new release in the near future? I would like to bind the Prometheus host on 0.0.0.0 which is possible since commit ef72e79 but hasn't made it into a release yet. I know I could always build from master but I've built an ansible role that fetches the release from the release page which makes it really simple :)

Some context, I've fetched the v2.2.0 binary and got:

$ jtimon --version
2020/05/18 18:23:19 Version: d7c3d5637f7582a612e97d693e98f70a617c9c76-master BuildTime 2019-04-23T17:55:07-0700

$ jtimon -h
Usage of jtimon:
      --compression string         Enable HTTP/2 compression (gzip)
      --config strings             Config file name(s)
      --config-file-list string    List of Config files
      --consume-test-data          Consume test data
      --explore-config             Explore full config of JTIMON and exit
      --generate-test-data         Generate test data
      --json                       Convert telemetry packet into JSON
      --log-mux-stdout             All logs to stdout
      --max-run int                Max run time in seconds
      --no-per-packet-goroutines   Spawn per packet go routines
      --pprof                      Profile JTIMON
      --pprof-port int32           Profile port (default 6060)
      --prefix-check               Report missing __prefix__ in telemetry packet
      --print                      Print Telemetry data
      --prometheus                 Stats for prometheus monitoring system
      --prometheus-port int32      Prometheus port (default 8090)
      --stats-handler              Use GRPC statshandler
      --version                    Print version and build-time of the binary and exit

Do not create go routine per point

Jtimon is creating a go routine per point. This makes jtimon consumes more memory when the write to the influx db database is slow. This never passes the back-pressure to the producer. Can this behaviour controlled via knob ?

jtimon --stats or --drop-check option

I don't see --stats or --drop-check option option. Am i missing anything? Any ideas?

root@Jumphost3:/usr/local/bin# jtimon --stats
unknown flag: --stats
Usage of jtimon:
--api Receive HTTP commands when running
--compression string Enable HTTP/2 compression (gzip, deflate)
--config strings Config file name(s)
--explore-config Explore full config of JTIMON and exit
--gnmi Use gnmi proto
--gnmi-encoding string gnmi encoding (proto | json | bytes | ascii | ietf-json (default "proto")
--gnmi-mode string Mode of gnmi (stream | once | poll (default "stream")
--grpc-headers Add grpc headers in DB
--gtrace Collect GRPC traces
--latency-profile Profile latencies. Place them in TSDB
--log-mux-stdout All logs to stdout
--max-run int Max run time in seconds
--pprof Profile JTIMON
--pprof-port int32 Profile port (default 6060)
--prefix-check Report missing prefix in telemetry packet
--print Print Telemetry data
--prometheus Stats for prometheus monitoring system
--prometheus-port int32 Prometheus port (default 8090)
--stats-handler Use GRPC statshandler
--version Print version and build-time of the binary and exit
unknown flag: --stats

tag keys defined by user

is there any way in which I can define my own tag keys to be inserted into influxdb ?

Example:

I'm gathering data for interfaces sensor (see below config)

    {
        "host": "10.102.187.188",
        "port": 50051,
        "cid": "mac1",
        "paths": [{
            "path": "/junos/system/linecard/interface/",
            "freq": 10000
        }],
        "influx": {
            "server": "172.17.0.2",
            "port": 8086,
            "dbname": "juniper",
            "user": "juniper",
            "password": "juniper",
            "recreate": true,
            "measurement": "/interface_test/"
        },
    
        "log": {
            "file": "test.log"
        }
    }

I'd need "/interfaces/interface/state/parent_ae_name" as tagKey in order to be able of grouping by this field . I know this can be done in jti_openconfig_telemetry plugin just setting the variable str_as_tags = true , that will treat all strings as tags.

Regards

Pablo

UDP server support

Hi

I see that the udp server support has gone - will it return ?

Regards
Michael

UDP server support

Hi @nileshsimaria,
is there plan to include UDP server for streamed data from Junos OS? MX80 and MX104 does not support OpenConfig + gRPC.

Seems that version in Juniper's repo is missing prometheus exporter for collected data. Reference: Juniper/jtimon@806930d. Is there intention to merge it into this repo?

Thanks a lot for reply.

Regards,
Lukas

Getting error with go build

I followed the steps and when doing go build, getting the below error, any ideas?

./main.go:195: cannot use conn (type *"google.golang.org/grpc".ClientConn) as type *"github.com/nileshsimaria/jtimon/vendor/google.golang.org/grpc".ClientConn in argument to authentication.NewLoginClient
./subscribe.go:81: cannot use conn (type *"google.golang.org/grpc".ClientConn) as type *"github.com/nileshsimaria/jtimon/vendor/google.golang.org/grpc".ClientConn in argument to telemetry.NewOpenConfigTelemetryClient
./subscribe_gnmi.go:115: cannot use conn (type *"google.golang.org/grpc".ClientConn) as type *"github.com/nileshsimaria/jtimon/vendor/google.golang.org/grpc".ClientConn in argument to gnmi.NewGNMIClient

Prometheus - metric has label dimensions inconsistent with previously collected metrics in the same metric family

Hi,

I'm using the latest version from git.

When trying to collect the endpoint /junos/services/label-switched-path/usage/ I'm presented with the following error messages when accessing the Prometheus metrics page:

An error has occurred during metrics gathering:

279 error(s) occurred:
* collected metric _mpls_lsps_constrained_path_tunnels_tunnel_state_counters_jnx_packet_rate label:<name:"_mpls_lsps_constrained_path_tunnels_tunnel__name" value:"lsp_name" > label:<name:"_mpls_lsps_constrained_path_tunnels_tunnel__source" value:"0.0.0.0" > label:<name:"_mpls_lsps_constrained_path_tunnels_tunnel_state_counters__name" value:"c-45772" > label:<name:"device" value:"lsp_device" > untyped:<value:87 >  has label dimensions inconsistent with previously collected metrics in the same metric family
[ above message repeatet 278 times with different labels and metrics all related to the endpoint mentioned above ]

There's various pieces of information here and here. The last link says that this limitation was removed in a later build of the Prometheus client library.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.