Currently we look at the defined message in the config and take a white space to be exactly 1 white space. We should change it so 1 white space will match 1+ white spaces in the actual message.

Turns out that's important when publishing over different channels where the receiver may not be able to unpack the binary serialised message, e.g., JS client that reads from a ZMQ PUB, when is easier to receive JSON, etc.

We should probably create a separate section called Clients, where to have:

• Salt
• StackStorm
• Scripts
  • Python
  • Java
  • Perl

Each syslog message will be parsed in the corresponding process.
They will share the same transport obj.

The adjacency reason message is simply a text currently. We should probably start building a list of the possible messages and map them into a standard structure rather than random bits of text. This can eventually be merged into the official openconfig-ospf model later - TDB. For the time being, let's just identify what messages we see. @lampwins could you start here a list of messages you've seen so far?

If the server restarts after the client has connected, then the client is left with old crypto info:

Traceback (most recent call last):
  File "/home/luke/git/napalm-logs/test-listen.py", line 19, in <module>
    decrypted = napalm_logs.utils.decrypt(data, vk, pk)
  File "/home/luke/git/napalm-logs/napalm_logs/utils/__init__.py", line 77, in decrypt
    raise BadSignatureException('Signature was forged or corrupt')
napalm_logs.exceptions.BadSignatureException: Signature was forged or corrupt

Therefore we need to have the clients restart, or allow them to use the old crypo info (not ideal)

Not only that there's a bug messing with global vars, but we could potentially have both index and message profiler in the same module (as some complex and obscure messages might belong to a class that don't use a generic prefix, but something very particular).

Start new child process serving auth requests
Implies new arg for auth port (default: publish + 1)
Authentication can be optional (if disabled, won't start the auth process). If disabled, accept and serve any client.
Main goal: the client has to be sure that it can trust the publisher
RSA and AES
The CLI script plugged to the authentication process through zmq IPC

Apparently it's faster

With a page for each of the supported platforms.

Starting from https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/concepts-message-ietfsyslog.html

See details in the parent issue napalm-automation/napalm-base#217

Example

1st input:

<149>Mar 30 12:45:19  edge01.xyz01 rpd[1234]: BGP_PREFIX_THRESH_EXCEEDED: 1.2.3.4 (External AS 123): Configured maximum prefix-limit threshold(160) exceeded for inet-unicast nlri: 181 (instance master)

1st Output

{'message_details': {'processId': '1234', 'pri': '149', 'processName': 'rpd', 'host': 'edge01.xyz01', 'tag': 'BGP_PREFIX_THRESH_EXCEEDED', 'time': '12:45:19', 'date': 'Mar 30', 'message': '1.2.3.4 (External AS 123): Configured maximum prefix-limit threshold(160) exceeded for inet-unicast nlri: 181 (instance master)'}, 'open_config': {'bgp': {'neighbors': {'neighbor': {u'1.2.3.4': {'state': {'peer_as': 123}, 'afi_safis': {'afi_safi': {u'inet': {'state': {'prefixes': {'received': 181}}, 'afi_safi_name': u'inet', 'ipv4_unicast': {'prefix_limit': {'state': {'max_prefixes': 160}}}}}}, 'neighbor_address': u'1.2.3.4'}}}}}, 'ip': '127.0.0.1', 'error': 'BGP_PREFIX_THRESH_EXCEEDED', 'host': 'edge01.xyz01', 'timestamp': '1490874319', 'os': 'junos', 'model_name': 'openconfig_bgp'}

2nd input:

<149>Mar 30 12:45:19  edge01.xyz01 rpd[1234]: BGP_PREFIX_THRESH_EXCEEDED: 5.6.7.8 (External AS 123): Configured maximum prefix-limit threshold(160) exceeded for inet-unicast nlri: 181 (instance master)

2nd Output:

{'message_details': {'processId': '1234', 'pri': '149', 'processName': 'rpd', 'host': 'edge01.xyz01', 'tag': 'BGP_PREFIX_THRESH_EXCEEDED', 'time': '12:45:19', 'date': 'Mar 30', 'message': '5.6.7.8 (External AS 123): Configured maximum prefix-limit threshold(160) exceeded for inet-unicast nlri: 181 (instance master)'}, 'open_config': {'bgp': {'neighbors': {'neighbor': {u'1.2.3.4': {'state': {'peer_as': 123}, 'afi_safis': {'afi_safi': {u'inet': {'state': {'prefixes': {'received': 181}}, 'afi_safi_name': u'inet', 'ipv4_unicast': {'prefix_limit': {'state': {'max_prefixes': 160}}}}}}, 'neighbor_address': u'1.2.3.4'}, u'5.6.7.8': {'state': {'peer_as': 123}, 'afi_safis': {'afi_safi': {u'inet': {'state': {'prefixes': {'received': 181}}, 'afi_safi_name': u'inet', 'ipv4_unicast': {'prefix_limit': {'state': {'max_prefixes': 160}}}}}}, 'neighbor_address': u'5.6.7.8'}}}}}, 'ip': '127.0.0.1', 'error': 'BGP_PREFIX_THRESH_EXCEEDED', 'host': 'edge01.xyz01', 'timestamp': '1490874319', 'os': 'junos', 'model_name': 'openconfig_bgp'}

As you can see in the example the details of the 1st input are present, this is not intended.

Analysis

This is due to us caching the openconfig object, each time we load a new dict into the model this is stored in the cached version.

Message example: re0.edge01.bjm01 rpd[10124]: BGP_PREFIX_THRESH_EXCEEDED: 1.2.3.4 (External AS 1234): Configured maximum prefix-limit threshold(80) exceeded for inet-unicast nlri: 87 (instance master)

At least for Debian.
@loverend would you be happy to do this after releasing the first beta on PyPi?

Keywords: socket, tcp.

Introduced in #168, we might need to evaluate on the long run if the current structure really makes sense cross-platform:

"system": {
    "configuration": {
        "error": true,
        "message": "Interface must already be defined under [edit interfaces]",
        "path": "[edit vlans VLANTEST l3-interface]",
        "statement": "l3-interface vlan.666"
    }
}

So it simply forwards the raw messages (with the sole advantage that is a little bit more structured, and the OS is identified + other details discovered).
This option should default to False.

Send data between processes using the IPC zmq proxy.

Given that many messages won't have a unique identification tag (e.g., something like ROUTING-BGP-5-MAXPFX or at least bgp_read_message) we'll need to find other ways to match the message. In that case it might be confusing to use tag where there's not. But that's just a detail, looking for more feedback.

Allowing the user to start only some processes, not everything.

Hello,

it seems like napalm-logs is missing some kind of translation for the ietf standard.

• Or an option to set syslog messages as ietf standard.
  I've configured the napalm-syslog server with basic configuration, no changes. Only set an ipaddress, port and disabled security.

Napalm-Logs is expecting the syslog message in following syntax:
<129>Oct 23 15:58:20 berlin cscript "message"

When i am looking at my tcpdump, juniper output as well as napalm log, the date format differs.

Juniper: Oct 23 15:58:20
TCP: 2017-10-23T16:02:38.950+02:00
napalm log: 2017-10-23T16:02:38.950+02:00

tcpdump:
Msg: 1 2017-10-23T16:02:38.950+02:00 berlin cscript - - - MX80 SN:XXXXX has booted 16.1R4-S4.3.
Uptime is 9 days, 23 hours, 7 minutes, 20 seconds

junos:
lab@berlin>show log messages | last 1
Oct 23 15:58:20 berlin cscript: MX80 SN:XXXXX has booted 16.1R4-S4.3. Uptime is 9 days, 23 hours, 3 minutes, 5 seconds

var/log/napalm/logs:
Dequeued message from <129>1 2017-10-23T16:02:38.950+02:00 berlin cscript - - - MX80 SN:XXXXX has booted 16.1R4-S4.3. Uptime is 9 days, 23 hours, 7 minutes, 20 seconds: 1508766848.76
2017-10-23 15:54:08,764,765 [napalm_logs.server][DEBUG ] Matching under junos
2017-10-23 15:54:08,765,765 [napalm_logs.server][DEBUG ] Matching using YAML-defined profiler:
2017-10-23 15:54:08,765,765 [napalm_logs.server][DEBUG ] <(\d+)>(\w+\s+\d+)\s+(\d\d:\d\d:\d\d)\s+(re\d.)?([^ ]+)\s+/?(\w+)[?(\d+)?]?:\s+([\w\s]+):(.*)
2017-10-23 15:54:08,765,765 [napalm_logs.server][DEBUG ] Match not found

Cheers!

If the specified certificate does not exist the cli script will error when a client tries to connect. It should produce an error during startup.

So you can run napalm-logs in non-sync'd locations; if the device does not send explicitly the timezone, it will use this one.
The option should default to UTC.
It may imply having another requirement, pytz.

We need to investigate more around zeromq/pyzmq#1009 in terms of latency and throughput over the IPC vs the multiprocessing Pipe.

I personally still believe that zmq should perform better. I also suspect that Pipe may miss messages when the buffer is full (send does not fail, hence we can't know whether the message has been queued or not). On the other hand, zmq does not miss any messages for sure, but we need to dig more into the delay issue.

And notify the clients to reinit the key exchange.

Currently an exception in the listener process only causes the listener to fail. Without the listener process the whole thing is pointless, therefore if the listener processes dies, all processes should die.

The service can be selected from on of the available transports and can have its separate options.

Which can be polled from an ZMQ IPC.

On the client must detach the nonce block and store the nonces during a certain delta seconds, to prevent replay attacks.

Currently there is no documentation regarding development testing for napalm-logs. Ideally this should cover the testing methodology (or at least refer to the napalm-automation general topic on this). It should document how new test cases for configured messages should be setup.

Just a small thread that all it does it replies with True when asked if napalm-logs is still alive.
It must be zmq IPC based.

We should add metrics so one can easily see how many messages have been processed, how many per device, rate per hour etc

Write script to profile CPU & mem usage + max messages able to process when injecting artificial messages.

Make security optional - there are cases when people would only need to send the data as-is to a log server, i.e. logstash etc. In that case, security is a bit overkill.
Ping @loverend - what do you think?