napalm-automation / napalm-base Goto Github PK

Relates to napalm-automation/napalm#280

Add setup.cfg and requirements-dev.txt file with the following content:

• `setup.cfg

[pylama]
linters = mccabe,pep8,pyflakes
ignore = D203,C901

[pylama:pep8]
max_line_length = 100

• requirements-dev.txt

pytest
pytest-cov
pytest-json
pytest-pythonpath
pylama
-r requirements.txt

napalm-base is reporting napalm-ios version (in released version)

try:
    __version__ = pkg_resources.get_distribution('napalm-ios').version
except pkg_resources.DistributionNotFound:
    __version__ = "Not installed"

For the hackathon, was looking at adding the functionality to copy files to a device with NAPALM.

napalm-base fails during installation. Errors are as below:

Searching for pyangbind Reading https://pypi.python.org/simple/pyangbind/ Best match: pyangbind 0.5.4 Downloading https://pypi.python.org/packages/e3/6c/28ab8289531e83540f886d1708bc6952667c49fe7e0d127a486ba41a3de3/pyangbind-0.5.4.tar.gz#md5=dcce61a1174c7a03e557e9b60800f961 Processing pyangbind-0.5.4.tar.gz Writing /tmp/easy_install-Yeb4YZ/pyangbind-0.5.4/setup.cfg Running pyangbind-0.5.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Yeb4YZ/pyangbind-0.5.4/egg-dist-tmp-eeiFdw Traceback (most recent call last): File "setup.py", line 33, in <module> 'cl_napalm_configure=napalm_base.clitools.cl_napalm_configure:main', File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/command/install.py", line 67, in run self.do_egg_install() File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/command/install.py", line 117, in do_egg_install cmd.run() File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 409, in run self.easy_install(spec, not self.no_deps) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 645, in easy_install return self.install_item(None, spec, tmpdir, deps, True) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 696, in install_item self.process_distribution(spec, dist, deps) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 741, in process_distribution [requirement], self.local_index, self.easy_install File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/pkg_resources/__init__.py", line 826, in resolve dist = best[req.key] = env.best_match(req, ws, installer) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1092, in best_match return self.obtain(req, installer) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1104, in obtain return installer(requirement) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 664, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 694, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 875, in install_eggs return self.build_and_install(setup_script, setup_base) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1114, in build_and_install self.run_setup(setup_script, setup_base, args) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1100, in run_setup run_setup(setup_script, args) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/sandbox.py", line 246, in run_setup raise File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context yield File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules saved_exc.resume() File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume six.reraise(type, exc, self._tb) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules yield saved File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context yield File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/sandbox.py", line 243, in run_setup DirectorySandbox(setup_dir).run(runner) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/sandbox.py", line 273, in run return func() File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/sandbox.py", line 242, in runner _execfile(setup_script, ns) File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/setuptools/sandbox.py", line 46, in _execfile exec(code, globals, locals) File "/tmp/easy_install-Yeb4YZ/pyangbind-0.5.4/setup.py", line 8, in <module> __author__ = 'David Barroso <[email protected]>' File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/pip/req/req_file.py", line 84, in parse_requirements filename, comes_from=comes_from, session=session File "/Users/nseshan/Desktop/temp/napalm-base/blah/lib/python2.7/site-packages/pip/download.py", line 417, in get_file_content 'Could not open requirements file: %s' % str(exc) pip.exceptions.InstallationError: Could not open requirements file: [Errno 2] No such file or directory: '/tmp/easy_install-Yeb4YZ/pyangbind-0.5.4/requirements.txt'

I have noticed another inconsistency in terms of default values when some fields cannot be filled with a corresponding value (for various reasons, e.g.: not provided by the vendor, inconsistent value - sometimes , for the same field a stupid vendor returns str or int etc.).

When this happens, we need to return a value though. For string is pretty obvious - returning empty value; for bool I think the right thing is False in most of the cases. But how about int or float? Do you think we should/can keep this homogenous?

Eventually we could define a new module under napalm-base, e.g.: defaults.py and define a hash such:

map = {
  str: u'',
  int: -1,
  float: -1.0
}

@napalm-automation/council what do you think?

Hello,
from now on we need to maintain a changelog file. It's interesting for several reasons but the main reason I want it is so I can refer to that when releasing a new version instead of having to go through the PRs. The idea os to have something like:

v1.14.0

• #13 [new_method] get_ntp_peers
• #22 [bugfix] blah blah blah
  ..

I think we should have the following tags:

• new_method, bugfix, enhancement, documentation

Thoughts?

@mirceaulinic @ktbyers @fooelisa

Given the new split up repositories, the install procedure for NAPALM is not self-evident.

We need to document:

1. Install procedure using pip
2. Install procedure using python setup.py install

Probably cover any differences if installing full NAPALM versus only support for specific platform (i.e EOS/JUNOS/IOS)

At line 117 of napalm_base/helpers.py the following code converts everything to a string:

entry[fsm_handler.header[index].lower()] = str(entry_value)

This includes lists that can be returned from the template. I suggest it changes to the following:

entry[fsm_handler.header[index].lower()] = entry_value

Matt.

I was wondering if it may make sense to have some operational validation capabilities into NAPALM, like having the possibility to run device.validate("bgp") and check if everything looks as expected.

If it's cool, what do you think may be something good to validate? BGP peers, reachability, LLDP peers..?

The napalm_base/test/models.py uses advertise_prefix_count whereas the documentation (http://napalm.readthedocs.io/en/latest/base.html) uses both. It would seem more natural for it to be advertised_prefix_count to match received/accepted/suppressed etc but it would be good to confirm which it is.

At the time being, all config format supported by NAPALM is only text. This is not a problem for OS such as IOS-XR, IOS or EOS, but for JunOS that allows few other formats, e.g.set, xml or json this constraint becomes a bit uncomfortable.
Right now, for JunOS the format is bespoken as text: https://github.com/napalm-automation/napalm-junos/blob/develop/napalm_junos/junos.py#L118 and does not allow the user at all to change this.
Some folks (including myself) find the set format more natural and easier to deal with.
Therefore, I would like to add one arg, e.g. format, without having any impact:

def load_merge_candidate(self, filename=None, config=None, format='text'):

@napalm-automation/council what do you think?

While trying the new compliance_report class with PR175, PR176 and the ping method on junos, I stumbled upon this error:

>>> from napalm_base import get_network_driver
>>> import pprint
>>> 
>>> 
>>> junos_driver = get_network_driver("junos")
>>> junos_config = {
...     "hostname": "myhostname",
...     "username": "xionox",
...     "password": "",
...     "optional_args": {"allow_agent": True, "use_keys": True},
... }
>>> with junos_driver(**junos_config) as junos:
...     pprint.pprint(junos.compliance_report("validation/myhostname.yml"))
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/.../network-automation/env/local/lib/python2.7/site-packages/napalm_base/base.py", line 60, in __exit__
    self.__raise_clean_exception(exc_type, exc_value, exc_traceback)
  File "/.../network-automation/env/local/lib/python2.7/site-packages/napalm_base/base.py", line 82, in __raise_clean_exception
    raise exc_value
NotImplementedError

After talking to dbarosso, I realized that the ping method wasn't implemented yet on the Junos driver.

Maybe this error could be more explicit on what is going on?

We have experienced service deployment issues when new napalm packages are released, but the package version number is not incremented.

I would like to pin our deployments to a particular napalm release until we have tested it, but with large changes being included without the version be incremented this is problematic.

An example of this is where napalm-panos was added as a package dependency to the base package, but the base version not changed. The panos module itself included extra Python dependencies which needed additional system packages installed.

As per #111, in order to ease the execution of arbitrary commands (both oper and config type) on devices grouped by vendor/os, we will need to implement a new method run_commands.

The body should be similar to:

def run commands(self, commands, config=False, **kwargs):
    " Commands is a list of commands"
    if config:
        # code to send set commands to the device or even use **kwargs to do some tweaks, this method is completely tied to particular vendors so we can have a similar approach as with the optional_args
    else:
       for command in commands:
            self.device.cli(command, **kwargs)

The goal is to be able to do something like:

import napalm_base
print(napalm_base.__version__)

See the following commit for details:

https://github.com/napalm-automation/napalm-base/pull/80/files

Currently the code builds raw dictionaries for the various methods. The tests check that the dictionaries match napalm_base.test.models structures.

Since the returned data structure is important, these test-only models could be "promoted" to fully-fledged classes of the design domain. The driver methods could then populate these data structs, and the structs could have a validate method to ensure the data adheres to a required standard. This would be good for program documentation and correctness.

May or may not be useful at this point, given that many methods have been implemented.

Most models in models.py are already using unicode instead of str. users and config are still using str, as bellow:

users = {
    'level': int,
    'password': str,
    'sshkeys': list
}

config = {
    'running': str,
    'startup': str,
    'candidate': str,
}

Shouldn't we just use unicode wherever any kind of text return is expected?

I bumped into this when implementing the get_config method for IOS. The TestIOSDriver.py returns unicode when reading from mock data files, and it causes the assert self._test_model test to fail.

See details in the parent issue napalm-automation/napalm#327

There is some work going on to build the getters support matrix automatically. For more information:

napalm-automation/napalm#324

However, there are a few requirements to accomplish that goal:

1. Migrate all the drivers to the new testing framework; #99
2. Add to all drivers on .travis.yml:

after_success:
   - if [ $TRAVIS_TAG ]; then curl -X POST https://readthedocs.org/build/napalm; fi

At present the remote peer router id is only available in the output of get_bgp_neighbors() and not in get_bgp_neighbors_detail().

@dbarrosop suggested that get_bgp_neighbors_detail()[my_vrf][my_neigh_ip] could be used to get this information (slight typo I think, probably should be get_bgp_neighbors()[my_vrf][my_neigh_ip]) but that supposes that you have the VRF information available (or else you have to loop through the VRFs to find the peer).

If 'routing_table' in get_bgp_neighbors_detail() is the same as the VRF in get_bgp_neighbors() then this isn't an issue. If they are not, then I propose adding the remote peer router id into the get_bgp_neighbors_detail() method.

To be spread for junos, iosxr and eos drivers

#128

There is an issue in the model: route_descriptor is incorrectly referenced instead of route_distinguisher.

In the test, we incorrectly attempt to iterate over the keys instead of key/value.

I tried using Napalm on a freshly installed Debian Jessie box and got the following error:

>>> from napalm_base import get_network_driver
>>> d = get_network_driver('junos')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/napalm_base/__init__.py", line 73, in get_network_driver
    install_name=module_install_name
napalm_base.exceptions.ModuleImportError: Cannot import "napalm_junos". Is the library installed?

After trying from napalm_junos import junos that @dbarrosop told me to try I got:

>>> from napalm_junos import junos
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/napalm_junos/__init__.py", line 16, in <module>
    from junos import JunOSDriver
  File "/usr/local/lib/python2.7/dist-packages/napalm_junos/junos.py", line 19, in <module>
    from napalm_junos.utils import junos_views
  File "/usr/local/lib/python2.7/dist-packages/napalm_junos/utils/junos_views.py", line 4, in <module>
    from jnpr.junos.factory import loadyaml
  File "/usr/local/lib/python2.7/dist-packages/jnpr/junos/__init__.py", line 1, in <module>
    from jnpr.junos.device import Device
  File "/usr/local/lib/python2.7/dist-packages/jnpr/junos/device.py", line 16, in <module>
    from ncclient import manager as netconf_ssh
  File "/usr/local/lib/python2.7/dist-packages/ncclient/manager.py", line 22, in <module>
    from ncclient import operations
  File "/usr/local/lib/python2.7/dist-packages/ncclient/operations/__init__.py", line 16, in <module>
    from ncclient.operations.rpc import RPC, RPCReply, RPCError, RaiseMode
  File "/usr/local/lib/python2.7/dist-packages/ncclient/operations/rpc.py", line 20, in <module>
    from ncclient.transport import SessionListener
  File "/usr/local/lib/python2.7/dist-packages/ncclient/transport/__init__.py", line 18, in <module>
    from ncclient.transport.ssh import SSHSession
  File "/usr/local/lib/python2.7/dist-packages/ncclient/transport/ssh.py", line 26, in <module>
    import paramiko
  File "/usr/local/lib/python2.7/dist-packages/paramiko/__init__.py", line 30, in <module>
    from paramiko.transport import SecurityOptions, Transport
  File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 53, in <module>
    from paramiko.dsskey import DSSKey
  File "/usr/local/lib/python2.7/dist-packages/paramiko/dsskey.py", line 27, in <module>
    from cryptography.hazmat.primitives.asymmetric.utils import (
ImportError: No module named utils

And after uninstalling the Debian provided module and installing from PIP it now works.

To make sure we have the proper napalm-base version we should make sure all public methods of a driver where called at least once during the testing phase.

See details in the parent issue napalm-automation/napalm#199

There have been some recent discussion about supporting VRFs in get_bgp_neighbors_detail() and bridge domains in get_mac_address_table() but these are not the only methods where there is potential missing support for some form of instance (routing, bridge etc).

I suggest that get_arp_table(), get_interfaces_ip(), get_route_to(), ping() and traceroute() should also be instance aware (or other methods added that are instance aware). It may be that lldp, ntp and probes also need to be on the list.

Thoughts?

Matt.

I was looking at the coverage results for napalm-junos: https://coveralls.io/builds/8872086/source?filename=napalm_junos%2Fjunos.py
and noticed there's no test for get_ntp_servers in https://github.com/napalm-automation/napalm-base/blob/develop/napalm_base/test/getters.py

I've been thinking about how to approach writing a driver for Cumulus Linux. This is a bit of a brain dump of where I am at present...

The getters() are relatively trivial as it's just a matter of parsing the relevant Linux commands. This made me think the best option maybe to start with a base Linux driver that any Linux based NOS could inherit from just adding any specifics on top.

The configuration side is more tricky as there is no one place where all the configuration is stored. You need to set multiple things in different files (or using different commands) with differing syntax. Cumulus provide a list of files (for backup) that may be touched during configuration which provides a starting place of where to look in terms of managing this - https://docs.cumulusnetworks.com/display/DOCS/Upgrading+Cumulus+Linux#UpgradingCumulusLinux-UnderstandingtheLocationsofConfigurationDataforManagement,Migration,andBackup

Cumulus also have some Ansible code that has some support for interfaces, VLANs and routing (Quagga) which might be used to assist with the configuration side - https://github.com/CumulusNetworks/cumulus-linux-ansible-modules

Another approach I'm considering is writing an interface that hides all this - something like the approach taken by Mikrotik with their Routerboard OS (ROS). Basically its just a translation from the CLI to/from the underlying configuration files or commands. Might be somewhat limiting however to only support what the CLI provides rather than the full range of commands available.

In #111 we have identified the need to make the helper function load_template more flexible in terms of config format, depending on the filename.

In order to achieve this, we'll need to include the format type in the filename, e.g. bgp_config.xml or bgp_config.xml. If not included, the format will be text.
If the template format is different than what the user specifies in the optional_args as config_format, we'll need to apply a hack to override this format and revert it back after loading the template.

Yet another proposal from my series of small handy features.

When using the load_template method, you know the template structure, but sometimes the result may be unknown, unexpected etc. and you don't know where to start looking from.
At the end of the day it is just a piece of software and the question Why this shit does not work? Is should! will come up sooner or later.

Whilst using it, I have identified two useful features:

• debug: return the result of the rendered template, without trying to load on the device
• dry run: load on the device -> compare config -> discard and return the diff

Changes implied:

• ATM the output of load_template is None, would be changed to str.
• add two arguments defaulting to false: test (for dry run) and debug (you guessed - for debug)

def load_template(cls, template_name, template_source=None, template_path=None,
                  openconfig=False, test=False, debug=False, **template_vars):

@napalm-automation/council thoughts?

I am working on a StackStorm pack for NAPALM (more on that later! 😄) and my sensor is designed to check on a few devices periodically for changes. In this case, a connection failure is perfectly normal, and I'd like to keep the stackstorm logs clean of such an event. However, this shows that the "catch-all" exception method is being printed, instead of leveraging a more traditional logging facility. When I'm looking at the console for my sensor, this makes things look dirty.

I can make the change, but wanted to open an issue in case this was done for a reason. The logging module is used elsewhere, could this be done here too instead of printing?

Disregard, I am just spamming the channel for the sake of testing.

As discussed this is going to be for the very long term, when we'll have proper documentation.

Opening this to keep tracking of features to be implementing.

This closes #117 as we can add the optional_args to the existing cli() method.

I'd like to start a discussion on whether get_mac_address_table() should be extended to support bridge domains rather than the current single table (albeit with VLANs). This would be analogous to VRFs for IP routing related functions.

My proposal would be for the outer dictionary key to be the bridge domain and the inner list would be the same (or similar) to the current output. Something like

{
  'bridge-domain1': [
    {
      'mac': <MAC>,
      'interface': <IF_NAME>,
      'vlan': <VLAN_ID>,
     etc...
    },
   {
     As above...
   },
  ],
  'bridge-domain2': [
   As above...
  ]
}

How VLAN fits into this I'm not sure as each bridge domain could be VLAN aware (or not). I guess a VLAN is a bridge domain so it may just need to move up a level.

Matt.

Hi guys, in many occasions network teams are not only responsible for switches/routers operation but also for other devices such as firewalls (or at least the way they connect/integrate with the network). I'd like to know what you guys think of adding firewall support to NAPALM. I envision this in two possible flavors:

• Regular network-device calls applied to firewalls such as show route/interface or the like
• Firewall specific functions involving objects, zones, rules, policies, etc

Looking forward to your comments

Best
Alejandro

Original ticket napalm-automation/napalm#291 created by @grizz

Add a function to get the full config as a text blob

Would obviously be platform specific, but would be handy for using NAPALM for development and device changes.

Two proposed additions:

• arg vrf
• next_hop_type key inside the output dict

Here are some improvements to load_template() I think we should make.

1. Support 'replace' operation. I would propose that we add a mode='merge' / mode='replace' parameter.

2. Have it behave properly if you specify 'my_template.j2' currently it will look for 'my_template.j2.j2'.

3. Be able to load templates from current working directory (as long as we can make this work in a reasonable way without breaking backwords compatibility).

4. Improve the docs here, http://napalm.readthedocs.io/en/latest/base.html

I think the behavior of some of the parameters needs clarified.

Without changing the current behaviour, the argument would increase the performances of the get_bgp_neighbors method. If not passed or empty/null, will return everything.
Of course we can retrieve everything and filter, but this is not optimal when there's a very high number of neighbors and you need the operational data for a specific peer only:

PoC:

Without neighbor_address:

>>> start = time.time(); w = e.get_bgp_neighbors(); print time.time() - start
8.1069419384

With neighbor_address:

>>> start = time.time(); w = e.get_bgp_neighbors('X.X.X.X'); print time.time() - start
0.508651018143

Make sure we set the latest napalm-base version as part of the requirements. The idea is that avoid breaking the guts of the driver without noticing (by changing an internal helper function or the testing framework or something like that).

Hey!

When using SSH key, asking for a password is not needed. Also, with an appropriate SSH configuration, we shouldn't override the user either. I suggest to set the default user to None if possible (for JunOS, py-eznc will use the current user if it doesn't find anything better).

References:

http://napalm.readthedocs.io/en/develop/development/testing_framework.html

napalm-automation/napalm-eos#47
#94

Create the file:

.github/ISSUE_TEMPLATE

With the following content: https://gist.github.com/dbarrosop/f056b131eddd1777ff77e1ec8ab79964

It makes sense to do this at the same time as napalm-automation/napalm#297

Currently, if the validate.yml file contains several ping statements

ping:
  _kwargs:
      destination: 8.8.8.8
      source: 192.168.1.1
  success:
      packet_loss: 0
  _mode: strict

ping:
  _kwargs:
      destination: 4.4.2.2
      source: 192.168.1.1
  success:
      packet_loss: 0
  _mode: strict

The end result will only consider one of the two:

            "ping": {
                "complies": true, 
                "extra": [], 
                "missing": [], 
                "present": {
                    "success": {
                        "complies": true, 
                        "nested": true
                    }
                }
            }

It would be useful to be able to test many different ping tasks for a given host. The same might apply to traceroute.

See details in the parent issue napalm-automation/napalm#297

This is a feature request.

It would be nice to have a method implemented across the various vendor drivers, such that light levels, ie, tx power and rx power, can be retrieved from the optics attached to the ports on the switch.

Is this something that is on the roadmap for napalm?