arista-netdevops-community / anta Goto Github PK

What do you call an ant with frogs legs?

License: Apache License 2.0

Python 99.83% Dockerfile 0.15% Jinja 0.01% Shell 0.02%

test-automation arista arista-networks arista-automation network-automation pytest network-testing network-validation network-verification nrfu

anta's Introduction

Arista Network Test Automation (ANTA) Framework

Code
License
GitHub
PyPi

ANTA is Python framework that automates tests for Arista devices.

ANTA provides a set of tests to validate the state of your network
ANTA can be used to:
- Automate NRFU (Network Ready For Use) test on a preproduction network
- Automate tests on a live network (periodically or on demand)
ANTA can be used with:
- As a Python library in your own application
- The ANTA CLI

Install ANTA library

The library will NOT install the necessary dependencies for the CLI.

# Install ANTA as a library
pip install anta

Install ANTA CLI

If you plan to use ANTA only as a CLI tool you can use pipx to install it. pipx is a tool to install and run python applications in isolated environments. Refer to pipx instructions to install on your system. pipx installs ANTA in an isolated python environment and makes it available globally.

This is not recommended if you plan to contribute to ANTA

# Install ANTA CLI with pipx
$ pipx install anta[cli]

# Run ANTA CLI
$ anta --help
Usage: anta [OPTIONS] COMMAND [ARGS]...

  Arista Network Test Automation (ANTA) CLI

Options:
  --version                       Show the version and exit.
  --log-file FILE                 Send the logs to a file. If logging level is
                                  DEBUG, only INFO or higher will be sent to
                                  stdout.  [env var: ANTA_LOG_FILE]
  -l, --log-level [CRITICAL|ERROR|WARNING|INFO|DEBUG]
                                  ANTA logging level  [env var:
                                  ANTA_LOG_LEVEL; default: INFO]
  --help                          Show this message and exit.

Commands:
  check  Commands to validate configuration files
  debug  Commands to execute EOS commands on remote devices
  exec   Commands to execute various scripts on EOS devices
  get    Commands to get information from or generate inventories
  nrfu   Run ANTA tests on devices

You can also still choose to install it with directly with pip:

$ pip install anta[cli]

Documentation

The documentation is published on ANTA package website. Also, a demo repository is available to facilitate your journey with ANTA.

Contribution guide

Contributions are welcome. Please refer to the contribution guide

Credits

Thank you to Jeremy Schulman for aio-eapi.

Thank you to Angélique Phillipps, Colin MacGiollaEáin, Khelil Sator, Matthieu Tache, Onur Gashi, Paul Lavelle, Guillaume Mulocher and Thomas Grimonet for their contributions and guidances.

anta's People

Contributors

Stargazers

Watchers

anta's Issues

collect sheduled show tech

#!/usr/bin/env python3

## This script collects all the tech-support files stored on Arista switches flash and copies them locally

import ssl
import paramiko
from jsonrpclib import Server
from scp import SCPClient
from time import strftime, gmtime
from argparse import ArgumentParser
from getpass import getpass
import os
from sys import exit
from socket import setdefaulttimeout

port = 22
ssl._create_default_https_context = ssl._create_unverified_context
date = strftime("%d %b %Y %H:%M:%S", gmtime())

def device_directories (device, root_dir):
    cwd = os.getcwd()
    show_tech_directory = cwd + '/' + root_dir
    device_directory = show_tech_directory + '/' + device
    for directory in [show_tech_directory, device_directory]:
        if not os.path.exists(directory):
            os.makedirs(directory)
    result = show_tech_directory, device_directory
    return result

def createSSHClient (device, port, username, password):
    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    client.connect(device, port, username, password)
    return client

def main():
    parser = ArgumentParser(
        description='Collect all the tech-support files'
        )
    parser.add_argument(
        '-i',
        help='Text file containing a list of switches',
        dest='file',
        required=True
        )
    parser.add_argument(
        '-u',
        help='Devices username',
        dest='username',
        required=True
        )
    parser.add_argument(
        '-o',
        help='Output directory',
        dest='output_directory',
        required=True
        )
    args = parser.parse_args()
    args.password = getpass(prompt='Device password: ')

    try:
        with open(args.file, 'r') as file:
            devices = file.readlines()
    except:
        print('Error opening ' + args.file)
        exit(1)

    for i,device in enumerate(devices):
        devices[i] = device.strip()

    # Deletes unreachable devices from devices list

    unreachable = []
    
    print('Connecting to devices .... please be patient ... ')
    
    for device in devices:
        try:
            setdefaulttimeout(5)
            url = 'https://%s:%s@%s/command-api' %(args.username, args.password, device)
            switch = Server(url)
            response = switch.runCmds(1, ['enable'])
        except:
            unreachable.append(device)
    
    for item in unreachable:
        devices.remove(item)
        print("Can not connect to device " + item)

    # Collects all the tech-support files stored on Arista switches flash and copies them locally

    for device in devices:
        url = "https://" + args.username + ":" + args.password + "@" + device + "/command-api"
        try:
            ## Create one zip file named all_files.zip on the device with the all the show tech-support files in it
            cmds=['enable','bash timeout 30 zip /mnt/flash/schedule/all_files.zip /mnt/flash/schedule/tech-support/*']
            switch = Server(url)
            switch.runCmds(1,cmds, 'text')
            ## Connect to the device using SSH
            ssh = createSSHClient(device, port, args.username, args.password)
            ## Create directories 
            output_dir = device_directories (device, args.output_directory)
            ## Get the zipped file all_files.zip using SCP and save it locally
            my_path = output_dir[1] + '/' + date + '.zip'
            ssh = createSSHClient(device, port, args.username, args.password)
            scp = SCPClient(ssh.get_transport())
            scp.get("/mnt/flash/schedule/all_files.zip",local_path = my_path)
            scp.close()
            ## Delete the created zip file on the device
            cmds=['enable','bash timeout 30 rm /mnt/flash/schedule/all_files.zip']
            switch.runCmds(1,cmds, 'text')
        except:
            print('You are unlucky today! ' + device + ' does not like this script')

if __name__ == '__main__':
    main()
    ```

add OSPF test to check if all neighbors are FULL

sh ip ospf neighbor | exclude FULL

Adding devices using network ranges

Hi,

Wondering if we can have network ranges specified on the devices.txt rather than using unique ip?

Thanks
Hari

Implement async test functions and inventory

One of the main concerns for usability of the framework is performance and the fact that tests are run in sequence for each inventory device. Most of the tests are RPC calls over the network and creating connections that cause delays for am inventory of hundreds of devices. Would be nice to have an async method where all the tests (or just a subset) can run in parallel and waiting time could be used efficiently.
Also async inventory could initiate connection to all inventory hosts in parallel.

Implement "AntaTest" and "AntaAsyncTest" as protocols

Sharing some ideas that could improve the framework... Need a way to enforce the format of the test functions to ensure all functions are written correctly (return boolean, etc) and the user can introduce more functions on runtime. Both normal/sequential and async modes could be supported, running the async tests in parallel on a "AntaAsyncTestRunner" and the other functions in a "AntaTestRunner". A test manager could handle the runners and dispatch the tests to each of them delegating the execution. So having the tests modelled in a flexible way could be the first step towards a more generic execution framework that can be customized/extended by the user on runtime.

rename the repo to arista-network-test-automation to match the package name

req an update of setup file

add markdown llint to the CI

This will require updates in the md files

generate tests files to test devices configured with Arista AVD

Publish to pypi the tests library

and remove from this repo the scripts not related to the tests library

Implement "AntaTest" and "AntaAsyncTest" as protocols

suffix filenames (reports, collected show commands) with date and time

Add support for custom defined tests

Description

In order to allow users to build their own test collection, it would be nice to add option to support catalog pointing to custom libraries.

Today, we only look at anta.tests package but we could extend YAML format to support both anta.tests and a custom package with the following format:

<package.name>:
      - <test function>:
           <function options>

An example would be:

anta.tests.software:
  - verify_eos_version: # Verifies the device is running one of the allowed EOS version.
      versions: # List of allowed EOS versions.
        - 4.25.4M
        - 4.26.1F
inetsix.tests.mydesign:
  - check_topology:

Potential impacts:

anta.loader.parse_catalog

add the failure reason in the output file

e.g:

13.01. verify_spanning_tree_blocked_ports

leaf-01: Fail

The following ports are blocked by STP:

Ethernet3
Ethernet4
Ethernet5

Add test to verify running/startup config vs golden config

There is already a verify_running_config_diffs. It would be nice to introduce a verify_config_compliance to check the running or startup config matches an expected or golden config.

feat: add a tool to convert AVD inventory to ANTA inventory

Add a CLI entrypoint to convert a YAML ansible inventory from AVD to ANTA

anta get from-avd <path to yaml file> --container-name fabric

merge BGP tests (count and state)

making the count (number arg) optionnal

Improve enable password handling

Currently, enable password is passed as argument when starting ANTA, stored as an attribute in the InventoryDevice model, and used in the test code to enter in privileged mode when needed.
This issue is to explore if we could move this logic in the AntaInventory class, add an attribute (e.g. device.privileged) in the InventoryDevice model to define if the device session has been established with privileged mode (similar to established and is_online).
This way, if a test needs privileged mode, we could have a decorator to check the device.privileged attribute and remove all the logic in coroutines that uses the enable password.

create a test_catalog to map tests name and tests id

create a test_catalog variable somewhere to map tests name and tests id

so we dont have anymore to define and use test id in this file https://github.com/arista-netdevops-community/network-test-automation/blob/master/examples/tests.yaml so it will be easier
also this will enforce the same test id for a given test name so it will be more readable
This will help if the futur if we want to compare 2 tests report

the tests id can have a format like xx.xx where the first 2 digits will be the category (BGP, BFD, ...) and the last 2 digits will be to identify the test itself.

CI workflow: swap linting and compiling

enhanced bfd tests to support all VRFs

collect tech-support files

Is it possible when using ANTA to collect the tech-support filles from a switch in the /mnt/flash/schedule/tech-support folder to only collect the latest file and not all the files?

Feat: Replace decorators for skipping tests with AntaTestFilter

Today we are using decorators on tests that should be skipped. We should leverage AntaTestFilter class instead for more flexibility.

Ex:

class VerifyBGPIPv4UnicastCount(AntaTest):
    test_filters = [SkipBGPFilter(family="ipv4")]

class SkipBGPFilter(AntaTestFilter):
    def __init__(self, family: str) -> None:
        """
        Class initialization with the BGP address family to verify.
        """
        self.family = family

add a test to check storm control drops

add a script that clear bgp evpn host-flap

enable password

remove enable password from code to test devices (check-devices.py and tests_eos/functions.py)?

Scripts can't run due to 'jsonrpclib' incompatibility

I am having an issue running the scripts, as jsonrpclib doesn't work with Python3:

testvm:~/network_tests_automation# ./check-devices-reachability.py --help
Traceback (most recent call last):
  File "./check-devices-reachability.py", line 5, in <module>
    from jsonrpclib import Server
ModuleNotFoundError: No module named 'jsonrpclib'

I've installed PIP dependencies as instructed, and even added 'pip install jsonrpclib' separately:

test-vm:~/network_tests_automation# /usr/bin/env python3 -V
Python 3.6.9

testvm:~# pip list | grep jsonrpc
jsonrpclib (0.1.7)
jsonrpclib-pelix (0.4.3.2)

There seems to be a known issue with this library and Python3: https://stackoverflow.com/questions/38664102/python-jsonrpclib-not-working-after-upgrade-to-python-3-5-2

Any ideas how to get around this?

update the script check-devices.py to create output directories if it/they doesnt exist

Feat: Support more than one template per test

Add support for more than one template per test.

Example:
template = AntaTestTemplate(template="test aaa group tacacs+ {user} {pass}", ofmt="text")

Ascii Art introduced is breaking help outuputs and shell completion

Help message

nta exec

                          ########                  #
                      #################            #
                   ######################         #
                  #########################      #
                ############################
               ##############################
               ###############################
              ###############################
              ##############################
                              #    ########   #
                 ##        ###        ####   ##
                                      ###   ###
                                    ####   ###
               ####          ##########   ####
               #######################   ####
                 ####################   ####
                  ##################  ####
                    ############      ##
                       ########        ###
                      #########        #####
                    ############      ######
                   ########      #########
                     #####       ########
                       ###       #########
                      ######    ############
                     #######################
                     #   #   ###  #   #   ##
                     ########################
                      ##     ##   ##     ##
                      https://www.anta.ninja


Usage: anta exec [OPTIONS] COMMAND [ARGS]...

  Execute commands to inventory devices

Options:
  --help  Show this message and exit.

Commands:
  clear-counters        Clear counter statistics on EOS devices
  collect-tech-support  Collect scheduled tech-support from eos devices.
  snapshot              Collect commands output from devices in inventory

Autocompletion

eval "$(_ANTA_COMPLETE=zsh_source anta)" > /dev/null

(eval):31: no such file or directory: https://www.anta.ninja

Add attributes 'label' and 'description' with function decorator

Hello,

Currently the only precision about the tests is the name of the function itself. It is therefore necessary to read the code of the function to understand how it works and to have more details.

An idea would be to add to each test function a python decorator with two attributes: label and description.

Example with this sample piece of code:

def decorator(**kwargs):
    def wrapper(function):
        for k, v in kwargs.items():
            setattr(function, k, v)
        return function
    return wrapper

@decorator(label='EOS version', description='Verifies the device is running one of the allowed EOS version')
def verify_eos_version():
    pass

print(f'label: {verify_eos_version.label}\ndescription: {verify_eos_version.description}')

>>> label: EOS version
>>> description: Verifies the device is running one of the allowed EOS version

Now, we can access easily this information, without having to add two fields in a .yaml file, for example.
Thank you very much.

enhanced BGP count (vrf aware)

count established neighbors using the arg VRF (all, default, vrf1, ...)

add test to check traffic distribution accross interfaces member (port channel)

sh port-channel load-balance

Map tags to inventory devices and test catalog

Increase flexibility by using tags on inventory devices and the test catalog, so that catalog tests can run on relevant devices only. For instance, in s typical EVPN scenario there.might be some tests that should run only in spines and some other tests only on leaf switches, having also some tests that run on all devices. Using tags or filters on tests and inventory devices could achieve this easily and add flexibility to the way tests execute. For inventory networks/ranges, tags would apply to the whole range/network...

compare 2 tests report

workflow to:

run tests
apply changes
run the same tests on the same devices
compare the 2 tests report and display the diffs

Bug: Complex commands from logging tests are sometimes failing

Intermittent errors when running some of the logging test cases that have complex commands.

ERROR    Command failed on NW-CORE: CLI command 2 of 2 'show logging | awk '/Trap logging/,/Sequence       models.py:239
                    numbers/{if(/Sequence numbers/) exit; print}'' failed: internal error

Raise on error in the report part for CI

split the module `tests` of the package `anta` into multiples modules

The module is too large (>1600 lines).
we can split it into diff modules:

routing
hardware
interfaces
...
This will require an update of the check-devices.py

Bump aio-eapi to v0.6.0

The minimal Python version is now 3.10 with aio-eapi v0.6.0

Feat: TestResult

Add category and description to TestResult

add a test to check log content (search for keywords in logs)

use the command show logging informational last 7 days in text format
and search for keywords (variables used in function arguments) in the output like as example TRANSCEIVER_REMOVED or EVPN-3-BLACKLISTED_DUPLICATE_MAC or UPDOWN
and return True if a keyword is NOT in the output

keywords = ["EVPN-3-BLACKLISTED_DUPLICATE_MAC", "TRANSCEIVER_REMOVED", "UPDOWN"]

def verify_keyword_not_in_logs(device, keywords):
   response = device.runCmds(1, ["show logging informational last 7 days"], 'text')
   if response[0]['output'] != "":
      data = response[0]['output'].split('\n')
      for keyword in keywords:
        for line in data:
          if keyword in line:
            return False
   return True

fix repo badges links

[Bug] - check-device.py fails with --by-host option

The --by-host knob is broken and returns a trace:

$ check-devices.py -i anta-inventory.yml -c tests-bases.yml -u xxx -p xxx --by-host --table

Traceback (most recent call last):
  File "/home/tom/.pyenv/versions/arista-avd-3.9/bin/check-devices.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/tom/Projects/arista/network-test-automation/scripts/check-devices.py", line 312, in <module>
    reporter.report_summary_hosts(
  File "/home/tom/Projects/arista/network-test-automation/anta/reporter/__init__.py", line 251, in report_summary_hosts
    for host_read in result_manager.get_hosts():
  File "/home/tom/Projects/arista/network-test-automation/anta/result_manager/__init__.py", line 197, in get_hosts
    if str(testcase.host) not in result_list:
AttributeError: 'TestResult' object has no attribute 'host'

This option was working in v0.3.0 and is a regression part of one of these PRs:

85dca38 (HEAD -> master, upstream/master, origin/master, origin/HEAD) Doc: Update test pattern in documentation (#126)
630bff2 feat(asyncio): update scripts (#127)
6d50ac7 feat(anta.inventory): use aioeapi to handle a device session (#122)
5b16412 Feat: add anta_test decorator (#123)
cbc6091 feat(anta.loader)!: Add support for path to custom tests (#119)

add BGP tests to check in/out queues are empty

Doc: Logging tests documentation is showing Security tests

Wrong tests showed (Security) in the Logging tests section.

https://www.anta.ninja/api/tests.logging/

Fixed in #205

Make Device class a parameter

to be able to give other Device Classes
requires an interface for what we expect from a device

Fix: anta nrfu is crashing for BGP tests

Errors:

/home/cbaillar/git_projects/anta/anta/decorators.py:78: RuntimeWarning: coroutine 'AntaTest.collect' was never awaited anta_test.collect() RuntimeWarning: Enable tracemalloc to get the object allocation traceback [13:12:56] ERROR Error when running tests: TypeError: argument of type 'NoneType' is not iterable

BGP check_bgp_family_enable decorator returns None and anta_test.collect() is not awaited.

anta exec snapshot -c .personal/snapshot.yml

Snapshot file

json_format:
    - show version
text_format:
    - show lldp neighbors

Error

ERROR    Could not collect commands on device spine01
DEBUG    Exception raised for device spine01 - KeyError: 'data'
DEBUG    Traceback (most recent call last):
          File "/Users/tgrimonet/Projects/arista-community/\
anta/anta/cli/exec/utils.py", line 69, in collect
            result = await dev.session.cli(
          File "/Users/tgrimonet/.pyenv/versions/3.9.9/envs/\
anta/lib/python3.9/site-packages/aioeapi/device.py",\
 line 183, in cli
            res = await self.jsonrpc_exec(jsonrpc)
          File "/Users/tgrimonet/.pyenv/versions/3.9.9/envs/\
anta/lib/python3.9/site-packages/aioeapi/device.py", \
line 250, in jsonrpc_exec
            cmd_data = err_data["data"]
        KeyError: 'data'

Information

Python version: 3.9.9
Anta version: 0.5.0 and devel