Code Monkey home page Code Monkey logo

nautobot-plugin-ssot-ipfabric's Introduction

The code in this repository has been migrated to the Nautobot SSoT Repository as an integration - read more about it in the SSoT Docs! As of August 2023 this repository has been FROZEN - all development / issues / discussions for this integration are in the Nautobot SSoT Repository going forward.

Nautobot SSoT IPFabric

An SSoT plugin providing a simple way to synchronize data between IPFabric and Nautobot. Ensure data stays consistent between the two platforms by leveraging DiffSync capabilities and allowing users to take full advantage of both platforms with up-to-date, synchronized data.

Version Matrix

Here is a compatibility matrix and the minimum versions required to run this plugin:

IP Fabric Python Nautobot ssot ssot-ipfabric python-ipfabric python-ipfabric-diagrams
4.4 3.7.1 1.1.0 1.0.1 0.10.0 0.11.0 1.2.7
5.0.1 3.7.1 1.2.0 1.0.1 1.0.0 5.0.11 5.0.2
6.0 3.7.1 1.4.0 1.2.0 2.0.0 6.0.9 6.0.2

Build Status

Branch Status
main Build Status
develop Build Status

Documentation

Documentation is hosted with Github Pages at Nautobot SSoT IP Fabric Documentation

Installation

To install the plugin from Pypi

pip install nautobot-ssot-ipfabric

To install the plugin manually from repository code

git clone [email protected]:nautobot/nautobot-plugin-ssot-ipfabric.git
cd nautobot-plugin-ssot-ipfabric
pip install .

For additional detailed instructions on how to install Nautobot Plugins, checkout the official documentation

Environment Variables

This plugin relies on user provided environment variables to interact with IP Fabric.

  • ipfabric_api_token - API Token for IP Fabric
  • ipfabric_host - IP Fabric URL
  • nautobot_host - Nautobot URL (This is used to generate url links for chatops)
  • ipfabric_ssl_verify- IP Fabric API SSL verification
  • ipfabric_timeout- IP Fabric API timeout

Example PLUGINS_CONFIG to be updated in nautobot_config.py after successful installation. The chatops configuration is optional, but if you'd like to have the ability to call the sync job through chatops, you will be required to configure it.

PLUGINS = ["nautobot_ssot", "nautobot_ssot_ipfabric"]

PLUGINS_CONFIG = {
    "nautobot_chatops": {
        "enable_slack": True,
        "slack_api_token": os.environ.get("SLACK_API_TOKEN"),
        "slack_signing_secret": os.environ.get("SLACK_SIGNING_SECRET"),
        "session_cache_timeout": 3600,
    },
    "nautobot_ssot_ipfabric": {
        "ipfabric_api_token": os.environ.get("IPFABRIC_API_TOKEN"),
        "ipfabric_host": os.environ.get("IPFABRIC_HOST"),
        "nautobot_host": os.environ.get("NAUTOBOT_HOST"),
        "ipfabric_ssl_verify": os.environ.get("IPFABRIC_SSL_VERIFY"),
        "ipfabric_timeout": os.environ.get("IPFABRIC_TIMEOUT"),
    },
    "nautobot_ssot": {"hide_example_jobs": True},
    "nautobot_chatops_ipfabric": {
        "IPFABRIC_API_TOKEN": os.environ.get("IPFABRIC_API_TOKEN"),
        "IPFABRIC_HOST": os.environ.get("IPFABRIC_HOST"),
    },
}

ChatOps

As part of the SSoT synchronization capabilities with IP Fabric, this plugin extends the Nautobot Plugin Chatops IPFabric by providing users with the ability to begin the sync job from a chatops command (Slack).

ssot-chatops-sync

Contributing

Pull requests are welcomed and automatically built and tested against multiple versions of Python and multiple version of Nautobot through Github Actions.

The project is packaged with a light development environment based on docker-compose to help with the local development of the project and to run the tests within Github Actions.

The project is following Network to Code software development guideline and is leveraging:

  • Black, Pylint, Bandit and pydocstyle for Python linting and formatting.
  • Django unit test to ensure the plugin is working properly.

Development Environment

The development environment can be used in 2 ways. First, with a local poetry environment if you wish to develop outside of Docker with the caveat of using external services provided by Docker for PostgresQL and Redis. Second, all services are spun up using Docker and a local mount so you can develop locally, but Nautobot is spun up within the Docker container.

Below is a quick start guide if you're already familiar with the development environment provided, but if you're not familiar, please read the Getting Started Guide.

Invoke

The PyInvoke library is used to provide some helper commands based on the environment. There are a few configuration parameters which can be passed to PyInvoke to override the default configuration:

  • nautobot_ver: the version of Nautobot to use as a base for any built docker containers (default: 1.1.4)
  • project_name: the default docker compose project name (default: nautobot_ssot_ipfabric)
  • python_ver: the version of Python to use as a base for any built docker containers (default: 3.7)
  • local: a boolean flag indicating if invoke tasks should be run on the host or inside the docker containers (default: False, commands will be run in docker containers)
  • compose_dir: the full path to a directory containing the project compose files
  • compose_files: a list of compose files applied in order (see Multiple Compose files for more information)

Using PyInvoke these configuration options can be overridden using several methods. Perhaps the simplest is simply setting an environment variable INVOKE_NAUTOBOT_SSOT_IPFABRIC_VARIABLE_NAME where VARIABLE_NAME is the variable you are trying to override. The only exception is compose_files, because it is a list it must be overridden in a yaml file. There is an example invoke.yml (invoke.example.yml) in this directory which can be used as a starting point.

Local Poetry Development Environment

  1. Copy development/creds.example.env to development/creds.env (This file will be ignored by Git and Docker)
  2. Uncomment the POSTGRES_HOST, REDIS_HOST, and NAUTOBOT_ROOT variables in development/creds.env
  3. Create an invoke.yml file with the following contents at the root of the repo (you can also cp invoke.example.yml invoke.yml and edit as necessary):
---
nautobot_ssot_ipfabric:
  local: true
  compose_files:
    - "docker-compose.requirements.yml"
  1. Run the following commands:
poetry shell
poetry install --extras nautobot
export $(cat development/dev.env | xargs)
export $(cat development/creds.env | xargs)
invoke start && sleep 5
nautobot-server migrate

If you want to develop on the latest develop branch of Nautobot, run the following command: poetry add --optional git+https://github.com/nautobot/nautobot@develop. After the @ symbol must match either a branch or a tag.

  1. You can now run nautobot-server commands as you would from the Nautobot documentation for example to start the development server:
nautobot-server runserver 0.0.0.0:8080 --insecure

Nautobot server can now be accessed at http://localhost:8080.

It is typically recommended to launch the Nautobot runserver command in a separate shell so you can keep developing and manage the webserver separately.

Docker Development Environment

This project is managed by Python Poetry and has a few requirements to setup your development environment:

  1. Install Poetry, see the Poetry Documentation for your operating system.
  2. Install Docker, see the Docker documentation for your operating system.

Once you have Poetry and Docker installed you can run the following commands to install all other development dependencies in an isolated python virtual environment:

poetry shell
poetry install
invoke start

Nautobot server can now be accessed at http://localhost:8080.

To either stop or destroy the development environment use the following options.

  • invoke stop - Stop the containers, but keep all underlying systems intact
  • invoke destroy - Stop and remove all containers, volumes, etc. (This results in data loss due to the volume being deleted)

CLI Helper Commands

The project is coming with a CLI helper based on invoke to help setup the development environment. The commands are listed below in 3 categories dev environment, utility and testing.

Each command can be executed with invoke <command>. Environment variables INVOKE_NAUTOBOT_SSOT_IPFABRIC_PYTHON_VER and INVOKE_NAUTOBOT_SSOT_IPFABRIC_NAUTOBOT_VER may be specified to override the default versions. Each command also has its own help invoke <command> --help

Docker dev environment

  build            Build all docker images.
  debug            Start Nautobot and its dependencies in debug mode.
  destroy          Destroy all containers and volumes.
  restart          Restart Nautobot and its dependencies.
  start            Start Nautobot and its dependencies in detached mode.
  stop             Stop Nautobot and its dependencies.

Utility

  cli              Launch a bash shell inside the running Nautobot container.
  create-user      Create a new user in django (default: admin), will prompt for password.
  makemigrations   Run Make Migration in Django.
  nbshell          Launch a nbshell session.
  shell-plus       Launch a shell_plus session, which uses iPython and automatically imports all models.

Testing

  bandit           Run bandit to validate basic static code security analysis.
  black            Run black to check that Python files adhere to its style standards.
  flake8           This will run flake8 for the specified name and Python version.
  pydocstyle       Run pydocstyle to validate docstring formatting adheres to NTC defined standards.
  pylint           Run pylint code analysis.
  tests            Run all tests for this plugin.
  unittest         Run Django unit tests for the plugin.

Project Documentation

Documentation is auto-generated with Sphinx, myst-paerser and sphinx-autoapi. The latest code in the main branch is providing the documentation with Github Pages. To build documentation locally as you are developing, perform the following commands.

sphinx-build -vvv -b html ./docs docs/public
cd docs/public
python -m http.server

Now you can access the documentation locally at http://localhost:8000/ or the IP of the server hosting the development environment.

Questions

For any questions or comments, please check the FAQ first and feel free to swing by the Network to Code slack channel (channel #networktocode). Sign up here

Screenshots

Main SSoT IP Fabric Dashboard Dashboard

Sync Details Dashboard

nautobot-plugin-ssot-ipfabric's People

Contributors

alhogan avatar armartirosyan avatar chadell avatar dependabot[bot] avatar fragmentedpacket avatar grelleum avatar h4ndzdatm0ld avatar jdrew82 avatar jjeff07 avatar pke11y avatar qduk avatar ubajze avatar whitej6 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nautobot-plugin-ssot-ipfabric's Issues

Remove site `iframe`

Environment

  • Python version:
  • Nautobot version:
  • nautobot-ssot-ipfabric version:

Expected Behavior

Observed Behavior

The site diagram iframe is not longer available on newer ipfabric versions

Steps to Reproduce

Add Site Model

Add Site model to sync site data from IPFabric to Nautobot

Add Device Type/Manufacturer

Make a call to part numbers in IP Fabric to map to vendor (manufacturer), model (device type), potentially platform? May require mapping? Serial number

Chatops: Start sync from Slack

Ability to use the chatops plugin to start syncing and reporting back status potentially? Raw dump of data we get from the job.

  • Even more stretch is dry run

Interfaces are always created when sync job runs

Interfaces always appear as created during diffsync reports. This should not be the case, once they are created, only updates should happen and diff sync should not be reporting on them, unless updated or deleted.

SSOT History Unknown filter field: overview

Environment

  • Python version: 3.9
  • Nautobot version: 1.5.10
  • nautobot-ssot version: 1.2.0
  • nautobot-ssot-ipfabric version: 2.0.0

Expected Behavior

Brings me to a filtered table for the history of a job and a specific action

Observed Behavior

URL: /plugins/ssot/logs/?overview=15cbe6d1-a0c3-4379-a17a-d7c2f2f4e16c&action=create

Invalid filters were specified:
overview
Unknown filter field

Steps to Reproduce

  1. Go to Plugins > SSOT History
  2. Click one of the numbered links in Create/Update/Delete/Failure/Error columns

Presentation Slide DEATH

Some sort of presentation slide to highlight how we structured and put together a winning plan from the start

ipfabric_timeout var should be cast as integer if coming from env var

Environment

  • Python version: 3.8.13
  • Nautobot version: 1.4.1
  • nautobot-ssot-ipfabric version: 0.10.0

Expected Behavior

Nautobot starts up when plugin is added with steps described in README

Observed Behavior

Nautobot worker crashes on startup with traceback below

ipfabric_worker_1 exited with code 1
nautobot_1  | Traceback (most recent call last):
nautobot_1  |   File "/usr/local/bin/nautobot-server", line 8, in <module>
nautobot_1  |     sys.exit(main())
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/nautobot/core/cli.py", line 54, in main
nautobot_1  |     run_app(
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 266, in run_app
nautobot_1  |     management.execute_from_command_line([runner_name, command] + command_args)
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
nautobot_1  |     utility.execute()
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
nautobot_1  |     django.setup()
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
nautobot_1  |     apps.populate(settings.INSTALLED_APPS)
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 122, in populate
nautobot_1  |     app_config.ready()
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/nautobot_ssot_ipfabric/__init__.py", line 37, in ready
nautobot_1  |     super().ready()
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/nautobot/extras/plugins/__init__.py", line 144, in ready
nautobot_1  |     jobs = import_object(f"{self.__module__}.{self.jobs}")
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/nautobot/extras/plugins/utils.py", line 45, in import_object
nautobot_1  |     spec.loader.exec_module(module)
nautobot_1  |   File "<frozen importlib._bootstrap_external>", line 843, in exec_module
nautobot_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/nautobot_ssot_ipfabric/jobs.py", line 77, in <module>
nautobot_1  |     CLIENT = IPFClient(
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/ipfabric/client.py", line 45, in __init__
nautobot_1  |     super().__init__(base_url, token, snapshot_id, username, password, **kwargs)
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/ipfabric/api.py", line 71, in __init__
nautobot_1  |     self.os_version = self.fetch_os_version()
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/ipfabric/api.py", line 97, in fetch_os_version
nautobot_1  |     res = self.get(url="os/version")
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1039, in get
nautobot_1  |     return self.request(
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 815, in request
nautobot_1  |     return self.send(request, auth=auth, follow_redirects=follow_redirects)
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 902, in send
nautobot_1  |     response = self._send_handling_auth(
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 930, in _send_handling_auth
nautobot_1  |     response = self._send_handling_redirects(
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 967, in _send_handling_redirects
nautobot_1  |     response = self._send_single_request(request)
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1003, in _send_single_request
nautobot_1  |     response = transport.handle_request(request)
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 218, in handle_request
nautobot_1  |     resp = self._pool.handle_request(req)
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 253, in handle_request
nautobot_1  |     raise exc
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 237, in handle_request
nautobot_1  |     response = connection.handle_request(request)
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 86, in handle_request
nautobot_1  |     raise exc
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 63, in handle_request
nautobot_1  |     stream = self._connect(request)
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 111, in _connect
nautobot_1  |     stream = self._network_backend.connect_tcp(**kwargs)
nautobot_1  |   File "/usr/local/lib/python3.8/site-packages/httpcore/backends/sync.py", line 86, in connect_tcp
nautobot_1  |     sock = socket.create_connection(
nautobot_1  |   File "/usr/local/lib/python3.8/socket.py", line 793, in create_connection
nautobot_1  |     sock.settimeout(timeout)
nautobot_1  | TypeError: an integer is required (got type str)

Steps to Reproduce

  1. Add nautobot-plugin-ssot-ipfabric to nautobot plugins as described in README
  2. Configure IPFABRIC_TIMEOUT environment variable to an integer (I used 15)
  3. Attempt to start Nautobot

To test if this was really the issue I wrapped the ipfabric_timeout value in the plugins config in the int() method and nautobot started as expected.

    "nautobot_ssot_ipfabric": {
        "ipfabric_api_token": os.environ.get("IPFABRIC_API_TOKEN"),
        "ipfabric_host": os.environ.get("IPFABRIC_HOST"),
        "nautobot_host": os.environ.get("NAUTOBOT_HOST"),
        "ipfabric_ssl_verify": os.environ.get("IPFABRIC_SSL_VERIFY"),
        "ipfabric_timeout": int(os.environ.get("IPFABRIC_TIMEOUT", 15)),
    },

Move topology ID from 'facility' field on site

As part of the topology rendering from IP Fabric, we display the a interactive image on sites which use the "facility" field to populate the necessary link.

The request is to not hijack this field for a plugin and use custom_field_data and call a simple .update("topology_id") to store this value on an individual site basis while running the sync job

Safe Delete Mode Disabled throws Error

Environment

  • Python version: 3.9
  • Nautobot version: 1.4.8
  • nautobot-ssot-ipfabric version: 1.0.0

Expected Behavior

SSOT errors when Safe Delete Mode is not enabled.

Observed Behavior

  File "/opt/nautobot/lib/python3.9/site-packages/nautobot_ssot/jobs/base.py", line 332, in run
    self.sync_data()
  File "/opt/nautobot/lib/python3.9/site-packages/nautobot_ssot_ipfabric/jobs.py", line 277, in sync_data
    dest.sync_from(ipfabric_source)
  File "/opt/nautobot/lib/python3.9/site-packages/diffsync/__init__.py", line 526, in sync_from
    self.sync_complete(source, diff, flags, syncer.base_logger)
  File "/opt/nautobot/lib/python3.9/site-packages/nautobot_ssot_ipfabric/diffsync/adapter_nautobot.py", line 72, in sync_complete
    nautobot_object.delete()
  File "/opt/nautobot/lib/python3.9/site-packages/django/db/models/base.py", line 960, in delete
    assert self.pk is not None, (
AssertionError: Interface object can't be deleted because its id attribute is set to None.

Steps to Reproduce

  1. Start new Sync
  2. Uncheck Safe Delete Mode

Default params from settings

Currently we have defaulted the following:

DEFAULT_DEVICE_ROLE = "Network Device"
DEFAULT_DEVICE_STATUS = "Active"

Allow for these values to be configurable through settings from the user, as there is no way to pull this from ipfabric

Job Logging

Job debug logging shows in output even if debug isn't checked. Logs show under 'default'

Finalize Documentation

We should try and get auto docs works to document API code
Document what gets synced over for each model, etc.

Sending none instead of empty string for interfaces

Interface objects populated from IPFabric seems to hit an error with device imports. When there is no description on the device, the plug-in should send an empty string for the interface, as per the second example below.

interface_obj.description = fields.get ("description")

interface_obj.description = fields.get ("description", โ€œโ€)

django.core.exceptions when using IP Fabric SSoT after upgrade to Nautobot v1.3.1

Environment

  • Python version: 3.8.10
  • Nautobot version: 1.3.1
  • nautobot-ssot-ipfabric version: 0.9.4

Expected Behavior

Normal completion at the end of the sync

Observed Behavior

An exception occurred: ValidationError: {'__all__': ['assigned_object_type and assigned_object_id must either both be null or both be non-null']}

Traceback (most recent call last): File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot/jobs/base.py", line 326, in run self.sync_data() File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/jobs.py", line 185, in sync_data dest.sync_from(ipfabric_source) File "/opt/nautobot/lib/python3.8/site-packages/diffsync/__init__.py", line 525, in sync_from result = syncer.perform_sync() File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 326, in perform_sync changed |= self.sync_diff_element(element) File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 390, in sync_diff_element changed |= self.sync_diff_element(child, parent_model=dst_model) File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 390, in sync_diff_element changed |= self.sync_diff_element(child, parent_model=dst_model) File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 366, in sync_diff_element changed, modified_model = self.sync_model(src_model=src_model, dst_model=dst_model, ids=ids, attrs=attrs) File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 415, in sync_model dst_model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs) File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/diffsync/diffsync_models.py", line 306, in create ip_address_obj = tonb_nbutils.create_ip( File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/utilities/nbutils.py", line 143, in create_ip tag_object(nautobot_object=ip_obj, custom_field="ssot-synced-from-ipfabric") File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/utilities/nbutils.py", line 253, in tag_object _tag_object(nautobot_object) File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/utilities/nbutils.py", line 251, in _tag_object nautobot_object.validated_save() File "/opt/nautobot/lib/python3.8/site-packages/nautobot/core/models/__init__.py", line 51, in validated_save self.full_clean() File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/base.py", line 1251, in full_clean raise ValidationError(errors) django.core.exceptions.ValidationError: {'__all__': ['assigned_object_type and assigned_object_id must either both be null or both be non-null']}

Steps to Reproduce

  1. Start the Sync between IP Fabric and Nautobot
  2. It works up to the diff and a few steps afterward:
    image
  3. Then it fails

ssot-sync-to-nautobot command failing

Environment

  • Python version:
  • Nautobot version: 1.5.7
  • ipfabric version: Latest
nautobot==1.5.7
nautobot-capacity-metrics==2.0.0
nautobot-chatops==1.10.0
nautobot-chatops-ipfabric==3.0.1
nautobot-ssot==1.2.0
nautobot-ssot-ipfabric==2.0.0

Expected Behavior

To start SSOT

Need to debug more but ran out of time.

Observed Behavior

*Sync failed. Here is the link to your job: https://nautobot-demo.ipf.cx/plugins/ssot/history/330fa397-f1d8-4068-af93-a1752549c0e6/*

  File "/opt/nautobot/lib/python3.9/site-packages/nautobot_ssot/jobs/base.py", line 332, in run
    self.sync_data()
  File "/opt/nautobot/lib/python3.9/site-packages/nautobot_ssot_ipfabric/jobs.py", line 225, in sync_data
    self.client.snapshot_id = self.kwargs["snapshot"]
KeyError: 'snapshot'

Steps to Reproduce

  1. Run /ipfabric ssot-sync-to-nautobot True True True

Utilities

Before testing any utilities, switch them over to get_or_create as appropriate

Device Role

Environment

  • nautobot version: 1.5.x
  • nautobot-ssot-ipfabric version: 2.0.0
  • ipfabric version: 6.x

Summary

IP Fabric classifies devices into different "Types" (column devType) which can be used as a Nautobot Device Role. Currently all devices are sync'ed using the default:

DEFAULT_DEVICE_ROLE = CONFIG.get("default_device_role", "Network Device")
DEFAULT_DEVICE_ROLE_COLOR = CONFIG.get("default_device_role_color", "ff0000")

Proposed Functionality

  • Have an option in nautobot_config.py that enables syncing the IP Fabric Device Type to Nautobot Device Role.
    • Device Types are mainly static and rarely change. Usually new types are added and normally none are removed.
    • Perhaps we just create new roles if it does not exist, and do not delete any roles.
  • IPF Device Type is found at:
    • API: FQDN/api/v#.#/tables/inventory/devices
    • UI: FQDN/inventory/devices
    • Column: devType
> devices = ipf.inventory.devices.all()
> dev_type = {d['devType'] for d in devices}
> print(dev_type)
{'switch', 'fw', 'waas', 'wlc', 'lb', 'l3switch', 'router'}

Open Questions

  • Default to Enabled or Disabled sync'ing type to role?
  • How do we map Device Role to a color?
    • Manually in UI after creation?
    • Allow for a config setting?
    • Create static example?

Use Case

  • "Role of Network Device is too generic."

Current Device Types

List of Device Types in IP Fabric v6.2.1 is shown below. Not all are used for Network Device Inventory purposes. For instance host is used for IPF discovered hosts.

[
  'aciLeaf',
  'aciSpine',
  'ap',
  'apic',
  'cloudInstance',
  'cloudInternetGw',
  'cloudLoadBalancer',
  'cloudNatGw',
  'cloudRouter',
  'cloudTransitHub',
  'cloudVpnGw',
  'fex',
  'fw',
  'host',
  'l3switch',
  'lb',
  'nx7000',
  'phone',
  'router',
  'securityManagement',
  'switch',
  'unknown',
  'vgw',
  'waas',
  'wlc',
]

Module tries to establish a connection to IPFabric when importing the module

Environment

  • Python version: 3.9
  • Nautobot version: 1.3.9
  • nautobot-ssot-ipfabric version: v0.10.0

Expected Behavior

A connection to the IPFabric server should be established only when triggered an action and not on module import.

Observed Behavior

When importing a module, this piece of code is executed:

try:
CLIENT = IPFClient(
IPFABRIC_HOST,
token=IPFABRIC_API_TOKEN,
verify=IPFABRIC_SSL_VERIFY,
timeout=IPFABRIC_TIMEOUT,
)
snapshots = CLIENT.get_snapshots()
except ConnectError:
CLIENT = None
snapshots = []

If an API token or a host is not set, we get an error:

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/nautobot/core/cli.py", line 54, in main
    run_app(
  File "/usr/local/lib/python3.9/site-packages/nautobot/core/runner/runner.py", line 266, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
    django.setup()
  File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/usr/local/lib/python3.9/site-packages/nautobot_ssot_ipfabric/__init__.py", line 37, in ready
    super().ready()
  File "/usr/local/lib/python3.9/site-packages/nautobot/extras/plugins/__init__.py", line 143, in ready
    jobs = import_object(f"{self.__module__}.{self.jobs}")
  File "/usr/local/lib/python3.9/site-packages/nautobot/extras/plugins/utils.py", line 45, in import_object
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/nautobot_ssot_ipfabric/jobs.py", line 77, in <module>
    CLIENT = IPFClient(
  File "/usr/local/lib/python3.9/site-packages/ipfabric/client.py", line 45, in __init__
    super().__init__(base_url, token, snapshot_id, username, password, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/ipfabric/api.py", line 64, in __init__
    raise RuntimeError("IP Fabric Token or Username/Password not provided.")
RuntimeError: IP Fabric Token or Username/Password not provided.

I think we shouldn't add any piece of code that establishes a connection outside a function or a class in the python file, as this is executed when importing the module.

Steps to Reproduce

  1. Install the plugin version 0.10.0.
  2. Update the Nautobot configuration file to enable the plugin.
  3. Set the host and the token configuration to None.
  4. Start Nautobot.

Devices with Serial Number > 50 characters cause sync to fail

Environment

  • Python version: 3.8.10
  • Nautobot version: naut (v1.2.8)
  • nautobot-ssot-ipfabric version: 0.9.4

Expected Behavior

When performing the sync between IP Fabric and Nautobot, I have an error, and the sync fails:
An exception occurred: DataError: value too long for type character varying(50)

Observed Behavior

Sync failure. Doing a few tests, I've added in a recent snapshot Azure devices, which have a very long SN. By removing those devices, I can sync without any problem. It seems that the limit of 50 is too small and causes this issue. Here is a couple of example of the SN we have for Azure devices:

Hostname Site Unique serial number
VNetGw.Express.WestEurope AZURE /subscriptions/7a329198-ceb1-4e25-xxxx-e98855c123456/resourceGroups/LAB-Static/providers/Microsoft.Network/virtualNetworkGateways/VNetGw.Express.WestEurope
nat-Gateway AZURE /subscriptions/7a329198-ceb1-4e25-xxxx-e98855c123456/resourceGroups/LAB-Static/providers/Microsoft.Network/natGateways/nat-Gateway

Here are the logs of the error:

Traceback (most recent call last):
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 573, in get_or_create
    return self.get(**kwargs), False
  File "/opt/nautobot/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 429, in get
    raise self.model.DoesNotExist(
nautobot.dcim.models.devices.Device.DoesNotExist: Device matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.StringDataRightTruncation: value too long for type character varying(50)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot/jobs/base.py", line 326, in run
    self.sync_data()
  File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/jobs.py", line 185, in sync_data
    dest.sync_from(ipfabric_source)
  File "/opt/nautobot/lib/python3.8/site-packages/diffsync/__init__.py", line 525, in sync_from
    result = syncer.perform_sync()
  File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 326, in perform_sync
    changed |= self.sync_diff_element(element)
  File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 390, in sync_diff_element
    changed |= self.sync_diff_element(child, parent_model=dst_model)
  File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 366, in sync_diff_element
    changed, modified_model = self.sync_model(src_model=src_model, dst_model=dst_model, ids=ids, attrs=attrs)
  File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 415, in sync_model
    dst_model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs)
  File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/diffsync/diffsync_models.py", line 191, in create
    new_device, _ = NautobotDevice.objects.get_or_create(
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 576, in get_or_create
    return self._create_object_from_params(kwargs, params)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 610, in _create_object_from_params
    obj = self.create(**params)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 447, in create
    obj.save(force_insert=True, using=self.db)
  File "/opt/nautobot/lib/python3.8/site-packages/nautobot/dcim/models/devices.py", line 735, in save
    super().save(*args, **kwargs)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/base.py", line 753, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/base.py", line 790, in save_base
    updated = self._save_table(
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/base.py", line 895, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/base.py", line 933, in _do_insert
    return manager._insert(
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 1254, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
    cursor.execute(sql, params)
  File "/opt/nautobot/lib/python3.8/site-packages/cacheops/transaction.py", line 93, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/nautobot/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.DataError: value too long for type character varying(50)

Steps to Reproduce

  1. add Azure devices to IP Fabric, or devices where SN is longer than 50 characters
  2. run the SSoT from Nautobot (not in Dry Run)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.