nautobot / nautobot-app-nornir Goto Github PK

Update FAQ

Expose as each nornir task as a job that is allowlisted.

Environment

• Python version: N/A
• Nautobot version: 2.x
• nautobot-plugin-nornir version: 2.x

Expected Behavior

nautobot-app-nornir should be working OK when there's a Secret of type "HTTP(S)" in a Device's SecretsGroup.

Observed Behavior

An exception is raised when executing nautobot-app-nornir (line 22 in _get_secret_value) and there's a Secret of type "HTTP(S)" in the device's SecretsGroup.

{'exc_type': 'NornirNautobotException', 'exc_module': 'nornir_nautobot.exceptions', 'exc_message': ["`E3001:` General Exception handler, original error message ```type object 'SecretsGroupAccessTypeChoices' has no attribute 'TYPE_HTTP(S)'```"]}

Steps to Reproduce

1. Create a Secret.
2. Create a SecretGroup that includes the Secret with assigned type "HTTP(S)".
3. Assign the SecretGroup to a Device.
4. Run BackupJob for the Device.

Nornir supports transformers for a use case similar to credentials, perhaps this should be the future?

Enable get_group_creds within the inventory.

Update readme

Environment

• Python version: latest
• Nautobot version: latest
• nautobot-plugin-nornir version: latest

Proposed Functionality

Pre-emptivly check, and allow sane error

    access_type_str = device_obj.get_config_context()["nautobot_plugin_nornir"]["secret_access_type"].upper()
KeyError: 'nautobot_plugin_nornir'

Use Case

More clear guidance to the user.

Environment

• Python version: 3.7
• Nautobot version: 11.1.6
• nautobot-plugin-nornir version: 0.9.7

Proposed Functionality

Netmiko and Napalm connection plugins for Nornir both support some additional parameters via the extras section of the connection plugin.
The proposal is to extend the Nornir Inventory to allow uses to specific a list of extras parameters and pass them to Nornir.
See below an example of what the configuration could look like

PLUGINS_CONFIG = {
  "nautobot_plugin_nornir": {
    "napalm_extras": {
         .. 
     },
    "netmiko_extras": {
      "global_delay_factor": 5,
      "banner_timeout":  40, 
      "conn_timeout": 30,
      },
    },
  }

Use Case

Allow users to customize Netmiko and Napalm behavior. One of the primary use case is to adjust some timers and timeout conn_timeout etc ..

• Enable Upstream Testing
• Update workflows/ci.yml with new standards to not duplicate jobs, etc.

Follow similar pattern as to: nautobot/nautobot-app-chatops#171

Environment

• Python version: 3.9.19
• Nautobot version: 2.2.3
• nautobot-plugin-nornir version: 2.0.1
• golden config version: 2.0.5

Expected Behavior

nornir tasks run for each device involved in the intended config job

Observed Behavior

no nornir task runs for any device in golden config jobs

Steps to Reproduce

1. Use "use_config_context": {"secrets": True}, in the nautobot_plugin_nornir PLUGIN_CONFIG

"nautobot_plugin_nornir": {
    "use_config_context": {"secrets": True},
    "nornir_settings": {
        "credentials": "nautobot_plugin_nornir.plugins.credentials.nautobot_secrets.CredentialsNautobotSecrets",
    }
},

2. Run the Intended Config job in the Golden Config app

left: job log when running nautobot-plugin-nornir 2.0.0
right: job log when running nautobot-plugin-nornir 2.0.1

slack thread for extra context https://networktocode.slack.com/archives/C01NWPK6WHL/p1716504848369429?thread_ts=1716491198.677389&cid=C01NWPK6WHL

Environment

• Python version: 3.6
• Nautobot version: 1.1.3
• nautobot-plugin-nornir version: 0.9.4

Steps to Reproduce

1. Install Golden Config Plugin (https://github.com/nautobot/nautobot-plugin-golden-config/blob/develop/docs/installation.md)
2. Only setup required Golden Config Plugin settings within nautobot_config.py

  nautobot_golden_config:
    enable_intended: true

3. Run nautobot-server migrate

Expected Behavior

That the default settings would be inherited and not crash while attempting to apply the migrations

Observed Behavior

   Traceback (most recent call last):
      File "/appserver/nautobot/bin/nautobot-server", line 8, in <module>
        sys.exit(main())
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot/core/cli.py", line 62, in main
        initializer=_configure_settings,  # Called after defaults
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot/core/runner/runner.py", line 266, in run_app
        management.execute_from_command_line([runner_name, command] + command_args)
      File "/appserver/nautobot/lib64/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
        utility.execute()
      File "/appserver/nautobot/lib64/python3.6/site-packages/django/core/management/__init__.py", line 377, in execute
        django.setup()
      File "/appserver/nautobot/lib64/python3.6/site-packages/django/__init__.py", line 24, in setup
        apps.populate(settings.INSTALLED_APPS)
      File "/appserver/nautobot/lib64/python3.6/site-packages/django/apps/registry.py", line 122, in populate
        app_config.ready()
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot/extras/plugins/__init__.py", line 94, in ready
        jobs = import_object(f"{self.__module__}.{self.jobs}")
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot/extras/plugins/utils.py", line 44, in import_object
        spec.loader.exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 678, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot_golden_config/jobs.py", line 13, in <module>
        from nautobot_golden_config.nornir_plays.config_intended import config_intended
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot_golden_config/nornir_plays/config_intended.py", line 17, in <module>
        from nautobot_plugin_nornir.constants import NORNIR_SETTINGS
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot_plugin_nornir/constants.py", line 11, in <module>
        PLUGIN_CFG = settings.PLUGINS_CONFIG["nautobot_plugin_nornir"]
    KeyError: 'nautobot_plugin_nornir'

And then if I set nautobot_plugin_nornir as blank dictionary, I get the following:

    Traceback (most recent call last):
      File "/appserver/nautobot/bin/nautobot-server", line 8, in <module>
        sys.exit(main())
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot/core/cli.py", line 62, in main
        initializer=_configure_settings,  # Called after defaults
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot/core/runner/runner.py", line 266, in run_app
        management.execute_from_command_line([runner_name, command] + command_args)
      File "/appserver/nautobot/lib64/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
        utility.execute()
      File "/appserver/nautobot/lib64/python3.6/site-packages/django/core/management/__init__.py", line 377, in execute
        django.setup()
      File "/appserver/nautobot/lib64/python3.6/site-packages/django/__init__.py", line 24, in setup
        apps.populate(settings.INSTALLED_APPS)
      File "/appserver/nautobot/lib64/python3.6/site-packages/django/apps/registry.py", line 122, in populate
        app_config.ready()
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot/extras/plugins/__init__.py", line 94, in ready
        jobs = import_object(f"{self.__module__}.{self.jobs}")
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot/extras/plugins/utils.py", line 44, in import_object
        spec.loader.exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 678, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot_golden_config/jobs.py", line 13, in <module>
        from nautobot_golden_config.nornir_plays.config_intended import config_intended
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot_golden_config/nornir_plays/config_intended.py", line 17, in <module>
        from nautobot_plugin_nornir.constants import NORNIR_SETTINGS
      File "/appserver/nautobot/lib64/python3.6/site-packages/nautobot_plugin_nornir/constants.py", line 12, in <module>
        NORNIR_SETTINGS = PLUGIN_CFG.get("nornir_settings", _NORNIR_SETTINGS)
    AttributeError: 'NoneType' object has no attribute 'get'`

In this case, we're not running the backup due to environment restrictions so we really don't need this plugin and expect the defaults to work since this doesn't appear to require anything.

I believe the bug fix is to change nautobot_nornir_plugin/constants.py

PLUGIN_CFG = settings.PLUGINS_CONFIG.get("nautobot_plugin_nornir", {})
NORNIR_SETTINGS = PLUGIN_CFG.get("nornir_settings", _NORNIR_SETTINGS)

I'm not entirely sure if this is more or less a bug with Nautobot and how it is setting the plugin settings so we can potentially ask that as well.

Environment

• Python version: 3.6.8
• Nautobot version: 1.1.2
• nautobot-plugin-nornir version: 0.9.4

Steps to Reproduce

1. Use the CredentialsSettingsVars and defined credentials in the nautobot_config.py
2. Attempt to backup a config (in my case Juniper)

Expected Behavior

Backup of configuration using credentials defined

Observed Behavior

It appears to select the correct platform but is not using the defined credentials as per the below

Executing dispatcher for cpe1 (juniper_junos)
Found driver nornir_nautobot.plugins.tasks.dispatcher.juniper_junos.NautobotNornirDriver
There was no username defined, preemptively failed

Environment

• Python version: 3.7
• Nautobot version: 1.0
• nautobot-plugin-nornir version: 0.9

Proposed Functionality

The ability to set credentials from not only environment variables, but instead from the plugin configuration.

Use Case

The credentials can be just as easily stored in nautobot_config.py as it can in environment variables.

Hi.

Nautobot is able to reach / authenticate my network devices using secrets + env variables.

For some strange reason when i try run a backup job, i get a message stating that the env variables are not available.

"SecretValueNotFoundError: Secret "cisco-user" (provider "EnvironmentVariableSecretsProvider"): Undefined environment variable "NAPALM_USERNAME"! "

here is my nautobot plugin config:

PLUGINS = ["nautobot_plugin_nornir", "nautobot_golden_config", "nautobot_device_lifecycle_mgmt"]

PLUGINS_CONFIG = {
    "nautobot_plugin_nornir": {
        "use_config_context": {"secrets": True, "connection_options": True},
        # Optionally set global connection options.
        "connection_options": {
            "napalm": {
                "extras": {
                    "optional_args": {"global_delay_factor": 1},
                },
            },
            "netmiko": {
                "extras": {
                    "global_delay_factor": 1,
                },
            },
        },
        "nornir_settings": {
            "credentials": "nautobot_plugin_nornir.plugins.credentials.nautobot_secrets.CredentialsNautobotSecrets",
            "runner": {
                "plugin": "threaded",
                "options": {
                    "num_workers": 20,
                },
            },
        },
    },
    "nautobot_golden_config": {
        "per_feature_bar_width": 0.15,
        "per_feature_width": 13,
        "per_feature_height": 4,
        "enable_backup": True,
        "enable_compliance": True,
        "enable_intended": True,
        "enable_sotagg": True,
        "sot_agg_transposer": None,
        "platform_slug_map": None,
        # "get_custom_compliance": "my.custom_compliance.func"
    },

}

Thanks in advance.

Testing is near non-existent, needs to be updated.

Environment

• Python version:
• Nautobot version:
• nautobot-plugin-nornir version:

Expected Behavior

Observed Behavior

Redundant use_config_context key

PLUGINS_CONFIG = {
    "nautobot_plugin_nornir": {
        "use_config_context": {"use_config_context": {"secrets": False, "connection_options": True}},
        ...

Steps to Reproduce

Environment

• Python version: 3.7+
• Nautobot version: 1.2.x
• nautobot-plugin-nornir version: current + develop branch

Steps to Reproduce

1. Setup golden config plugin
2. Run against high number of devices.

Currently an intermittent issue that has only been hit a few times without exact replication steps.

Expected Behavior

Job completes as expected.

Observed Behavior

Job succeeds and then every device thereafter fails with too many clients error in the job result.

Initial Theories

the problem is a conflict between uwsgi and celery spawning processes and leaving connections open
basically by setting to serial, we just leave it up to uwsgi to thread it and the connections aren’t left open

This is a symptom not the issue. Needs to be investigated further.

Workaround

Set the runner to serial and disabled the num of workers.

I think it would be a good idea to update the README.md of ntc-nautobot-plugin-nornir to replace the sample config:

PLUGINS_CONFIG = {
    "nautobot_plugin_nornir": {
        # ...
        "dispatcher_mapping": None,
        "username": "ntc",
        "password": "password123",
        "secret": "password123",
    }
}

with

PLUGINS_CONFIG = {
    "nautobot_plugin_nornir": {
        "nornir_settings": {
            "credentials": "nautobot_plugin_nornir.plugins.credentials.settings_vars.CredentialsSettingsVars
            # ...
        },
        # ...
        "dispatcher_mapping": None,
        "username": "ntc",
        "password": "password123",
        "secret": "password123",
    }
}

It would be much more explicit and avoid some mistake

Environment

• Python version: 3.7+
• Nautobot version: 1.3+
• nautobot-plugin-nornir version: 1.0.0

Proposed Functionality

Utilize netutils mapping functions or user defined mappings to tranform the inventory platform for different nornir task plugins.

Use Case

Platform definitions between nautobot, netmiko, scrapli, etc aren't a one to one mapping. If you using nautobot inventory but want to call netmiko or another nornir task, you currently have to go manually update the platform entry in the nornir inventory host object.

Having a easy way to transpose this would be helpful and avoid having different platform mapping functions spread out between different plugins.

Environment

• Python version: 3.8
• Nautobot version: 1.0.0
• nautobot-plugin-nornir version: 0.9.0

The documentation hosted on PyPi incorrectly refers to the package name as nautobot_nornir for both the plugin and plugins_config sections

Steps to Reproduce

1. open a web browser
2. visit pypi page https://pypi.org/project/nautobot-plugin-nornir/
3. look at the documentation

Expected Behavior

Expected to read "nautobot_plugin_nornir"

Oberved reading "nautobot_nornir"

Admin guide link in Readme leads to 404.

Environment

• Python version: 3.8
• Nautobot version: 1.5
• nautobot-plugin-nornir version: latest

Steps to Reproduce

1. Create two devices, eg router and switch, with same platform eg cisco_ios
2. Create two secrets group. With diff creds.
3. Assign one group to each device.
4. Run a shim job (GC) with both devices selected

Expected Behavior

Both devices work and use the secrets info from the group attached to the device object.

Observed Behavior

Each device object has the correct user/pass based on secrets group; however the secret that is created within the extras in configuration_options will both be assigned the "last" object that is looped throughs secret.

From initial testing something with the groups is causing this. In nautobot_orm.py.

            # secrets are populated correctly at this point.
            for group in hosts[device.name].groups:
                if group not in groups.keys():
                    groups[group] = Group(name=group, defaults=defaults)
            # secrets are broken at this point.          

Seems like because both devices are part of the same "platform" group, its causing the problem.

Environment

• nautobot-plugin-nornir version:

Expected Behavior

The config used by NORNIR_SETTINGS is consistent either is coming from settings.PLUGINS_CONFIG or via default _NORNIR_SETTINGS, but they look to be different.

Observed Behavior

default settings: https://github.com/nautobot/nautobot-plugin-nornir/blob/develop/nautobot_plugin_nornir/constants.py#L5-L9

_NORNIR_SETTINGS = {
    "inventory": "nautobot_plugin_nornir.plugins.inventory.nautobot_orm.NautobotORMInventory",
    "credentials": "nautobot_plugin_nornir.plugins.credentials.env_vars.CredentialsEnvVars",
    "nornir.core": {"num_workers": 20},
}

Definitive settings are taken from Plugin config "nornir_settings" ( if available: https://github.com/nautobot/nautobot-plugin-nornir/blob/develop/nautobot_plugin_nornir/constants.py#L12
and the data is:

PLUGINS_CONFIG = {
    "nautobot_plugin_nornir": {
        "nornir_settings": {
            "credentials": "nautobot_plugin_nornir.plugins.credentials.env_vars.CredentialsEnvVars",
            "runner": {
                "plugin": "threaded",
                "options": {
                    "num_workers": 20,
                },
            },
        },
    },
}

so, comparing both:

"credentials": "nautobot_plugin_nornir.plugins.credentials.env_vars.CredentialsEnvVars",
"runner": {
   "plugin": "threaded",
       "options": {
            "num_workers": 20,
       },
   },

So, I see a difference between the defaults, using "nornir.core" to define the workers, and the plugin config using "runner"/"options".

For instance, in one example of library usage, in https://github.com/nautobot/nautobot-plugin-golden-config/blob/0c3e902582d1ce68094d731aa614f35fc28e701f/nautobot_golden_config/nornir_plays/config_backup.py#L114 , it is looking for "runner" that is not defined by default in the runner

Maybe I'm missing something?

Environment

• Python version: 3.6.2
• Nautobot version: 1.2.1
• nautobot-plugin-nornir version: 0.9.7

Proposed Functionality

Implement Nautobot Secrets as a backend option to retrieve secrets

Use Case

Be able to eventually integrate with other backends such as Hashicorp Vault or AWS Secrets Manager

Support more credential types.

Environment

• Python version: 3.7
• Nautobot version: 1.2.5
• nautobot-plugin-nornir version: Current

Steps to Reproduce

1. Install plugin and define plugin configuration without a credentials class setting.

PLUGINS_CONFIG = {
    "nautobot_plugin_nornir": {
        "dispatcher_mapping": None,
        "username": "cool_user",
        "password": "supersecret",
        "secret": "supersecret",
        "nornir_settings": {
            #"credentials": "nautobot_plugin_nornir.plugins.credentials.env_vars.CredentialsEnvVars",
            "runner": {
                "plugin": "threaded",
                "options": {
                    "num_workers": 20,
                },
            },
        },
    },
.....
}

Expected Behavior

If a setting is required like credential fail sanely.

Observed Behavior

Traceback with limited information on how to fix the problem.

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/nautobot/extras/jobs.py", line 1041, in _run_job
    output = job.run(data=data, commit=commit)
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_golden_config/jobs.py", line 48, in inner
    return method(obj, data, commit)
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_golden_config/jobs.py", line 103, in run
    config_compliance(self, data)
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_golden_config/nornir_plays/config_compliance.py", line 142, in config_compliance
    with InitNornir(
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nornir/init_nornir.py", line 72, in InitNornir
    inventory=load_inventory(config),
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nornir/init_nornir.py", line 20, in load_inventory
    inv = inventory_plugin(**config.inventory.options).load()
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_plugin_nornir/plugins/inventory/nautobot_orm.py", line 89, in __init__
    self.cred_class = import_string(credentials_class)
  File "/usr/local/lib/python3.9/site-packages/django/utils/module_loading.py", line 13, in import_string
    module_path, class_name = dotted_path.rsplit('.', 1)
AttributeError: 'NoneType' object has no attribute 'rsplit'

https://github.com/nautobot/nautobot-plugin-nornir/blob/53e89de3012ab942da2a318d5d7055d86689aaa4/nautobot_plugin_nornir/plugins/inventory/nautobot_orm.py#L201

Now that nornir-pyntc is open sourced it'd make sense to add it in as a default option.

Add the ability to add fqdn to all devices names.

In pyproject.toml the dev dependencies for this plugin should probably be pinned to "*" to inherit the actual version from Nautobot.

Allow user/operator defined grouping via the ORM inventory

Environment

• Python version: 3.10
• Nautobot version: 1.6
• nautobot-plugin-nornir version: 1.0.0

Proposed Functionality

golden-config#527 involves the ability to pass settings for the jinja environment into nornir-nautobot.

That capability has been added to nornir-nautobot v2.5.0, but nautobot-plugin-nornir still depends on v2.3.0, and nautobot-plugin-golden-config depends on nautobot-plugin-nornir. golden-config#527 is blocked until this can be fixed

Use Case

The default environment variables for env credentials type are NAPALM_USER and NAPALM_PASSWORD, should these also accept NAUTOBOT_ prepended as is the default in the various environments for deploying Nautobot and consistency with other environment variables?