Environment

• Nautobot version: 1.2.0
• nautobot-ssot version: 1.0.1

Proposed Functionality

Add support for the new JobLogEntry table.

Use Case

If Nautobot 1.2.0 is installed, the Job Logs within the SSOT Sync page will not load.

Environment

• Python version: Example: 3.7.12
• Nautobot version: Example: 1.2.8
• nautobot-ssot version: 1.1.0

Expected Behavior

SSoT synchronization completing successfully for the object with a field of type datetime.date.

Observed Behavior

An exception is raised, see details below, when the SSoT tries to save the sync diff dict.

An exception occurred: TypeError: Object of type date is not JSON serializable

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot_ssot/jobs/base.py", line 326, in run
    self.sync_data()
  File "/source/ssot/jobs.py", line 220, in sync_data
    self.sync.save()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 754, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 792, in save_base
    force_update, using, update_fields,
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 873, in _save_table
    forced_update)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 926, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 803, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1522, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1143, in execute_sql
    sql, params = self.as_sql()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1488, in as_sql
    val = field.get_db_prep_save(val, connection=self.connection)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 823, in get_db_prep_save
    return self.get_db_prep_value(value, connection=connection, prepared=False)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 818, in get_db_prep_value
    value = self.get_prep_value(value)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/json.py", line 93, in get_prep_value
    return json.dumps(value, cls=self.encoder)
  File "/usr/local/lib/python3.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type date is not JSON serializable

Steps to Reproduce

1. Create DiffSync model with field of datetime.date type.
2. Attempt synchronization where the date field is being updated.

Environment

• Python version: 3.9
• Nautobot version: 1.4.7
• nautobot-ssot version: 1.1.0

Expected Behavior

My SSoT job with a diffsync JSON diff representation of >268435455 bytes rusn through without an error.

This is 36126 sites with a good couple of fields and custom fields.

Observed Behavior

An exception occurred: OperationalError: total size of jsonb object elements exceeds the maximum of 268435455 bytes LINE 1: ...c_memory_peak" = NULL, "dry_run" = true, "diff" = '{"site": ... ^
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django_prometheus/db/common.py", line 71, in execute
    return super().execute(*args, **kwargs)
psycopg2.errors.ProgramLimitExceeded: total size of jsonb object elements exceeds the maximum of 268435455 bytes
LINE 1: ...c_memory_peak" = NULL, "dry_run" = true, "diff" = '{"site": ...
                                                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot_ssot/jobs/base.py", line 332, in run
    self.sync_data()
  File "/usr/local/lib/python3.7/site-packages/nautobot_ssot/jobs/base.py", line 155, in sync_data
    self.calculate_diff()
  File "/usr/local/lib/python3.7/site-packages/nautobot_ssot/jobs/base.py", line 84, in calculate_diff
    self.sync.save()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 740, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 778, in save_base
    force_update, using, update_fields,
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 859, in _save_table
    forced_update)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 912, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 802, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/cacheops/transaction.py", line 97, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django_prometheus/db/common.py", line 71, in execute
    return super().execute(*args, **kwargs)
django.db.utils.OperationalError: total size of jsonb object elements exceeds the maximum of 268435455 bytes
LINE 1: ...c_memory_peak" = NULL, "dry_run" = true, "diff" = '{"site": ...
                                                             ^

Steps to Reproduce

1. Have a lot of objects to sync such that the diff of the SSoT job becomes bigger than 268435455 bytes
2. Execute the SSoT job

Environment

• Nautobot version: 1.1.3
• nautobot-ssot version: 1.0.1

Proposed Functionality

While looking at some recent projects that are built on top of the ssot plugin, I noticed that all sync_data function pretty much looks the same and have been copy/pasted from the documentation.

It feels like it would make sense to offer an option to the datasource developer to only define the source and the target adapter and have a standard/default sync_data function

Use Case

Beyond the developer experience improvement, I think it would help to instrument the execution of the Job (#11) if we have a standard workflow.

Environment

• Nautobot version:
• nautobot-ssot-infoblox version:

Proposed Functionality

Simplify code by getting rid of custom Infoblox API client, using the official SDK

Use Case

Housekeeping

Environment

• Python version: 3.8
• Nautobot version: 1.3.7
• nautobot-ssot-infoblox version: 0.4.10

Expected Behavior

All Networks are imported into Nautobot.

Observed Behavior

Crash occurs if a duplicate network is found.

Steps to Reproduce

1. Stand up Nautobot with Infoblox SSoT.
2. Create multiple network views with duplicate networks in Infoblox.
3. Attempt SSoT import from Infoblox to Nautobot.

Environment

• Nautobot version: 1.2.5
• nautobot-ssot version: 1.1.0

Proposed Functionality

As a Tools Integrator, I want the SSoT plugin to support bulk operations within the nautobot ORM (or other target systems which support like functionality) so that jobs that deal with a lot of data can operate more performantly.

Use Case

I have several use cases which involve full inventory syncs between systems involving upwards of 75k+ objects. The run time of these jobs is currently measured in hours due to the serial nature of processing objects after the diff is calculated.

There is a tradeoff in nautobot concerning bulk ORM operations (bulk_create, bulk_update, etc) in terms of change logging and other signally, but in my case, I am willing to accept those downsides to be able to process the SSoT sync with better performance.

Proposed Functionality

Add an option to disable/enable import of models of a plugin.

Use Case

Sometimes, even though an SSoT plugin have mapping implemented for dozens of models, the specific use-case only requires a few of them, so it would make sense to offer an easy way to enable/disable them.
In case some of these models had some dependencies, for instance a Site model requires the Region model we could show the dependency issue and help the user to select the models that make sense.

Restricting the sync to the contents of a single Site is currently enforced due to nautobot/nautobot#1233. Once that's fixed and a new release is made available, the site_filter ObjectVar should be changed to required=False and we should verify that it's possible to do a full all-sites sync with reasonable time/memory performance.

Environment

• nautobot version: 1.5.x
• nautobot-ssot-ipfabric version: 2.0.0
• ipfabric version: 6.x

Summary

IP Fabric has the ability to discover detailed information about a device's inventory.
This is found in the "Modules and Part Numbers Inventory" table located at:

• API: /api/v#.#/tables/inventory/pn
• UI: /inventory/part-numbers

Proposed Functionality

• Sync IPF Device Inventory to Nautobot Device
• Then Sync IPF device part numbers to Nautobot Device Inventory Items

Open Questions

• How to enable/disable sync'ing inventory?
• How to deal with null part names?
• How to deal with duplicate part names on a single device?
  • See HWLAB-B-SW02-ARISTA-7050SX 2
• Since IPF can also query SFPs this table could have hundreds of thousands or more rows.
  • Can Nautobot handle this?
  • Previously we had issues sync'ing large datasets due to job timeouts.
• Do we allow filtering of parts?
  • Create a filter in IPF and then add it to nautobot_config?

Use Case

Allow sync'ing of device inventory information useful for capturing parts and their serial numbers.

Currently the job always runs with SKIP_UNMATCHED_DST so that records found in ServiceNow that don't correspond to records in Nautobot are simply ignored.

• Implement the diffsync model delete() API(s)
• Add an opt-in option to the job to enable deletion (running without SKIP_UNMATCHED_DST)
• Add appropriate testing

Proposed Functionality

By default, diffsync is executed for all the objects in the source and target adapted. We could apply filtering to only sync objects that match some criteria.

Filtering objects to execute diffsync could help to simplify some cases when only one or a few objects need to be evaluated for a sync.

Use Cases

As a network engineer, I would like to define filter criteria in the SSoT execution so the loading of objects in the source and destination datasets are restricted, and the whole process is narrowed down to a specific scope. For instance, Nautobot and an external Asset Management system contain 1000s of Sites/Location, but when a change on a specific one happens I would like to run the SSoT only for this location, from both the source and destination. So, the loading function should take into account the filtering for both adapters, and load only the related information. If the location contains Devices, and Devices contain Interfaces, I would only be interested on comparing this related data, leaving all the rest out of the comparison.

Environment

• Nautobot version: 1.3.9
• nautobot-ssot-infoblox version: 0.6.1

Proposed Functionality

The current import of Extensibility Attributes as CustomFields are all done assuming that they're strings. However, in reality, there are a variety of types that the Attributes can be such as:

• Integer
• List
• E-mail
• URL
• Date

Use Case

In order to have the data types match what is being imported, we should update the CustomFields to match. This will require that we get information about the setup of each EA in use and have the CustomField created to match.

Environment

• Python version: 3.8.3
• Nautobot version: 1.3.8 (and 1.4.1)
• nautobot-ssot-aci version: 1.0 (from pypi)

Expected Behavior

nautobot-plugin-ssot-aci should accept the same dependencies as Nautobot itself and the nautobot-plugin-ssot plugin.
It shouldn't conflict with version of Jinja2 and PyAML

Observed Behavior

Installing nautobot-plugin-ssot-aci make a downgrade to nautobot 1.2.11.
When re-upgrading nautobot (as of date of posting : 1.4.1), it conflicts with Jinja2 and PyAML versions.

Steps to Reproduce

1. The starting version of Nautobot was 1.3.8
2. Installing through pip nautobot-plugin-ssot-aci :

$ pip install --upgrade nautobot-ssot-aci
[-- cut --]
Attempting uninstall: nautobot
    Found existing installation: nautobot 1.3.8
    Uninstalling nautobot-1.3.8:
      Successfully uninstalled nautobot-1.3.8
Successfully installed Django-3.1.14 GitPython-3.1.18 Jinja2-2.11.3 MarkupSafe-2.0.1 PyYAML-5.4.1 django-cacheops-5.1 django-cors-headers-3.7.0 django-filter-2.4.0 django-jinja-2.7.1 django-mptt-0.11.0 django-prometheus-2.1.0 django-rq-2.4.1 django-tables2-2.3.4 django-taggit-1.3.0 djangorestframework-3.12.4 jsonschema-3.2.0 nautobot-1.2.11 nautobot-ssot-aci-1.0.0 netutils-1.0.0 psycopg2-binary-2.8.6 pycryptodome-3.10.4 social-auth-app-django-4.0.0

3. Re-upgrade Nautobot

$ pip install --upgrade nautobot
[-- cut --]
Attempting uninstall: nautobot
    Found existing installation: nautobot 1.2.11
    Uninstalling nautobot-1.2.11:
      Successfully uninstalled nautobot-1.2.11
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nautobot-ssot-aci 1.0.0 requires Jinja2<3, but you have jinja2 3.0.3 which is incompatible.
nautobot-ssot-aci 1.0.0 requires PyYAML<6.0.0,>=5.4.1, but you have pyyaml 6.0 which is incompatible.
Successfully installed Django-3.2.15 GitPython-3.1.27 Jinja2-3.0.3 Markdown-3.3.7 MarkupSafe-2.1.1 Pillow-9.2.0 PyYAML-6.0 celery-5.2.7 django-cacheops-6.0 django-cors-headers-3.13.0 django-extensions-3.2.0 django-filter-21.1 django-jinja-2.10.2 django-mptt-0.13.4 django-prometheus-2.2.0 django-rq-2.5.1 django-tables2-2.4.1 django-taggit-1.5.1 django-tree-queries-0.11.0 djangorestframework-3.13.1 jsonschema-4.7.2 nautobot-1.4.1 netutils-1.1.0 psycopg2-binary-2.9.3 pycryptodome-3.13.0 social-auth-app-django-5.0.0

Environment

• Python version: 3.8.13
• Nautobot version: 1.3.3
• nautobot-ssot version: 1.1.1

Expected Behavior

I expected the SSoT job not to crash.

Observed Behavior

The SSoT job crashed.

An exception occurred: AttributeError: 'dict' object has no attribute 'summary'

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nautobot_ssot/jobs/base.py", line 327, in run
    self.sync_data()
  File "/usr/local/lib/python3.8/site-packages/nautobot_ssot/jobs/base.py", line 153, in sync_data
    self.calculate_diff()
  File "/usr/local/lib/python3.8/site-packages/nautobot_ssot/jobs/base.py", line 83, in calculate_diff
    self.log_info(message=self.sync.diff.summary())
AttributeError: 'dict' object has no attribute 'summary'

Steps to Reproduce

1. TBD

Docs

• Can we flesh out the intro paragraph to more than just "A plugin for Nautobot"? Provide some details about what this plugin is for and why you might want it.
• Having "Home" and "Readme" as separate pages seems less than ideal to me - can they be merged in some way?
• "Build Status" badges don't appear to render in GitHub pages:
  • 
• "TODO: /slash command screenshot" (README) and "Screenshots: TODO" (rendered docs)
• Please review the "Local Poetry Development Environment" versus "Docker Development Environment" sections for correctness and completeness
  • Installing invoke and poetry is a prerequisite for both environments, yet it's only mentioned in the Docker section?
  • Creating a creds.env is a prerequisite for both environments, yet it's only mentioned in the Local section?
  • The local development example invoke.yml: if you're running with local: true I would expect that compose_files should be unnecessary since you're not using docker?
  • Since the Docker environment is simpler to start with than the local environment, perhaps it should be listed first?
• What's the significance of GETTING_STARTED.md - should it be linked or referenced from either README.md or from the rendered docs, or is it redundant in some way?
• Why do the GitHub Pages docs look to have completely different content than the files in docs/ -- changelog.md, plugin.md, and index.rst all appear to have content completely different from the rendered docs?

Development Environment

• Consider moving from Python 3.6 to 3.7 as the default due to CVE-2021-23727 (Celery vulnerability only fixed in 3.7 and later)
• What's the significance of custom_logging_levels.py - why not just set the logging directly in development/nautobot_config.py?
• There are a lot of settings in development/nautobot_config.py that are just matching Nautobot defaults as of 1.1.x. This may just be a cookiecutter thing but this file could be pruned substantially if desired.

Code

• Given that nautobot_chatops and nautobot_ssot use lowercase settings in PLUGINS_CONFIG (enable_slack, hide_example_jobs, etc.) it would probably be good for consistency to also use lowercase settings for this plugin, rather than uppercase ones.
  • Documentation is missing for the DEFAULT_DEVICE_ROLE and DEFAULT_DEVICE_STATUS config settings.
  • Documentation is missing for the DEFAULT_INTERFACE_* config settings as well.
• template_content.py has a hard-coded ipfabric.demo.networktocode.com that probably should be replaced with the appropriate config variable? Looks like there may also be some hard-coded query parameters in that URL to review.
  • You also have a hard-coded URL in the diagram.html template file.
• If this is building atop the existing nautobot-chatops-ipfabric plugin, I'm surprised to see that this plugin is registering a top-level ipfabric command of its own - how does this coexist with the other plugin, or does it clobber it?
  • Is it possible to use this plugin without chatops? If so the nautobot-chatops and nautobot-chatops-ipfabric should be optional packaging dependencies, but right now they're declared as required in pyproject.toml.
• Dependency on netutils should be updated to ^1.0 as otherwise this plugin won't be usable with Nautobot 1.2.
• nautobot_ssot_ipfabric/api folder can be removed if you don't have or plan to have any REST API for this plugin.
• Same for nautobot_ssot_ipfabric/migrations/ since you have no models to migrate.
• Looks like the sys_id and pk attributes on your diffsync models are unused and can be removed
• Interface.create() and Interface.update() are calling device_obj.validated_save() - this doesn't appear to be necessary?
• For my own education, what does tonb mean?
• Is there a reason you're not using get_or_create() in your create_site and create_status functions, given that you're using it in the other create_* functions?
• I'm slightly concerned about the performance of the ipfabric adapter if you have a large number of devices with a large number of interfaces - you're calling load_device_interfaces once per device, and in that function, every time it's called you're iterating over all interfaces to find the ones applicable to the device in question. It might be more efficient to preprocess the get_interface_inventory response once to map each interface to its associated device so that you don't have to iterate over the entire list every time.
• Nautobot adapter, line 59 - shouldn't this be something like self.get(self.interface, f"{interface_record.name}__{interface_record.device.name}")? I don't see where self.name is coming from.
  • Similarly, at line 114, shouldn't that be self.get(self.vlan, f"{vlan_record.name}__{vlan_record.site.name}")?
  • Actually, for all of the load_* functions, why do you even need to try to load an existing record at all? The whole point of these functions is to load fresh data, right? There should never be an existing record matching what you're processing, I think?
• It'd be good if the adapter test cases actually checked correctness of the loaded data values, rather than just simple hasattr checks

Proposed Functionality

Load Source and Targets concurrently

Use Case

Loading adapters, being I/O operations that have no dependencies between them, could be run in parallel and save time

Environment

• DiffSync version: 1.6.0

Proposed Functionality

It would be useful if there was a screen that showed the objects to be imported from your DataSource and allow you to select the objects out of the batch you wish to import. Due to large datasets we would need the ability to pick groups of objects. Similar to #37 it'd be useful to filter the objects to import either before the Job starts or once the diff has completed and you can see what CRUD operations would occur.

Use Case

As a end-user I'd like the ability to pick individual objects, either Sites or Devices or related, and have only those imported. This would be extremely useful when there's only one change in a SoR that I want to import.

Environment

• Nautobot version:
• nautobot-ssot-ipfabric version:

Proposed Functionality

IP Fabric allows for webhooks to send to external systems when discovery completes. It would be nice to have IPF send the webhook to Nautobot and that it automatically runs the SSOT job.

Use Case

Currently it's hard to understand from a job view as to which way a sync may happen.

Original with no job name defined under the Meta class from the SSoT dashboard:

The tables make it clear as to what the source and target is and it also makes it clear when you click into either data target/source.

A problem arises if we look at the Extensibility -> Jobs:

It can be hard to quickly clean the direction of the sync without reading a description. It was proposed during preparation for NFD27 to have it more explicit within the name.

Now we'll change the name attribute within the Meta class for each of the three jobs and see an improvement:

But the downside is the redundancy that exists now within the two other views:

Adding @glennmatthews as he had some ideas around the change for this.

Environment

• Nautobot version: 1.1.4
• nautobot-ssot-infoblox version: 0.4.2

Proposed Functionality

Infoblox has the ability to store additional information on any piece of data like Networks or IP Addresses. These are akin to custom fields in Nautobot. These can contain important information that could be relevant to import into Nautobot.

Use Case

As an end-user I'd like to have the additional information stored in extensible attributes in Infoblox I'd like that data imported into Nautobot.

Environment

• Nautobot version: 1.1.3
• aristacv-sync version: 1.0.2

Proposed Functionality

Adding a default platform in the same way as we can add e.g. device_role automatically:

PLUGINS_CONFIG = {

"nautobot_ssot" : {

ADD YOUR SETTINGS HERE

}

"nautobot_ssot_aristacv": {

(...)

"from_cloudvision_default_site": "",

"from_cloudvision_default_device_role": "",

I would like to have a possibility to set a default platform here, the default value could be e.g. EOS in plugin settings.
Then all device types should be associated to this device platform.

Use Case

Environment

• Nautobot version: 1.4.5
• nautobot-ssot version: 1.2.0

Proposed Functionality

It would be useful to have defined models for the Nautobot SoR that can be reused across SSoT plugins.

Use Case

During the course of building an SSoT plugin there is a lot of replicated code on the Nautobot side as the models and CRUD operations don't change. It'd be helpful if we had defined models that cover all attributes of importable Nautobot models and the appropriate CRUD methods for each and allow the plugin developer to define the objects to import and attributes to match on.

Environment

• Nautobot version: 1.3.7
• nautobot-ssot-infoblox version: 0.4.10

Proposed Functionality

The plug-in currently supports importing network containers from Infoblox as Aggregates as long as they're RFC1918 space. This limitation was put in place as it was difficult to determine the RIR to assign an Aggregate to at the time. Since then I've found the following projects that may make it much easier:

https://github.com/yoda66/RIRTools
https://github.com/Marvin182/rir-database

I propose that we utilize the rir-database project to first create the database and then the RIRTools project to create a queryable database of those RIRs and Aggregates. This along with a setting for private RIR name should allow for us to better document IPAM information for the end user.

Use Case

For customers that track their RIRs and Aggregates, this feature could be extremely useful.

Environment

• Python version: N/A
• Nautobot version: N/A
• nautobot-ssot-infoblox version: 0.7.4

Expected Behavior

Logging of information to be done in a reasonable manner. Ie, keep informational logging to just necessary information so the user knows where in the process the Job is and trim out unnecessary logging and only necessary debug logging.

Observed Behavior

Logging isn't consistent and missing information that could aide the user in knowing what is happening.

Environment

• Python version: 3.7
• Nautobot version: 1.5.1
• nautobot-ssot-servicenow version: 0.9.2

Expected Behavior

You could sync a site that has no parent region

Observed Behavior

a site without a parent region is ignored

Steps to Reproduce

1. Create a site
2. Diff or sync
3. It's not loadad

Environment

• Nautobot version: 1.2.8
• nautobot-ssot version: 1.1.2

Proposed Functionality

As a plugin developer I would like to be able to conditionally look up objects based on the action type applied to given diffsync record.

Use Case

As a plugin developer I want to specify that the object lookup should not happen for records that were deleted. I only want to perform object lookup for records that were created or updated.

Environment

• Nautobot version: 1.2.8
• nautobot-ssot version: 1.1.0

Proposed Functionality

As an end-user, it'd be very helpful to have the SSoT logs available via the API for programmatic searching. This is especially important as the number of SoRs imported and the number of objects imported increases.

Use Case

As an end-user, when I have an inquiry about why a large number of devices weren't imported it can be rather arduous to review the logs via the GUI. Being able to programmatically search them via the API would make this task much easier.

The mappings.yaml file format is not overly complex, but it needs to be documented clearly for ease of future expansion to support more models and/or fields.

Topics:

• Scoping of models loaded initially from both Nautobot and the remote system needs to match to avoid unintended creates/deletes
• naming patterns for Jobs?
• Adapter customization
• Limit x model
  • can become confusing because of dependency (such as it doesn’t make sense to sync Interfaces but not Devices, but it might make sense to sync Devices but not Interfaces)
• Limit x objects of a model
  • Figure out What filters to support
  • maybe using dynamic group

Adding @h4ndzdatm0ld and @glennmatthews for any further insight.

Environment

• nautobot-ssot version: 1.1.2

Proposed Functionality

Make base plugin usable with Mysql database, and add it's testing in the CI

Use Case

Support Mysql database, as Nautobot is supporting it.

Environment

• nautobot version: 1.5.x
• nautobot-ssot-ipfabric version: 2.0.0
• ipfabric version: 6.x

Summary

IP Fabric records two serial numbers:

• "Unique Serial Number"
  • Column sn
• "Serial Number"
  • Column snHw (Hardware Serial Number)

Nautobot has a single serial number for a device under "Hardware" with a description of "Chassis serial number".

Currently, the SSOT syncs the IPF sn and not the snHw which is not correct from the description above.

Proposed Functionality

• Add new custom field IPFabric Serial Number or IPFabric Unique Serial Number?
• Sync jobs still use the sn field as this is a unique value and ensures all data is sync'ed correctly.
• Update sync jobs to use snHw for the Device Serial Number
• Add sn data to custom field.

Open Questions

• Can Nautobot have duplicate Serial Numbers?
  • YES
• Can this be configurable?
  • CONFIG.get("default_device_serial_number", "sn" or "snHw")
• How do we migrate users?
  • SN might be an important field for their current automation.

Use Case

Device: L71FW9

Hostname,Unique serial number,Serial Number
hostname,sn,snHw
L71FW9/root,FOSVM1HHJXBALLD7/root,FOSVM1HHJXBALLD7
L71FW9/ipf,FOSVM1HHJXBALLD7/ipf,FOSVM1HHJXBALLD7

Environment

• Nautobot version: 1.1.4
• nautobot-ssot version: 1.1.2

Proposed Functionality

The ability to dynamically collapse/expand the diff tree in a given sync detail page. Sometimes, when a sync is too large, the sync detail page does not load (or takes a very long time to load). Having the ability to dynamically close and open parts of the diff could help with this.

Use Case

In some cases an end user may not want to see the whole diff output. At times, the full diff can be a lot to parse through. Having the ability to dynamically collapse and expand parts of the diff could make it easier to digest.

Environment

• Python version: 3.6.15
• Nautobot version: 1.2.10
• nautobot-ssot version: 1.1.0

Expected Behavior

Being able to open job history detail in SSoT

Observed Behavior

08:44:37.325 ERROR   django.request :
  Internal Server Error: /plugins/ssot/history/82d43bc2-fcac-46b4-aee4-237fd6a58c73/
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/nautobot/utilities/views.py", line 94, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/nautobot/core/views/generic.py", line 124, in get
    **self.get_extra_context(request, instance),
  File "/usr/local/lib/python3.6/site-packages/django/shortcuts.py", line 19, in render
    content = loader.render_to_string(template_name, context, request, using=using)
  File "/usr/local/lib/python3.6/site-packages/django/template/loader.py", line 62, in render_to_string
    return template.render(context, request)
  File "/usr/local/lib/python3.6/site-packages/django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 170, in render
    return self._render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/library.py", line 192, in render
    output = self.func(*resolved_args, **resolved_kwargs)
  File "/usr/local/lib/python3.6/site-packages/nautobot_ssot/templatetags/render_diff.py", line 66, in render_diff
    return format_html(result)
  File "/usr/local/lib/python3.6/site-packages/django/utils/html.py", line 115, in format_html
    return mark_safe(format_string.format(*args_safe, **kwargs_safe))
KeyError: "'asw_owner'"

Steps to Reproduce

1. add attribute type dict to DiffSyncModel class _attributes with some default value (no need to fill data later, dict must contain at least 1 key)
   cfs: typing.Dict[str, str] = {"asw_owner": ""}
2. run job
3. go to run detail in SSoT

Environment

• Python version: current
• Nautobot version: current
• nautobot-ssot version: current

Expected Behavior

The RTD to render similar to the Github rendering:

Observed Behavior

Steps to Reproduce

1. Go to: https://nautobot-plugin-ssot.readthedocs.io/en/latest/developing_jobs/

Environment

• Nautobot version: 1.2.4
• nautobot-ssot-ipfabric version: 0.9.4

Proposed Functionality

Nautobot GUI should show IP Fabric SSOT running jobs. Currently it only lists Completed or Failed.

Use Case

If you navigate away from the sync details you have no way of knowing if a job is running or not.

Environment

• Python version: 3.7.15
• Nautobot version: 1.5.4
• nautobot-ssot-aci version: 1.0.0

Expected Behavior

Nautobot versions 1.4 and above require an interface status to be set when an interface is created. When running the plugin on Nautobot v1.4 or above the interface status pulled from ACI should be populated in the 'status' field. Alternatively a default value for 'status' should be set for all populated interfaces based on a plugin variable set in nautobot_config.py

Observed Behavior

When populating the database the job will fail with the following error

Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/nautobot_ssot/jobs/base.py", line 332, in run self.sync_data() File "/usr/local/lib/python3.7/site-packages/nautobot_ssot/jobs/base.py", line 167, in sync_data self.execute_sync() File "/usr/local/lib/python3.7/site-packages/nautobot_ssot/jobs/base.py", line 95, in execute_sync self.source_adapter.sync_to(self.target_adapter, flags=self.diffsync_flags) File "/usr/local/lib/python3.7/site-packages/diffsync/__init__.py", line 579, in sync_to target.sync_from(self, diff_class=diff_class, flags=flags, callback=callback, diff=diff) File "/usr/local/lib/python3.7/site-packages/diffsync/__init__.py", line 557, in sync_from result = syncer.perform_sync() File "/usr/local/lib/python3.7/site-packages/diffsync/helpers.py", line 327, in perform_sync changed |= self.sync_diff_element(element) File "/usr/local/lib/python3.7/site-packages/diffsync/helpers.py", line 367, in sync_diff_element changed, modified_model = self.sync_model(src_model=src_model, dst_model=dst_model, ids=ids, attrs=attrs) File "/usr/local/lib/python3.7/site-packages/diffsync/helpers.py", line 416, in sync_model dst_model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs) File "/usr/local/lib/python3.7/site-packages/nautobot_ssot_aci/diffsync/models/nautobot.py", line 281, in create _interface.validated_save() File "/usr/local/lib/python3.7/site-packages/nautobot/core/models/__init__.py", line 51, in validated_save self.full_clean() File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 1251, in full_clean raise ValidationError(errors) django.core.exceptions.ValidationError: {'status': ['This field cannot be blank.']}

Steps to Reproduce

1. Be on Nautobot version 1.4+
2. Install nautobot_ssot_aci plugin
3. Define ACI environment via ENV vars
4. Run job (not dry-run) against an ACI environment

Environment

• Python version: 3.9
• Nautobot version: 1.1.4
• nautobot-ssot version: 1.01

Triggering the diffsycn from Remote Nautobot (from demo.nautobot.com) to local Nautobot should execute successfully.

Observed Behavior

Because pagination is disabled to load objects from Remote Nautobot (https://github.com/nautobot/nautobot-plugin-ssot/blob/develop/nautobot_ssot/jobs/examples.py#L293) it loads only 50 Regions and 50 Sites, so some of the Sites are referencing to Regions that do not exist, failing the execution.

Steps to Reproduce

1. Plugins -> Dashboard
2. Data Sources -> Example Data Source
3. Sync Now without Dry Run

Environment

• Nautobot version: 1.4.7
• nautobot-ssot version: 1.2.0

Problem

Facilitate handling of SSoT data sync jobs that total a considerable amount of objects. In the case of SSoT Jobs that need to synchronize objects in the hundreds of thousands, a User can increase their RAM to facilitate the run but it will ultimately error if they are wanting to utilise the diff_from() or diff_to() methods as when saving the output dict to the database it will write a row totalling more than 1GB in some instances.

Additionally, in some cases although the row will not total more than 1GB, the diff that will be displayed is too large to even be rendered on a webpage

Proposed Functionality

Potentially implement a check to disable the diff_from / diff_to methods from running if the SSoT sync exceeds a certain size to avoid the erroring

(Maybe even just somewhere in the documentation mentioning that if the SSoT is expected to exceed a certain number of objects to remove the diff_from / diff_to)

Use Case

When developing an SSoT Data Job, if the dataset you are working with exceeds the limitations of the database it should be handled alternatively

Make ssot-safe-delete and ssot-synced-from-ipfabric configurable by the user

Environment

• Nautobot version: 1.1.3
• nautobot-ssot version: 1.0.1

Proposed Functionality

Currently only the total duration of the Job execution is being captured and reported in the Sync Result view.

It would be great to have more information available to help identify any performance issue like :

• individual time to load each adapter
• time to execute the diff
• time to execute the sync
• memory utilized

Use Case

Improve developer experience and help identify any performance issue

Environment

• Python version: 3.7
• Nautobot version: 1.3.9
• nautobot-ssot-infoblox version: 0.6.0

Expected Behavior

I expect to see a date and timestamp in the Custom Field value.

Observed Behavior

The field's value appears to be blank.

Steps to Reproduce

1. Setup Nautobot with the Infoblox SSoT plugin.
2. Setup the Infoblox SSoT plugin and sync to your instance.
3. Look at one of the objects that was imported and validated the Custom Field of "Last synced to Infoblox on".

Environment

• Nautobot version: 1.4.5
• nautobot-ssot version: 1.2.0

Proposed Functionality

Given multiple Sources of Truth, and duplicate devices, data about the device is going to be added into Nautobot. It would be helpful if there was a prioritization option on either the source or on the job run.

Use Case

As an engineer, when information about an existing device is going to be added we can determine if we should override or not. Then when records that conflict between multiple SSoT plugins, we would have the ability to select which one “wins”.

Environment

• Python version: 3.7.15
• Nautobot version: 1.5.4
• nautobot-ssot-aci version: 1.0.0

Expected Behavior

Absolute paths for directories should be based on the plugin install location, not the expected install user (e.g. nautobot)

Observed Behavior

When the nautobot-ssot-aci plugin is installed with another user, like root, the plugin continues to look for plugin files in the nautobot directory instead of the directory where the module is installed.

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot_ssot/jobs/base.py", line 332, in run
    self.sync_data()
  File "/usr/local/lib/python3.7/site-packages/nautobot_ssot/jobs/base.py", line 137, in sync_data
    self.load_source_adapter()
  File "/usr/local/lib/python3.7/site-packages/nautobot_ssot_aci/jobs.py", line 70, in load_source_adapter
    self.source_adapter.load()
  File "/usr/local/lib/python3.7/site-packages/nautobot_ssot_aci/diffsync/adapters/aci.py", line 380, in load
    self.load_devices()
  File "/usr/local/lib/python3.7/site-packages/nautobot_ssot_aci/diffsync/adapters/aci.py", line 348, in load_devices
    if f"{self.devices[key]['model']}.yaml" in os.listdir("nautobot_ssot_aci/diffsync/device-types"):
FileNotFoundError: [Errno 2] No such file or directory: 'nautobot_ssot_aci/diffsync/device-types'

Steps to Reproduce

1. su root
2. pip install nautobot-ssot-aci
3. sudo -iu nautobot
4. nautobot-server start
5. Go to Nautobot WebUI and start a new SSOT ACI job and it will error out with a FileNotFoundError exception

Environment

• nautobot-ssot-ipfabric version: 0.9.6

Expected Behavior

When you change the device_type of a device, it should be updated properly

Observed Behavior

The update method doesn't work as expected, it breaks because assumes that there is a vendor also defined.

Steps to Reproduce

Currently there is tons of data that can be synced due to the amount of IP addresses and prefixes available.

We need to investigate how we can potentially map extrattrs on these objects to ones within Nautobot to narrow jobs to speed them up as well as limit the scope.

Environment

• Python version: 3.10
• Nautobot version: 1.2.3
• nautobot-ssot version: 1.0.1

nautobot_ssot would show up in plugins list after being installed

nautobot_ssot plugin not in plugins list, error in nautobot log:

File "/Users/alex/Projects/python/nautobot/.venv/lib/python3.10/site-packages/nautobot_ssot/jobs/__init__.py", line 7, in <module>
    from .base import DataSource, DataTarget
  File "/Users/alex/Projects/python/nautobot/.venv/lib/python3.10/site-packages/nautobot_ssot/jobs/base.py", line 14, in <module>
    import structlog
ModuleNotFoundError: No module named 'structlog'

Steps to Reproduce

1. install nautobot_ssot as described in the documentation
2. start nautobot

Environment

• Python version: 3.6.15
• Nautobot version: 1.2.10
• nautobot-ssot version: 1.1.0

Expected Behavior

SSoT job history shown

Observed Behavior

15:35:06.108 ERROR   django.request :
  Internal Server Error: /plugins/ssot/history/7881d7af-7e32-4711-b468-c6a578f5776c/
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/nautobot/utilities/views.py", line 94, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/nautobot/core/views/generic.py", line 124, in get
    **self.get_extra_context(request, instance),
  File "/usr/local/lib/python3.6/site-packages/django/shortcuts.py", line 19, in render
    content = loader.render_to_string(template_name, context, request, using=using)
  File "/usr/local/lib/python3.6/site-packages/django/template/loader.py", line 62, in render_to_string
    return template.render(context, request)
  File "/usr/local/lib/python3.6/site-packages/django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 170, in render
    return self._render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/defaulttags.py", line 312, in render
    return nodelist.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 988, in render
    output = self.filter_expression.resolve(context)
  File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 698, in resolve
    new_obj = func(obj, *arg_vals)
  File "/usr/local/lib/python3.6/site-packages/nautobot_ssot/templatetags/humanize_bytes.py", line 20, in humanize_bytes
    if abs(size) < 1024.0:
TypeError: bad operand type for abs(): 'NoneType'

Steps to Reproduce

(not really sure except where it shows)

1. Run SSoT job with memory profiling
2. Open history of last run

I'm wondering if providing a base job with a sync_data method that loads the adapters, performs the diff, and implements the dry run functionality might be worth it?

This is more for a discussion to see if it's worth it as it may not be with some of the specifics for each plugin, but potentially people can use it to call super() after custom logic.

It seems like we could adapt this more into a parent/child setup due to prefixes should be present in both systems and then IP addresses set within that.

I'm sure there is more than one way, but we should definitely perform some testing on the best relationships for this plugin.