Merge pull request #611 from nautobot/develop

Merge 1.6.1 into main

README.md

@@ -17,13 +17,16 @@ The Golden Config plugin is a Nautobot plugin that provides a NetDevOps approach
 
 ### Key Use Cases
 
-This plugin enable five (5) key use cases.
+This plugin enable six (6) key use cases.
 
 1. **Configuration Backups** - Is a Nornir process to connect to devices, optionally parse out lines/secrets, backup the configuration, and save to a Git repository.
 2. **Intended Configuration** - Is a Nornir process to generate configuration based on a Git repo of Jinja files to combine with a GraphQL generated data and a Git repo to store the intended configuration.
 3. **Source of Truth Aggregation** - Is a GraphQL query per device that creates a data structure used in the generation of configuration.
 4. **Configuration Compliance** - Is a process to run comparison of the actual (via backups) and intended (via Jinja file creation) CLI configurations upon saving the actual and intended configuration. This is started by either a Nornir process for cli-like configurations or calling the API for json-like configurations
-5. **Configuration Postprocessing** - (beta) This process renders a valid configuration artifact from an intended configuration, that can be pushed to devices. The current implementation renders this configuration; however, **it doesn't push it** to the target device.
+5. **Configuration Remediation** - Is a process of generating a partial device configuration that would get a configuration feature into a compliant state. 
+6. **Configuration Deployment** - Is a process to generate a device configuration and push it to the network device. It supports compliance features, remediation engine and manual definitions.
+
+> Notice: **Configuration Postprocessing** - (beta feature) This process renders a valid configuration artifact from an intended configuration, that can be pushed to devices. The current implementation renders this configuration; however, **it doesn't push it** to the target device.
 
 > Notice: The operators of their own Nautobot instance are welcome to use any combination of these features. Though the appearance may seem like they are tightly coupled, this isn't actually the case. For example, one can obtain backup configurations from their current RANCID/Oxidized process and simply provide a Git Repo of the location of the backup configurations, and the compliance process would work the same way. Also, another user may only want to generate configurations, but not want to use other features, which is perfectly fine to do so.
 

development/nautobot_config.py

@@ -3,7 +3,6 @@
 import os
 import sys
 
-from django.utils.module_loading import import_string
 from nautobot.core.settings import *  # noqa: F403
 from nautobot.core.settings_funcs import is_truthy, parse_redis_connection
 
@@ -177,7 +176,7 @@
         "postprocessing_callables": os.environ.get("POSTPROCESSING_CALLABLES", []),
         "postprocessing_subscribed": os.environ.get("POSTPROCESSING_SUBSCRIBED", []),
         "jinja_env": {
-            "undefined": import_string("jinja2.StrictUndefined"),
+            "undefined": "jinja2.StrictUndefined",
             "trim_blocks": is_truthy(os.getenv("NAUTOBOT_JINJA_ENV_TRIM_BLOCKS", True)),
             "lstrip_blocks": is_truthy(os.getenv("NAUTOBOT_JINJA_ENV_LSTRIP_BLOCKS", False)),
         },

docs/admin/admin_install.md

@@ -62,7 +62,7 @@ PLUGINS_CONFIG = {
         "postprocessing_subscribed": [],
         "platform_slug_map": None,
         "jinja_env": {
-            "undefined": StrictUndefined,  # jinja2.StrictUndefined
+            "undefined": "jinja2.StrictUndefined",
             "trim_blocks": True,
             "lstrip_blocks": False,
         },
@@ -123,7 +123,7 @@ The plugin behavior can be controlled with the following list of settings.
 
     ```python
         jinja_env = {
-            "undefined": import_string("jinja2.StrictUndefined"),
+            "undefined": "jinja2.StrictUndefined",
             "trim_blocks": True,
             "lstrip_blocks": False,
         }

docs/admin/release_notes/version_1.6.md

@@ -5,13 +5,29 @@
 - Add functionality to compliance result to provide a Remediation plan.
 - Supports Nautobot >=1.6.1,<2.0.0.
 
+## v1.6.1 - 2023-09
+
+### Changed
+
+- [#600](https://github.com/nautobot/nautobot-plugin-golden-config/pull/600) - Updated readme to include the additional use cases covered.
+
+### Fixed
+
+- [#603](https://github.com/nautobot/nautobot-plugin-golden-config/pull/603) - Fix missing fields from the "AllDevicesGoldenConfig" Job.
+- [#609](https://github.com/nautobot/nautobot-plugin-golden-config/pull/609) - Fixed issue where not all jinja filers, specifically netutils were being loaded into Jinja environment.
+- [#609](https://github.com/nautobot/nautobot-plugin-golden-config/pull/609) - Fixed issues if a Job was never created since the feature was disabled, it would cause a stacktrace.
+- [#609](https://github.com/nautobot/nautobot-plugin-golden-config/pull/609) - Fixed issue where in GoldenConfigSetting page, dynamic group selection would not show all of the eligible options.
+- [#609](https://github.com/nautobot/nautobot-plugin-golden-config/pull/609) - Fixed issue where you could not fill in `jinja_env['undefined']` vars as a string, only a complex class.
+- [#609](https://github.com/nautobot/nautobot-plugin-golden-config/pull/609) - Added the ability to generate remediation configurations and store in ConfigRemediation model
+
 ## v1.6.0 - 2023-09
 
 ### Added
 
 - [#573](https://github.com/nautobot/nautobot-plugin-golden-config/pull/573) - Added the ability to generate remediation configurations and store in ConfigRemediation model
 - [#573](https://github.com/nautobot/nautobot-plugin-golden-config/pull/573) - Added the ability to generate configurations that you plan to deploy from a variety of methods, such as Remediation, intended, manual, etc. via the ConfigPlan model.
 - [#573](https://github.com/nautobot/nautobot-plugin-golden-config/pull/573) - Added the ability to Deploy configurations from the ConfigPlan configurations to your network devices.
+- [#578](https://github.com/nautobot/nautobot-plugin-golden-config/pull/578) - Updated ComplianceRule and ComplianceRule forms to include tags.
 
 ### Fixed
 

nautobot_golden_config/__init__.py

@@ -5,7 +5,6 @@
 
 __version__ = metadata.version(__name__)
 
-from jinja2 import StrictUndefined
 from django.db.models.signals import post_migrate
 from nautobot.core.signals import nautobot_database_ready
 from nautobot.extras.plugins import PluginConfig
@@ -38,7 +37,7 @@ class GoldenConfig(PluginConfig):
         "per_feature_height": 4,
         "get_custom_compliance": None,
         "jinja_env": {
-            "undefined": StrictUndefined,
+            "undefined": "jinja2.StrictUndefined",
             "trim_blocks": True,
             "lstrip_blocks": False,
         },

nautobot_golden_config/forms.py

@@ -338,7 +338,8 @@ class Meta:
 class GoldenConfigSettingForm(NautobotModelForm):
     """Filter Form for GoldenConfigSettingForm instances."""
 
-    dynamic_group = utilities_forms.DynamicModelChoiceField(queryset=DynamicGroup.objects.all(), required=False)
+    slug = SlugField()
+    dynamic_group = forms.ModelChoiceField(queryset=DynamicGroup.objects.all(), required=False)
 
     class Meta:
         """Filter Form Meta Data for GoldenConfigSettingForm instances."""

nautobot_golden_config/jobs.py

@@ -211,7 +211,7 @@ def run(self, data, commit):
             ComplianceJob().run.__func__(self, data, True)  # pylint: disable=too-many-function-args
 
 
-class AllDevicesGoldenConfig(Job):
+class AllDevicesGoldenConfig(Job, FormEntry):
     """Job to to run all three jobs against multiple devices."""
 
     class Meta:

nautobot_golden_config/nornir_plays/config_intended.py

@@ -18,7 +18,7 @@
 
 from nautobot_golden_config.models import GoldenConfig
 from nautobot_golden_config.nornir_plays.processor import ProcessGoldenConfig
-from nautobot_golden_config.utilities.constant import PLUGIN_CFG
+from nautobot_golden_config.utilities.constant import JINJA_ENV
 from nautobot_golden_config.utilities.db_management import close_threaded_db_connections
 from nautobot_golden_config.utilities.graphql import graph_ql_query
 from nautobot_golden_config.utilities.helper import (
@@ -32,7 +32,6 @@
 LOGGER = logging.getLogger(__name__)
 
 # Use a custom Jinja2 environment instead of Django's to avoid HTML escaping
-JINJA_ENV = PLUGIN_CFG["jinja_env"]
 jinja_env = SandboxedEnvironment(**JINJA_ENV)
 
 # Retrieve filters from the Django jinja template engine

nautobot_golden_config/tests/test_utilities/test_helpers.py

@@ -4,6 +4,7 @@
 
 from django.contrib.contenttypes.models import ContentType
 from django.test import TestCase
+from django.template import engines
 from jinja2 import exceptions as jinja_errors
 from nautobot.dcim.models import Device, Platform, Site
 from nautobot.extras.models import DynamicGroup, GitRepository, GraphQLQuery, Status, Tag
@@ -148,6 +149,17 @@ def test_render_jinja_template_success_with_filter(self, mock_device):
         rendered_template = render_jinja_template(mock_device, "logger", "{{ data | return_a }}")
         self.assertEqual(rendered_template, "a")
 
+    @patch("nautobot.dcim.models.Device")
+    def test_render_filters_work(self, mock_device):
+        """Test Jinja filters are still there."""
+        # This has failed because of import issues in the past, see #607 for an example failure and fix.
+        self.assertIn("is_ip", engines["jinja"].env.filters)
+        self.assertIn("humanize_speed", engines["jinja"].env.filters)
+        rendered_template = render_jinja_template(mock_device, "logger", "{{ '10.1.1.1' | is_ip }}")
+        self.assertEqual(rendered_template, "True")
+        rendered_template = render_jinja_template(mock_device, "logger", "{{ 100000 | humanize_speed }}")
+        self.assertEqual(rendered_template, "100 Mbps")
+
     @patch("nornir_nautobot.utils.logger.NornirLogger")
     @patch("nautobot.dcim.models.Device", spec=Device)
     def test_render_jinja_template_exceptions_undefined(self, mock_device, mock_nornir_logger):

nautobot_golden_config/utilities/constant.py

@@ -1,5 +1,6 @@
 """Storage of data that will not change throughout the life cycle of application."""
 from django.conf import settings
+from django.utils.module_loading import import_string
 
 PLUGIN_CFG = settings.PLUGINS_CONFIG["nautobot_golden_config"]
 
@@ -18,3 +19,9 @@
     "sotagg": ENABLE_SOTAGG,
     "postprocessing": ENABLE_POSTPROCESSING,
 }
+
+JINJA_ENV = PLUGIN_CFG["jinja_env"]
+if not JINJA_ENV.get("undefined"):
+    raise ValueError("The `jinja_env` setting did not include the required key for `undefined`.")
+if isinstance(JINJA_ENV["undefined"], str):
+    JINJA_ENV["undefined"] = import_string(JINJA_ENV["undefined"])

nautobot_golden_config/utilities/helper.py

@@ -171,6 +171,8 @@ def add_message(inbound):
     multiple_messages = []
     for item in inbound:
         job, request, feature_enabled = item
+        if not job:
+            continue
         if not isinstance(feature_enabled, list):
             feature_enabled = [feature_enabled]
         if not job.enabled and any(feature_enabled):

nautobot_golden_config/views.py

@@ -15,6 +15,7 @@
 from django.db.models import Count, ExpressionWrapper, F, FloatField, Max, ProtectedError, Q
 from django.forms import ModelMultipleChoiceField, MultipleHiddenInput
 from django.shortcuts import redirect, render
+from django.utils.module_loading import import_string
 from django.views.generic import View
 from django_pivot.pivot import pivot
 from nautobot.core.views import generic
@@ -31,7 +32,6 @@
 
 from nautobot_golden_config import filters, forms, models, tables
 from nautobot_golden_config.api import serializers
-from nautobot_golden_config.jobs import DeployConfigPlans
 from nautobot_golden_config.utilities import constant
 from nautobot_golden_config.utilities.config_postprocessing import get_config_postprocessing
 from nautobot_golden_config.utilities.graphql import graph_ql_query
@@ -59,7 +59,7 @@ class GoldenConfigListView(generic.ObjectListView):
 
     def extra_context(self):
         """Boilerplace code to modify data before returning."""
-        job = Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob")
+        job = Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob").first()
         add_message([[job, self.request, constant.ENABLE_COMPLIANCE]])
         return constant.CONFIG_FEATURES
 
@@ -219,7 +219,7 @@ def alter_queryset(self, request):
 
     def extra_context(self):
         """Boilerplate code to modify before returning data."""
-        job = Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob")
+        job = Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob").first()
         add_message([[job, self.request, constant.ENABLE_COMPLIANCE]])
         return {"compliance": constant.ENABLE_COMPLIANCE}
 
@@ -253,7 +253,7 @@ class ConfigComplianceView(generic.ObjectView):
 
     def get_extra_context(self, request, instance):
         """A Add extra data to detail view for Nautobot."""
-        job = Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob")
+        job = Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob").first()
         add_message([[job, request, constant.ENABLE_COMPLIANCE]])
         return {}
 
@@ -694,7 +694,7 @@ def get_global_aggr(self, request):
     def extra_context(self):
         """Extra content method on."""
         # add global aggregations to extra context.
-        job = Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob")
+        job = Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob").first()
         add_message([[job, self.request, constant.ENABLE_COMPLIANCE]])
         return self.extra_content
 
@@ -748,7 +748,7 @@ class ComplianceFeatureUIViewSet(NautobotUIViewSet):
 
     def get_extra_context(self, request, instance=None):
         """A ComplianceFeature helper function to warn if the Job is not enabled to run."""
-        job = Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob")
+        job = Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob").first()
         add_message([[job, request, constant.ENABLE_COMPLIANCE]])
         return {}
 
@@ -768,7 +768,7 @@ class ComplianceRuleUIViewSet(NautobotUIViewSet):
 
     def get_extra_context(self, request, instance=None):
         """A ComplianceRule helper function to warn if the Job is not enabled to run."""
-        job = Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob")
+        job = Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob").first()
         add_message([[job, request, constant.ENABLE_COMPLIANCE]])
         return {}
 
@@ -791,35 +791,37 @@ def get_extra_context(self, request, instance=None):
         jobs = []
         jobs.append(
             [
-                Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="BackupJob"),
+                Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="BackupJob").first(),
                 request,
                 constant.ENABLE_BACKUP,
             ]
         )
         jobs.append(
             [
-                Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="IntendedJob"),
+                Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="IntendedJob").first(),
                 request,
                 constant.ENABLE_INTENDED,
             ]
         )
         jobs.append(
             [
-                Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="DeployConfigPlans"),
+                Job.objects.filter(
+                    module_name="nautobot_golden_config.jobs", job_class_name="DeployConfigPlans"
+                ).first(),
                 request,
                 constant.ENABLE_DEPLOY,
             ]
         )
         jobs.append(
             [
-                Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob"),
+                Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob").first(),
                 request,
                 constant.ENABLE_COMPLIANCE,
             ]
         )
         jobs.append(
             [
-                Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="AllGoldenConfig"),
+                Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="AllGoldenConfig").first(),
                 request,
                 [
                     constant.ENABLE_BACKUP,
@@ -832,7 +834,9 @@ def get_extra_context(self, request, instance=None):
         )
         jobs.append(
             [
-                Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="AllDevicesGoldenConfig"),
+                Job.objects.filter(
+                    module_name="nautobot_golden_config.jobs", job_class_name="AllDevicesGoldenConfig"
+                ).first(),
                 request,
                 [
                     constant.ENABLE_BACKUP,
@@ -862,7 +866,7 @@ class ConfigRemoveUIViewSet(NautobotUIViewSet):
 
     def get_extra_context(self, request, instance=None):
         """A ConfigRemove helper function to warn if the Job is not enabled to run."""
-        job = Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="BackupJob")
+        job = Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="BackupJob").first()
         add_message([[job, request, constant.ENABLE_BACKUP]])
         return {}
 
@@ -882,7 +886,7 @@ class ConfigReplaceUIViewSet(NautobotUIViewSet):
 
     def get_extra_context(self, request, instance=None):
         """A ConfigReplace helper function to warn if the Job is not enabled to run."""
-        job = Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="BackupJob")
+        job = Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="BackupJob").first()
         add_message([[job, request, constant.ENABLE_BACKUP]])
         return {}
 
@@ -902,7 +906,7 @@ class RemediationSettingUIViewSet(NautobotUIViewSet):
 
     def get_extra_context(self, request, instance=None):
         """A RemediationSetting helper function to warn if the Job is not enabled to run."""
-        job = Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob")
+        job = Job.objects.filter(module_name="nautobot_golden_config.jobs", job_class_name="ComplianceJob").first()
         add_message([[job, request, constant.ENABLE_COMPLIANCE]])
         return {}
 
@@ -931,23 +935,27 @@ def get_extra_context(self, request, instance=None):
         jobs = []
         jobs.append(
             [
-                Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="GenerateConfigPlans"),
+                Job.objects.filter(
+                    module_name="nautobot_golden_config.jobs", job_class_name="GenerateConfigPlans"
+                ).first(),
                 request,
                 constant.ENABLE_PLAN,
             ]
         )
         jobs.append(
             [
-                Job.objects.get(module_name="nautobot_golden_config.jobs", job_class_name="DeployConfigPlans"),
+                Job.objects.filter(
+                    module_name="nautobot_golden_config.jobs", job_class_name="DeployConfigPlans"
+                ).first(),
                 request,
                 constant.ENABLE_DEPLOY,
             ]
         )
         jobs.append(
             [
-                Job.objects.get(
+                Job.objects.filter(
                     module_name="nautobot_golden_config.jobs", job_class_name="DeployConfigPlanJobButtonReceiver"
-                ),
+                ).first(),
                 request,
                 constant.ENABLE_DEPLOY,
             ]
@@ -976,7 +984,7 @@ def post(self, request):
 
         result = JobResult.enqueue_job(
             func=run_job,
-            name=DeployConfigPlans.class_path,
+            name=import_string("nautobot_golden_config.jobs.DeployConfigPlans").class_path,
             obj_type=get_job_content_type(),
             user=request.user,
             data=job_data,