Closes #11559: Implement config template rendering (#11769)

* WIP

* Add config_template field to Device

* Pre-fetch referenced templates

* Correct up_to_date callable

* Add config_template FK to Device

* Update & merge migrations

* Add config_template FK to Platform

* Add tagging support for ConfigTemplate

* Catch exceptions when rendering device templates in UI

* Refactor ConfigTemplate.render()

* Add support for returning plain text content

* Add ConfigTemplate model documentation

* Add feature documentation for config rendering

Author: jeremystretch

docs/features/configuration-rendering.md

@@ -0,0 +1,38 @@
+# Configuration Rendering
+
+One of the critical aspects of operating a network is ensuring that every network node is configured correctly. By leveraging configuration templates and [context data](./context-data.md), NetBox can render complete configuration files for each device on your network.
+
+```mermaid
+flowchart TD
+    ConfigContext & ConfigTemplate --> Config{{Rendered configuration}}
+
+click ConfigContext "../../models/extras/configcontext/"
+click ConfigTemplate "../../models/extras/configtemplate/"
+```
+
+## Configuration Templates
+
+Configuration templates are written in the [Jinja2 templating language](https://jinja.palletsprojects.com/), and may be automatically populated from remote data sources. Context data is applied to a template during rendering to output a complete configuration file. Below is an example template.
+
+```jinja2
+{% extends 'base.j2' %}
+
+{% block content %}
+    system {
+        host-name {{ device.name }};
+        domain-name example.com;
+        time-zone UTC;
+        authentication-order [ password radius ];
+        ntp {
+            {% for server in ntp_servers %}
+                server {{ server }};
+            {% endfor %}
+        }
+    }
+    {% for interface in device.interfaces.all() %}
+        {% include 'common/interface.j2' %}
+    {% endfor %}
+{% endblock %}
+```
+
+When rendered for a specific NetBox device, the template's `device` variable will be populated with the device instance, and `ntp_servers` will be pulled from the device's available context data. The resulting output will be a valid configuration segment that can be applied directly to a compatible network device.

docs/features/context-data.md

@@ -11,6 +11,8 @@ Configuration context data (or "config contexts" for short) is a powerful featur
 }
 ```
 
+Context data can be consumed by remote API clients, or it can be employed natively to render [configuration templates](./configuration-rendering.md).
+
 Config contexts can be computed for objects based on the following criteria:
 
 | Type          | Devices          | Virtual Machines |

docs/models/dcim/device.md

@@ -72,6 +72,10 @@ The device's operational status.
 
 A device may be associated with a particular [platform](./platform.md) to indicate its operating system. Note that only platforms assigned to the associated manufacturer (or to no manufacturer) will be available for selection.
 
+### Configuration Template
+
+The [configuration template](../extras/configtemplate.md) from which the configuration for this device can be rendered. If set, this will override any config template referenced by the device's role or platform.
+
 ### Primary IPv4 & IPv6 Addresses
 
 Each device may designate one primary IPv4 address and/or one primary IPv6 address for management purposes.

docs/models/dcim/devicerole.md

@@ -19,3 +19,7 @@ The color used when displaying the role in the NetBox UI.
 ### VM Role
 
 If selected, this role may be assigned to [virtual machines](../virtualization/virtualmachine.md)
+
+### Configuration Template
+
+The default [configuration template](../extras/configtemplate.md) for devices assigned to this role.

docs/models/dcim/platform.md

@@ -22,6 +22,10 @@ A unique URL-friendly identifier. (This value can be used for filtering.)
 
 If designated, this platform will be available for use only to devices assigned to this [manufacturer](./manufacturer.md). This can be handy e.g. for limiting network operating systems to use on hardware produced by the relevant vendor. However, it should not be used when defining general-purpose software platforms.
 
+### Configuration Template
+
+The default [configuration template](../extras/configtemplate.md) for devices assigned to this platform.
+
 ### NAPALM Driver
 
 The [NAPALM driver](https://napalm.readthedocs.io/en/latest/support/index.html) associated with this platform.

docs/models/extras/configtemplate.md

@@ -0,0 +1,29 @@
+# Configuration Templates
+
+Configuration templates can be used to render [devices](../dcim/device.md) configurations from [context data](../../features/context-data.md). Templates are written in the [Jinja2 language](https://jinja.palletsprojects.com/) and can be associated with devices roles, platforms, and/or individual devices.
+
+Context data is made available to [devices](../dcim/device.md) and/or [virtual machines](../virtualization/virtualmachine.md) based on their relationships to other objects in NetBox. For example, context data can be associated only with devices assigned to a particular site, or only to virtual machines in a certain cluster.
+
+See the [configuration rendering documentation](../../features/configuration-rendering.md) for more information.
+
+## Fields
+
+### Name
+
+A unique human-friendly name.
+
+### Weight
+
+A numeric value which influences the order in which context data is merged. Contexts with a lower weight are merged before those with a higher weight.
+
+### Data File
+
+Template code may optionally be sourced from a remote [data file](../core/datafile.md), which is synchronized from a remote data source. When designating a data file, there is no need to specify template code: It will be populated automatically from the data file.
+
+### Template Code
+
+Jinja2 template code, if being defined locally rather than replicated from a data file.
+
+### Environment Parameters
+
+A dictionary of any additional parameters to pass when instantiating the [Jinja2 environment](https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.Environment). Jinja2 supports various optional parameters which can be used to modify its default behavior.

mkdocs.yml

@@ -74,6 +74,7 @@ nav:
         - Contacts: 'features/contacts.md'
         - Search: 'features/search.md'
         - Context Data: 'features/context-data.md'
+        - Configuration Rendering: 'features/configuration-rendering.md'
         - Change Logging: 'features/change-logging.md'
         - Journaling: 'features/journaling.md'
         - Auth & Permissions: 'features/authentication-permissions.md'
@@ -196,6 +197,7 @@ nav:
         - Extras:
             - Branch: 'models/extras/branch.md'
             - ConfigContext: 'models/extras/configcontext.md'
+            - ConfigTemplate: 'models/extras/configtemplate.md'
             - CustomField: 'models/extras/customfield.md'
             - CustomLink: 'models/extras/customlink.md'
             - ExportTemplate: 'models/extras/exporttemplate.md'

netbox/dcim/api/serializers.py

@@ -9,6 +9,7 @@
 from dcim.choices import *
 from dcim.constants import *
 from dcim.models import *
+from extras.api.nested_serializers import NestedConfigTemplateSerializer
 from ipam.api.nested_serializers import (
     NestedASNSerializer, NestedIPAddressSerializer, NestedL2VPNTerminationSerializer, NestedVLANSerializer,
     NestedVRFSerializer,
@@ -605,8 +606,8 @@ class DeviceRoleSerializer(NetBoxModelSerializer):
     class Meta:
         model = DeviceRole
         fields = [
-            'id', 'url', 'display', 'name', 'slug', 'color', 'vm_role', 'description', 'tags', 'custom_fields',
-            'created', 'last_updated', 'device_count', 'virtualmachine_count',
+            'id', 'url', 'display', 'name', 'slug', 'color', 'vm_role', 'config_template', 'description', 'tags',
+            'custom_fields', 'created', 'last_updated', 'device_count', 'virtualmachine_count',
         ]
 
 
@@ -619,8 +620,8 @@ class PlatformSerializer(NetBoxModelSerializer):
     class Meta:
         model = Platform
         fields = [
-            'id', 'url', 'display', 'name', 'slug', 'manufacturer', 'napalm_driver', 'napalm_args', 'description',
-            'tags', 'custom_fields', 'created', 'last_updated', 'device_count', 'virtualmachine_count',
+            'id', 'url', 'display', 'name', 'slug', 'manufacturer', 'config_template', 'napalm_driver', 'napalm_args',
+            'description', 'tags', 'custom_fields', 'created', 'last_updated', 'device_count', 'virtualmachine_count',
         ]
 
 
@@ -651,14 +652,15 @@ class DeviceSerializer(NetBoxModelSerializer):
     cluster = NestedClusterSerializer(required=False, allow_null=True)
     virtual_chassis = NestedVirtualChassisSerializer(required=False, allow_null=True, default=None)
     vc_position = serializers.IntegerField(allow_null=True, max_value=255, min_value=0, default=None)
+    config_template = NestedConfigTemplateSerializer(required=False, allow_null=True, default=None)
 
     class Meta:
         model = Device
         fields = [
             'id', 'url', 'display', 'name', 'device_type', 'device_role', 'tenant', 'platform', 'serial', 'asset_tag',
             'site', 'location', 'rack', 'position', 'face', 'parent_device', 'status', 'airflow', 'primary_ip',
             'primary_ip4', 'primary_ip6', 'cluster', 'virtual_chassis', 'vc_position', 'vc_priority', 'description',
-            'comments', 'local_context_data', 'tags', 'custom_fields', 'created', 'last_updated',
+            'comments', 'config_template', 'local_context_data', 'tags', 'custom_fields', 'created', 'last_updated',
         ]
 
     @swagger_serializer_method(serializer_or_field=NestedDeviceSerializer)

netbox/dcim/api/views.py

@@ -366,7 +366,7 @@ class InventoryItemTemplateViewSet(NetBoxModelViewSet):
 #
 
 class DeviceRoleViewSet(NetBoxModelViewSet):
-    queryset = DeviceRole.objects.prefetch_related('tags').annotate(
+    queryset = DeviceRole.objects.prefetch_related('config_template', 'tags').annotate(
         device_count=count_related(Device, 'device_role'),
         virtualmachine_count=count_related(VirtualMachine, 'role')
     )
@@ -379,7 +379,7 @@ class DeviceRoleViewSet(NetBoxModelViewSet):
 #
 
 class PlatformViewSet(NetBoxModelViewSet):
-    queryset = Platform.objects.prefetch_related('tags').annotate(
+    queryset = Platform.objects.prefetch_related('config_template', 'tags').annotate(
         device_count=count_related(Device, 'platform'),
         virtualmachine_count=count_related(VirtualMachine, 'platform')
     )

netbox/dcim/filtersets.py

@@ -3,6 +3,7 @@
 from django.utils.translation import gettext as _
 
 from extras.filtersets import LocalConfigContextFilterSet
+from extras.models import ConfigTemplate
 from ipam.models import ASN, L2VPN, IPAddress, VRF
 from netbox.filtersets import (
     BaseFilterSet, ChangeLoggedModelFilterSet, OrganizationalModelFilterSet, NetBoxModelFilterSet,
@@ -776,6 +777,10 @@ def search(self, queryset, name, value):
 
 
 class DeviceRoleFilterSet(OrganizationalModelFilterSet):
+    config_template_id = django_filters.ModelMultipleChoiceFilter(
+        queryset=ConfigTemplate.objects.all(),
+        label=_('Config template (ID)'),
+    )
 
     class Meta:
         model = DeviceRole
@@ -794,6 +799,10 @@ class PlatformFilterSet(OrganizationalModelFilterSet):
         to_field_name='slug',
         label=_('Manufacturer (slug)'),
     )
+    config_template_id = django_filters.ModelMultipleChoiceFilter(
+        queryset=ConfigTemplate.objects.all(),
+        label=_('Config template (ID)'),
+    )
 
     class Meta:
         model = Platform
@@ -936,6 +945,10 @@ class DeviceFilterSet(NetBoxModelFilterSet, TenancyFilterSet, ContactModelFilter
         method='_virtual_chassis_member',
         label=_('Is a virtual chassis member')
     )
+    config_template_id = django_filters.ModelMultipleChoiceFilter(
+        queryset=ConfigTemplate.objects.all(),
+        label=_('Config template (ID)'),
+    )
     console_ports = django_filters.BooleanFilter(
         method='_console_ports',
         label=_('Has console ports'),

netbox/dcim/forms/bulk_edit.py

@@ -6,6 +6,7 @@
 from dcim.choices import *
 from dcim.constants import *
 from dcim.models import *
+from extras.models import ConfigTemplate
 from ipam.models import ASN, VLAN, VLANGroup, VRF
 from netbox.forms import NetBoxModelBulkEditForm
 from tenancy.models import Tenant
@@ -454,16 +455,20 @@ class DeviceRoleBulkEditForm(NetBoxModelBulkEditForm):
         widget=BulkEditNullBooleanSelect,
         label=_('VM role')
     )
+    config_template = DynamicModelChoiceField(
+        queryset=ConfigTemplate.objects.all(),
+        required=False
+    )
     description = forms.CharField(
         max_length=200,
         required=False
     )
 
     model = DeviceRole
     fieldsets = (
-        (None, ('color', 'vm_role', 'description')),
+        (None, ('color', 'vm_role', 'config_template', 'description')),
     )
-    nullable_fields = ('color', 'description')
+    nullable_fields = ('color', 'config_template', 'description')
 
 
 class PlatformBulkEditForm(NetBoxModelBulkEditForm):
@@ -475,17 +480,20 @@ class PlatformBulkEditForm(NetBoxModelBulkEditForm):
         max_length=50,
         required=False
     )
-    # TODO: Bulk edit support for napalm_args
+    config_template = DynamicModelChoiceField(
+        queryset=ConfigTemplate.objects.all(),
+        required=False
+    )
     description = forms.CharField(
         max_length=200,
         required=False
     )
 
     model = Platform
     fieldsets = (
-        (None, ('manufacturer', 'napalm_driver', 'description')),
+        (None, ('manufacturer', 'config_template', 'napalm_driver', 'description')),
     )
-    nullable_fields = ('manufacturer', 'napalm_driver', 'description')
+    nullable_fields = ('manufacturer', 'config_template', 'napalm_driver', 'description')
 
 
 class DeviceBulkEditForm(NetBoxModelBulkEditForm):
@@ -540,6 +548,10 @@ class DeviceBulkEditForm(NetBoxModelBulkEditForm):
         max_length=200,
         required=False
     )
+    config_template = DynamicModelChoiceField(
+        queryset=ConfigTemplate.objects.all(),
+        required=False
+    )
     comments = CommentField(
         widget=forms.Textarea,
         label='Comments'
@@ -550,6 +562,7 @@ class DeviceBulkEditForm(NetBoxModelBulkEditForm):
         ('Device', ('device_role', 'status', 'tenant', 'platform', 'description')),
         ('Location', ('site', 'location')),
         ('Hardware', ('manufacturer', 'device_type', 'airflow', 'serial')),
+        ('Configuration', ('config_template',)),
     )
     nullable_fields = (
         'location', 'tenant', 'platform', 'serial', 'airflow', 'description', 'comments',

netbox/dcim/forms/bulk_import.py

@@ -8,6 +8,7 @@
 from dcim.choices import *
 from dcim.constants import *
 from dcim.models import *
+from extras.models import ConfigTemplate
 from ipam.models import VRF
 from netbox.forms import NetBoxModelImportForm
 from tenancy.models import Tenant
@@ -307,11 +308,17 @@ class Meta:
 
 
 class DeviceRoleImportForm(NetBoxModelImportForm):
+    config_template = CSVModelChoiceField(
+        queryset=ConfigTemplate.objects.all(),
+        to_field_name='name',
+        required=False,
+        help_text=_('Config template')
+    )
     slug = SlugField()
 
     class Meta:
         model = DeviceRole
-        fields = ('name', 'slug', 'color', 'vm_role', 'description', 'tags')
+        fields = ('name', 'slug', 'color', 'vm_role', 'config_template', 'description', 'tags')
         help_texts = {
             'color': mark_safe(_('RGB color in hexadecimal (e.g. <code>00ff00</code>)')),
         }
@@ -325,10 +332,18 @@ class PlatformImportForm(NetBoxModelImportForm):
         to_field_name='name',
         help_text=_('Limit platform assignments to this manufacturer')
     )
+    config_template = CSVModelChoiceField(
+        queryset=ConfigTemplate.objects.all(),
+        to_field_name='name',
+        required=False,
+        help_text=_('Config template')
+    )
 
     class Meta:
         model = Platform
-        fields = ('name', 'slug', 'manufacturer', 'napalm_driver', 'napalm_args', 'description', 'tags')
+        fields = (
+            'name', 'slug', 'manufacturer', 'config_template', 'napalm_driver', 'napalm_args', 'description', 'tags',
+        )
 
 
 class BaseDeviceImportForm(NetBoxModelImportForm):
@@ -434,12 +449,18 @@ class DeviceImportForm(BaseDeviceImportForm):
         required=False,
         help_text=_('Airflow direction')
     )
+    config_template = CSVModelChoiceField(
+        queryset=ConfigTemplate.objects.all(),
+        to_field_name='name',
+        required=False,
+        help_text=_('Config template')
+    )
 
     class Meta(BaseDeviceImportForm.Meta):
         fields = [
             'name', 'device_role', 'tenant', 'manufacturer', 'device_type', 'platform', 'serial', 'asset_tag', 'status',
             'site', 'location', 'rack', 'position', 'face', 'parent', 'device_bay', 'airflow', 'virtual_chassis',
-            'vc_position', 'vc_priority', 'cluster', 'description', 'comments', 'tags',
+            'vc_position', 'vc_priority', 'cluster', 'description', 'config_template', 'comments', 'tags',
         ]
 
     def __init__(self, data=None, *args, **kwargs):