Merge branch 'develop' into feature

Author: jeremystretch

.github/ISSUE_TEMPLATE/bug_report.yaml

@@ -14,7 +14,7 @@ body:
     attributes:
       label: NetBox version
       description: What version of NetBox are you currently running?
-      placeholder: v3.5.8
+      placeholder: v3.5.9
     validations:
       required: true
   - type: dropdown

.github/ISSUE_TEMPLATE/feature_request.yaml

@@ -14,7 +14,7 @@ body:
     attributes:
       label: NetBox version
       description: What version of NetBox are you currently running?
-      placeholder: v3.5.8
+      placeholder: v3.5.9
     validations:
       required: true
   - type: dropdown

contrib/generated_schema.json

@@ -332,6 +332,7 @@
                         "100gbase-x-cfp",
                         "100gbase-x-cfp2",
                         "200gbase-x-cfp2",
+                        "400gbase-x-cfp2",
                         "100gbase-x-cfp4",
                         "100gbase-x-cxp",
                         "100gbase-x-cpak",

docs/customization/reports.md

@@ -111,7 +111,7 @@ The following methods are available to log results within a report:
 
 The recording of one or more failure messages will automatically flag a report as failed. It is advised to log a success for each object that is evaluated so that the results will reflect how many objects are being reported on. (The inclusion of a log message is optional for successes.) Messages recorded with `log()` will appear in a report's results but are not associated with a particular object or status. Log messages also support using markdown syntax and will be rendered on the report result page.
 
-To perform additional tasks, such as sending an email or calling a webhook, before or after a report is run, extend the `pre_run()` and/or `post_run()` methods, respectively. The status of a completed report is available as `self.failed` and the results object is `self.result`.
+To perform additional tasks, such as sending an email or calling a webhook, before or after a report is run, extend the `pre_run()` and/or `post_run()` methods, respectively.
 
 By default, reports within a module are ordered alphabetically in the reports list page. To return reports in a specific order, you can define the `report_order` variable at the end of your module. The `report_order` variable is a tuple which contains each Report class in the desired order. Any reports that are omitted from this list will be listed last.
 

docs/installation/upgrading.md

@@ -59,7 +59,7 @@ Copy `local_requirements.txt`, `configuration.py`, and `ldap_config.py` (if pres
 
 ```no-highlight
 # Set $OLDVER to the NetBox version currently installed
-NEWVER=3.4.9
+OLDVER=3.4.9
 sudo cp /opt/netbox-$OLDVER/local_requirements.txt /opt/netbox/
 sudo cp /opt/netbox-$OLDVER/netbox/netbox/configuration.py /opt/netbox/netbox/netbox/
 sudo cp /opt/netbox-$OLDVER/netbox/netbox/ldap_config.py /opt/netbox/netbox/netbox/

docs/reference/filtering.md

@@ -61,13 +61,14 @@ These lookup expressions can be applied by adding a suffix to the desired field'
 
 Numeric based fields (ASN, VLAN ID, etc) support these lookup expressions:
 
-| Filter | Description |
-|--------|-------------|
-| `n` | Not equal to |
-| `lt` | Less than |
-| `lte` | Less than or equal to |
-| `gt` | Greater than |
-| `gte` | Greater than or equal to |
+| Filter  | Description              |
+|---------|--------------------------|
+| `n`     | Not equal to             |
+| `lt`    | Less than                |
+| `lte`   | Less than or equal to    |
+| `gt`    | Greater than             |
+| `gte`   | Greater than or equal to |
+| `empty` | Is empty/null (boolean)  |
 
 Here is an example of a numeric field lookup expression that will return all VLANs with a VLAN ID greater than 900:
 
@@ -79,18 +80,18 @@ GET /api/ipam/vlans/?vid__gt=900
 
 String based (char) fields (Name, Address, etc) support these lookup expressions:
 
-| Filter | Description |
-|--------|-------------|
-| `n` | Not equal to |
-| `ic` | Contains (case-insensitive) |
-| `nic` | Does not contain (case-insensitive) |
-| `isw` | Starts with (case-insensitive) |
-| `nisw` | Does not start with (case-insensitive) |
-| `iew` | Ends with (case-insensitive) |
-| `niew` | Does not end with (case-insensitive) |
-| `ie` | Exact match (case-insensitive) |
-| `nie` | Inverse exact match (case-insensitive) |
-| `empty` | Is empty (boolean) |
+| Filter  | Description                            |
+|---------|----------------------------------------|
+| `n`     | Not equal to                           |
+| `ic`    | Contains (case-insensitive)            |
+| `nic`   | Does not contain (case-insensitive)    |
+| `isw`   | Starts with (case-insensitive)         |
+| `nisw`  | Does not start with (case-insensitive) |
+| `iew`   | Ends with (case-insensitive)           |
+| `niew`  | Does not end with (case-insensitive)   |
+| `ie`    | Exact match (case-insensitive)         |
+| `nie`   | Inverse exact match (case-insensitive) |
+| `empty` | Is empty/null (boolean)                |
 
 Here is an example of a lookup expression on a string field that will return all devices with `switch` in the name:
 

docs/release-notes/version-3.5.md

@@ -1,6 +1,32 @@
 # NetBox v3.5
 
-## v3.5.9 (FUTURE)
+## v3.5.9 (2023-08-28)
+
+### Enhancements
+
+* [#12489](https://github.com/netbox-community/netbox/issues/12489) - Dynamically render location and device lists under site and location views
+* [#12825](https://github.com/netbox-community/netbox/issues/12825) - Display assigned values count per obejct type under custom field view
+* [#13313](https://github.com/netbox-community/netbox/issues/13313) - Enable filtering IP ranges by containing prefix
+* [#13415](https://github.com/netbox-community/netbox/issues/13415) - Include request object in custom link renderer on tables
+* [#13536](https://github.com/netbox-community/netbox/issues/13536) - Move child VLANs list to a separate tab under VLAN group view
+* [#13542](https://github.com/netbox-community/netbox/issues/13542) - Pass additional HTTP headers through to custom script context
+* [#13585](https://github.com/netbox-community/netbox/issues/13585) - Introduce `empty` lookup for numeric value filters
+
+### Bug Fixes
+
+* [#11272](https://github.com/netbox-community/netbox/issues/11272) - Fix localization support for device position field
+* [#13358](https://github.com/netbox-community/netbox/issues/13358) - Git backend should send HTTP auth headers only if credentials have been defined
+* [#13477](https://github.com/netbox-community/netbox/issues/13477) - Fix filtering of modified objects after bulk import/update
+* [#13478](https://github.com/netbox-community/netbox/issues/13478) - Fix filtering of export templates by content type under web UI
+* [#13500](https://github.com/netbox-community/netbox/issues/13500) - Fix form validation for bulk update of L2VPN terminations via bulk import form
+* [#13503](https://github.com/netbox-community/netbox/issues/13503) - Fix utilization graph proportions when localization is enabled
+* [#13507](https://github.com/netbox-community/netbox/issues/13507) - Avoid raising exception for invalid content type during global search
+* [#13516](https://github.com/netbox-community/netbox/issues/13516) - Plugin utility functions should be importable from `extras.plugins`
+* [#13530](https://github.com/netbox-community/netbox/issues/13530) - Ensure script log messages can be serialized as JSON data
+* [#13543](https://github.com/netbox-community/netbox/issues/13543) - Config context tab under device/VM view should not require `extras.view_configcontext` permission
+* [#13544](https://github.com/netbox-community/netbox/issues/13544) - Ensure `reindex` command clears all cached values when not in lazy mode
+* [#13556](https://github.com/netbox-community/netbox/issues/13556) - Correct REST API representation of VDC status choice
+* [#13569](https://github.com/netbox-community/netbox/issues/13569) - Fix selection widgets for related interfaces when bulk editing interfaces under device view
 
 ---
 

netbox/core/data_backends.py

@@ -125,12 +125,13 @@ def fetch(self):
         }
 
         if self.url_scheme in ('http', 'https'):
-            clone_args.update(
-                {
-                    "username": self.params.get('username'),
-                    "password": self.params.get('password'),
-                }
-            )
+            if self.params.get('username'):
+                clone_args.update(
+                    {
+                        "username": self.params.get('username'),
+                        "password": self.params.get('password'),
+                    }
+                )
 
         logger.debug(f"Cloning git repo: {self.url}")
         try:

netbox/dcim/api/serializers.py

@@ -758,6 +758,7 @@ class VirtualDeviceContextSerializer(NetBoxModelSerializer):
     primary_ip = NestedIPAddressSerializer(read_only=True, allow_null=True)
     primary_ip4 = NestedIPAddressSerializer(required=False, allow_null=True)
     primary_ip6 = NestedIPAddressSerializer(required=False, allow_null=True)
+    status = ChoiceField(choices=VirtualDeviceContextStatusChoices)
 
     # Related object counts
     interface_count = serializers.IntegerField(read_only=True)

netbox/dcim/forms/model_forms.py

@@ -421,12 +421,13 @@ class DeviceForm(TenancyForm, NetBoxModelForm):
         label=_('Position'),
         required=False,
         help_text=_("The lowest-numbered unit occupied by the device"),
+        localize=True,
         widget=APISelect(
             api_url='/api/dcim/racks/{{rack}}/elevation/',
             attrs={
                 'disabled-indicator': 'device',
                 'data-dynamic-params': '[{"fieldName":"face","queryParam":"face"}]'
-            }
+            },
         )
     )
     device_type = DynamicModelChoiceField(

netbox/dcim/views.py

@@ -398,32 +398,8 @@ def get_extra_context(self, request, instance):
             (Circuit.objects.restrict(request.user, 'view').filter(terminations__site=instance).distinct(), 'site_id'),
         )
 
-        locations = Location.objects.add_related_count(
-            Location.objects.all(),
-            Rack,
-            'location',
-            'rack_count',
-            cumulative=True
-        )
-        locations = Location.objects.add_related_count(
-            locations,
-            Device,
-            'location',
-            'device_count',
-            cumulative=True
-        ).restrict(request.user, 'view').filter(site=instance)
-
-        nonracked_devices = Device.objects.filter(
-            site=instance,
-            rack__isnull=True,
-            parent_bay__isnull=True
-        ).prefetch_related('device_type__manufacturer', 'parent_bay', 'role')
-
         return {
             'related_models': related_models,
-            'locations': locations,
-            'nonracked_devices': nonracked_devices.order_by('-pk')[:10],
-            'total_nonracked_devices_count': nonracked_devices.count(),
         }
 
 
@@ -495,16 +471,8 @@ def get_extra_context(self, request, instance):
             (Device.objects.restrict(request.user, 'view').filter(location__in=locations), 'location_id'),
         )
 
-        nonracked_devices = Device.objects.filter(
-            location=instance,
-            rack__isnull=True,
-            parent_bay__isnull=True
-        ).prefetch_related('device_type__manufacturer', 'parent_bay', 'role')
-
         return {
             'related_models': related_models,
-            'nonracked_devices': nonracked_devices.order_by('-pk')[:10],
-            'total_nonracked_devices_count': nonracked_devices.count(),
         }
 
 
@@ -2055,7 +2023,6 @@ class DeviceConfigContextView(ObjectConfigContextView):
     base_template = 'dcim/device/base.html'
     tab = ViewTab(
         label=_('Config Context'),
-        permission='extras.view_configcontext',
         weight=2000
     )
 

netbox/extras/forms/filtersets.py

@@ -136,7 +136,7 @@ class ExportTemplateFilterForm(SavedFiltersMixin, FilterForm):
     fieldsets = (
         (None, ('q', 'filter_id')),
         (_('Data'), ('data_source_id', 'data_file_id')),
-        (_('Attributes'), ('content_types', 'mime_type', 'file_extension', 'as_attachment')),
+        (_('Attributes'), ('content_type_id', 'mime_type', 'file_extension', 'as_attachment')),
     )
     data_source_id = DynamicModelMultipleChoiceField(
         queryset=DataSource.objects.all(),
@@ -151,10 +151,10 @@ class ExportTemplateFilterForm(SavedFiltersMixin, FilterForm):
             'source_id': '$data_source_id'
         }
     )
-    content_types = ContentTypeMultipleChoiceField(
-        label=_('Content types'),
+    content_type_id = ContentTypeMultipleChoiceField(
         queryset=ContentType.objects.filter(FeatureQuery('export_templates').get_query()),
-        required=False
+        required=False,
+        label=_('Content types')
     )
     mime_type = forms.CharField(
         required=False,

netbox/extras/management/commands/reindex.py

@@ -69,10 +69,7 @@ def handle(self, *model_labels, **kwargs):
         if not kwargs['lazy']:
             self.stdout.write('Clearing cached values... ', ending='')
             self.stdout.flush()
-            content_types = [
-                ContentType.objects.get_for_model(model) for model in indexers.keys()
-            ]
-            deleted_count = search_backend.clear(content_types)
+            deleted_count = search_backend.clear()
             self.stdout.write(f'{deleted_count} entries deleted.')
 
         # Index models

netbox/extras/plugins/__init__.py

@@ -11,6 +11,7 @@
 from .navigation import *
 from .registration import *
 from .templates import *
+from .utils import *
 
 # Initialize plugin registry
 registry['plugins'].update({

netbox/extras/scripts.py

@@ -401,23 +401,23 @@ def as_form(self, data=None, files=None, initial=None):
 
     def log_debug(self, message):
         self.logger.log(logging.DEBUG, message)
-        self.log.append((LogLevelChoices.LOG_DEFAULT, message))
+        self.log.append((LogLevelChoices.LOG_DEFAULT, str(message)))
 
     def log_success(self, message):
         self.logger.log(logging.INFO, message)  # No syslog equivalent for SUCCESS
-        self.log.append((LogLevelChoices.LOG_SUCCESS, message))
+        self.log.append((LogLevelChoices.LOG_SUCCESS, str(message)))
 
     def log_info(self, message):
         self.logger.log(logging.INFO, message)
-        self.log.append((LogLevelChoices.LOG_INFO, message))
+        self.log.append((LogLevelChoices.LOG_INFO, str(message)))
 
     def log_warning(self, message):
         self.logger.log(logging.WARNING, message)
-        self.log.append((LogLevelChoices.LOG_WARNING, message))
+        self.log.append((LogLevelChoices.LOG_WARNING, str(message)))
 
     def log_failure(self, message):
         self.logger.log(logging.ERROR, message)
-        self.log.append((LogLevelChoices.LOG_FAILURE, message))
+        self.log.append((LogLevelChoices.LOG_FAILURE, str(message)))
 
     # Convenience functions
 

netbox/extras/tests/test_filtersets.py

@@ -1109,11 +1109,13 @@ def setUpTestData(cls):
             Site(name='Site 1', slug='site-1'),
             Site(name='Site 2', slug='site-2'),
             Site(name='Site 3', slug='site-3'),
+            Site(name='Site 4', slug='site-4'),
         )
         Site.objects.bulk_create(sites)
 
         # Simulate *creation* changelog records for two of the sites
         request_id = uuid.uuid4()
+        cls.create_request_id = request_id
         objectchanges = (
             ObjectChange(
                 changed_object_type=content_type,
@@ -1132,6 +1134,7 @@ def setUpTestData(cls):
 
         # Simulate *update* changelog records for two of the sites
         request_id = uuid.uuid4()
+        cls.update_request_id = request_id
         objectchanges = (
             ObjectChange(
                 changed_object_type=content_type,
@@ -1148,14 +1151,36 @@ def setUpTestData(cls):
         )
         ObjectChange.objects.bulk_create(objectchanges)
 
+        # Simulate *create* and *update* changelog records for two of the sites
+        request_id = uuid.uuid4()
+        cls.create_update_request_id = request_id
+        objectchanges = (
+            ObjectChange(
+                changed_object_type=content_type,
+                changed_object_id=sites[2].pk,
+                action=ObjectChangeActionChoices.ACTION_CREATE,
+                request_id=request_id
+            ),
+            ObjectChange(
+                changed_object_type=content_type,
+                changed_object_id=sites[3].pk,
+                action=ObjectChangeActionChoices.ACTION_UPDATE,
+                request_id=request_id
+            ),
+        )
+        ObjectChange.objects.bulk_create(objectchanges)
+
     def test_created_by_request(self):
-        request_id = ObjectChange.objects.filter(action=ObjectChangeActionChoices.ACTION_CREATE).first().request_id
-        params = {'created_by_request': request_id}
+        params = {'created_by_request': self.create_request_id}
         self.assertEqual(self.filterset(params, self.queryset).qs.count(), 2)
-        self.assertEqual(self.queryset.count(), 3)
+        self.assertEqual(self.queryset.count(), 4)
 
     def test_updated_by_request(self):
-        request_id = ObjectChange.objects.filter(action=ObjectChangeActionChoices.ACTION_UPDATE).first().request_id
-        params = {'updated_by_request': request_id}
+        params = {'updated_by_request': self.update_request_id}
+        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 2)
+        self.assertEqual(self.queryset.count(), 4)
+
+    def test_modified_by_request(self):
+        params = {'modified_by_request': self.create_update_request_id}
         self.assertEqual(self.filterset(params, self.queryset).qs.count(), 2)
-        self.assertEqual(self.queryset.count(), 3)
+        self.assertEqual(self.queryset.count(), 4)

netbox/extras/views.py

@@ -46,6 +46,21 @@ class CustomFieldListView(generic.ObjectListView):
 class CustomFieldView(generic.ObjectView):
     queryset = CustomField.objects.select_related('choice_set')
 
+    def get_extra_context(self, request, instance):
+        related_models = ()
+
+        for content_type in instance.content_types.all():
+            related_models += (
+                content_type.model_class().objects.restrict(request.user, 'view').exclude(
+                    Q(**{f'custom_field_data__{instance.name}': ''}) |
+                    Q(**{f'custom_field_data__{instance.name}': None})
+                ),
+            )
+
+        return {
+            'related_models': related_models
+        }
+
 
 @register_model_view(CustomField, 'edit')
 class CustomFieldEditView(generic.ObjectEditView):