Error when adding/editing secrets with webhooks for secrets enabled

[tyldum]

Environment

• Python version: 3.8.6
• NetBox version: 2.9.9

Steps to Reproduce

1. Enable a webhook on secrets
2. Secret on a device
3. Save

Expected Behavior

No errors, return to list of secrets

Observed Behavior

Got an error:

<class '_pickle.PicklingError'>

Can't pickle <function paginator_number at 0x7fcbf0afb670>: it's not the same object as django.contrib.admin.templatetags.admin_list.paginator_number

The secret is stored without the tag.
Tags can be added afterwards without further issues.

(in light of the proposal to move this functionality into a plugin, I realize it might quickly become obsolete)

[jeremystretch]

I'm not able to reproduce this on v2.9.10: Secrets are created with tags as expected. Additionally, the exception being reported makes no sense. Have you modified your NetBox code base in any way? If not, please post the entire stack trace (enable debugging by setting DEBUG=True in configuration.py).

[tyldum]

Running the stock docker image with -ldap. Upgraded to v2.9.10, same thing.

<class '_pickle.PicklingError'>

Can't pickle <function paginator_number at 0x7f6de3c74a60>: it's not the same object as django.contrib.admin.templatetags.admin_list.paginator_number

Python version: 3.8.6
NetBox version: 2.9.10

Stack trace:

Environment:


Request Method: POST
Request URL: http://xxxx/secrets/secrets/add/

Django Version: 3.1
Python Version: 3.8.6
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'cacheops',
 'corsheaders',
 'debug_toolbar',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'mptt',
 'rest_framework',
 'taggit',
 'timezone_field',
 'circuits',
 'dcim',
 'ipam',
 'extras',
 'secrets',
 'tenancy',
 'users',
 'utilities',
 'virtualization',
 'django_rq',
 'drf_yasg']
Installed Middleware:
['debug_toolbar.middleware.DebugToolbarMiddleware',
 'django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'utilities.middleware.ExceptionHandlingMiddleware',
 'utilities.middleware.RemoteUserMiddleware',
 'utilities.middleware.LoginRequiredMiddleware',
 'utilities.middleware.APIVersionMiddleware',
 'extras.middleware.ObjectChangeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']



Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 179, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 73, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/netbox/netbox/secrets/views.py", line 97, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/opt/netbox/netbox/utilities/views.py", line 393, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/opt/netbox/netbox/utilities/views.py", line 124, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 101, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/netbox/netbox/secrets/views.py", line 128, in post
    secret.save()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 750, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 798, in save_base
    post_save.send(
  File "/usr/local/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 177, in send
    return [
  File "/usr/local/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 178, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/opt/netbox/netbox/utilities/utils.py", line 283, in _curried
    return _curried_func(*args, *moreargs, **{**kwargs, **morekwargs})
  File "/opt/netbox/netbox/extras/signals.py", line 42, in _handle_changed_object
    enqueue_webhooks(instance, request.user, request.id, action)
  File "/opt/netbox/netbox/extras/webhooks.py", line 57, in enqueue_webhooks
    webhook_queue.enqueue(
  File "/usr/local/lib/python3.8/site-packages/rq/queue.py", line 418, in enqueue
    return self.enqueue_call(
  File "/usr/local/lib/python3.8/site-packages/django_rq/queues.py", line 68, in enqueue_call
    return self.original_enqueue_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django_rq/queues.py", line 64, in original_enqueue_call
    return super(DjangoRQ, self).enqueue_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rq/queue.py", line 365, in enqueue_call
    job = self.enqueue_job(job, at_front=at_front)
  File "/usr/local/lib/python3.8/site-packages/rq/queue.py", line 473, in enqueue_job
    job.save(pipeline=pipe)
  File "/usr/local/lib/python3.8/site-packages/rq/job.py", line 591, in save
    mapping = self.to_dict(include_meta=include_meta)
  File "/usr/local/lib/python3.8/site-packages/rq/job.py", line 537, in to_dict
    'data': zlib.compress(self.data),
  File "/usr/local/lib/python3.8/site-packages/rq/job.py", line 232, in data
    self._data = self.serializer.dumps(job_tuple)

Exception Type: PicklingError at /secrets/secrets/add/
Exception Value: Can't pickle <function paginator_number at 0x7f18bdc131f0>: it's not the same object as django.contrib.admin.templatetags.admin_list.paginator_number

[jeremystretch]

Running the stock docker image

Please note that all bug reports must be reproducible in an unmodified instance of the latest official release.

The stack trace you posted indicates a problem with a webhook you've created, which was not mentioned in your original report. Disabling or correcting the webhook will likely resolve this issue.

[tyldum]

The webhook is valid and the error is triggered by creating or changing a secret with the webhook enabled. Deleting the secret works without errors and issues the webhook as expected.

The webhook is as simple as you can define it, no headers or body. Same webhook is used on several object types for audit alerts, without issues (and works on delete events on secrets).

I'll see if I can replicate on a virgin install. Updating title as it is misleading.

[jeremystretch]

Well, I've gotten as far as reproducing this. It might take some time to unpack exactly what's going on, though.

@tyldum you've only seen this with secrets, correct? No other models?

[tyldum]

That is correct.
Reported mostly for consistency and correct behavior. We can live without it, if you decide the correct behavior is to remove webhooks on secrets.

[jeremystretch]

No no, it should definitely work as-is. When you first reported that it was due to a webhook I figured it was due to some invalid template code, because the exception message didn't make any sense. And it still doesn't.

[DanSheps]

Have we tried to disable caching to see if it is a caching related issue?

[jeremystretch]

Yeah, I've been testing it without caching enabled. Best I can tell it has something to do with the serializer for secrets, but I'm not sure what specifically is causing the exception.

I did come across this very similar-looking bug but it seems to have been something related to SQLite. Still need to dig into this more.

[jeremystretch]

All I've been able to identify so far is that omitting the url field from SecretSerializer avoids triggering the PicklingError exception. Of course this makes no sense, because every model uses a HyperlinkedIdentityField in its serializer, and there are other instances of HyperlinkedIdentityField within nested serializers within SecretSerializer.

[jeremystretch]

Closing this out as it will ultimately be addressed by #5278 in v2.12.