Django app on K8s crashes when auto-instrumented via the opentelemetry-operator without explicitly setting the PYTHONPATH and DJANGO_SETTINGS_MODULE

[srprash]

Describe your environment Describe any aspect of your environment relevant to the problem, including your Python version, platform, version numbers of installed dependencies, information about your cloud hosting provider, etc. If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on main.

• Python version: 3.11
• Platform: Kubernetes
• Installed dependencies:
  • Django~=5.0.4
  • requests==2.31.0
• Hosting: hosting locally on minikube, but same issue when hosted on AWS EKS

Steps to reproduce
Sample app code is here: https://github.com/srprash/otel-python-k8s-samples/tree/main/django

1. Install Docker and Minikube, and run both.
2. Run eval $(minikube docker-env) so that minikube can use the docker images from local repository.
3. Build the application docker image using docker build -t django-app .
4. Deploy the application to minikube using kubectl apply -f k8s/deployment.yaml and kubectl apply -f k8s/service.yaml
5. Enable port-forwarding kubectl port-forward svc/django-service 8000
6. Make a request to the application on http://127.0.0.1:8000/outgoing-http-call endpoint

What is the expected behavior?
What did you expect to see?

• The application should run without an issues and produce a server span for the incoming request and a client span for the outgoing http request

What is the actual behavior?
What did you see instead?

• The application fails to start with the error:

  Defaulted container "django-app" out of: django-app, opentelemetry-auto-instrumentation-python (init)
  CommandError: You must set settings.ALLOWED_HOSTS if DEBUG is False.
  

• When I uncomment these lines (that is, explicitly set the PYTHONPATH and DJANGO_SETTINGS_MODULE), only then the application works correctly and is instrumented.

Additional context

• Another related issue: Python autoinstrumentation setting of PYTHONPATH is not compliant with Python's module resolution behavior, breaking Django applications opentelemetry-operator#2302
• Note that I have another application using Flask and this issue is not seen there.
• The issue with Django is specifically when using the opentelemetry-operator to do the auto-instrumentation by adding the instrumentation.opentelemetry.io/inject-python: "true" annotation to the application deployment manifest. Auto-instrumentation via the opentelemetry-instrument command works fine (likely due to this code that handles the current working directory).

[srprash]

This issue was also discussed in the Python SIG meeting on May 02, 2024. Notes: https://docs.google.com/document/d/1CIMGoIOZ-c3-igzbd6_Pnxx1SjAkjwqoYSUWxPY8XIs/edit

Between the meeting and discussion on this slack thread, it seems like the operator login needs to be fixed on how the pythonpath is handled.

[srprash]

From the SIG meeting discussion, tagging @jeremydvoss since he encountered this issue as well and to take a look.

[xrmx]

Maybe I'm doing something wrong but with otel-python-k8s-samples (a7aef5d310e283ea8d3a4ddc5619003920d9abce) the Django application starts and works fine (I get "[Django] Ok! tracing outgoing http call") but I don't get any span in stdout (minikube kubectl -- logs -f django-deployment-75879b9d74-bbp5l).

UPDATE: I guess because opentelemetry-operator is not injected or something given the os.environ I have after trimming the kubernetes related one:

{
  'PATH': '/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
  'OTEL_TRACES_EXPORTER': 'console',
  'OTEL_METRICS_EXPORTER': 'none',
  'OTEL_LOGS_EXPORTER': 'none',
  'OTEL_PYTHON_LOG_LEVEL': 'debug',
  'HOME': '/root',
  'SERVER_SOFTWARE': 'gunicorn/22.0.0',
  'DJANGO_SETTINGS_MODULE': 'djangoApp.settings',
  'TZ': 'UTC'
}

UPDATE 2: the instructions above does not include the operator installation so:

minikube kubectl -- apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.1/cert-manager.yaml
minikube kubectl -- apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml