Python - Allow auto installation in k8s clusters

[estolfo]

Some work should go upstream, here are the relevant issues:

• Python autoinstrumentation setting of PYTHONPATH is not compliant with Python's module resolution behavior, breaking Django applications open-telemetry/opentelemetry-operator#2302
• Django app on K8s crashes when auto-instrumented via the opentelemetry-operator without explicitly setting the PYTHONPATH and DJANGO_SETTINGS_MODULE open-telemetry/opentelemetry-python-contrib#2495

Tasks

Preview Give feedback

1. Handle grpc exporter for the kubernetes operator #96
   basic-distro

[xrmx]

PR for cleanup the k8s operator handling of used OTLP protocol https://github.com/open-telemetry/opentelemetry-operator/pull/3165

[xrmx]

Wondering if running django as python -m django runserver instead of python manage.py runserver would make a difference as per https://docs.python.org/3/library/sys.html#sys.path we should get the current working directory in sys.path if we call our script that way. The question is that sys.path is relevant for django.

[xrmx]

Built a custom docker image for the k8s operator and it's evident that for Python this won't be a silver bullet. Problem is that a lot of libraries depends on C extensions that are platform dependant.

For example psutil (from system metrics instrumentation) built on an alpine distro but run into an ubuntu container will fail because of:

ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

and this import (File "/otel-auto-instrumentation-python/elasticotel/distro/__init__.py", line 25, in <module>\nfrom opentelemetry.instrumentation.system_metrics import () comes from our distro so it should be fixed.

grpc aswell:

ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/otel-auto-instrumentation-python/grpc/_cython/__init__.py)

From a different path though:

  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 39, in initialize
    _load_instrumentors(distro)
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation/_load.py", line 91, in _load_instrumentors
    raise exc
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation/_load.py", line 87, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)

At the moment a failure of an instrumentor makes the whole process fail and in this scenario (or even generally?) it's not want we want.

[xrmx]

So at the moment we are building one image based on the Python 3.12 chainguard image (glibc based). In order to cover more environments we should probably increase the matrix of the images we build with more python versions and/or with musl based base images.

[xrmx]

A few otel operator related PRs to improve things:

• one to set a default for logs exporting if one decide to enable them https://github.com/open-telemetry/opentelemetry-operator/pull/3330
• one to handle musl based containers, if accepted we can work out of the box https://github.com/open-telemetry/opentelemetry-operator/pull/3332
• the docs one documenting the issue with binary wheels https://github.com/open-telemetry/opentelemetry.io/pull/5338

[xrmx]

Closing this and using #158 for the first round of followup improvements