Non-config keys queried when using system metrics with custom configuration

[ocelotl]

Was trying this example but with this code instead:

from opentelemetry.metrics import set_meter_provider
from opentelemetry.instrumentation.system_metrics import SystemMetricsInstrumentor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader
from time import sleep
exporter = ConsoleMetricExporter()
set_meter_provider(MeterProvider([PeriodicExportingMetricReader(exporter)]))
configuration = {
    "system.memory.usage": ["used", "free", "cached"],
    "system.cpu.time": ["idle", "user", "system", "irq"],
    "system.network.io": ["trasmit", "receive"],
    "runtime.memory": ["rss", "vms"],
    "runtime.cpu.time": ["user", "system"],
}
SystemMetricsInstrumentor().instrument()
SystemMetricsInstrumentor(config=configuration).instrument()
# SystemMetricsInstrumentor().instrument()

sleep(0.1)

Got these errors:

Attempting to instrument while already instrumented
Callback failed for instrument system.cpu.utilization.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 343, in _get_system_cpu_utilization
    for metric in self._config["system.cpu.utilization"]:
KeyError: 'system.cpu.utilization'
Callback failed for instrument system.memory.utilization.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 371, in _get_system_memory_utilization
    for metric in self._config["system.memory.utilization"]:
KeyError: 'system.memory.utilization'
Callback failed for instrument system.swap.usage.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 385, in _get_system_swap_usage
    for metric in self._config["system.swap.usage"]:
KeyError: 'system.swap.usage'
Callback failed for instrument system.swap.utilization.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 399, in _get_system_swap_utilization
    for metric in self._config["system.swap.utilization"]:
KeyError: 'system.swap.utilization'
Callback failed for instrument system.disk.io.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 412, in _get_system_disk_io
    for metric in self._config["system.disk.io"]:
KeyError: 'system.disk.io'
Callback failed for instrument system.disk.operations.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 426, in _get_system_disk_operations
    for metric in self._config["system.disk.operations"]:
KeyError: 'system.disk.operations'
Callback failed for instrument system.disk.time.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 440, in _get_system_disk_time
    for metric in self._config["system.disk.time"]:
KeyError: 'system.disk.time'
Callback failed for instrument system.network.dropped_packets.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 473, in _get_system_network_dropped_packets
    for metric in self._config["system.network.dropped.packets"]:
KeyError: 'system.network.dropped.packets'
Callback failed for instrument system.network.packets.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 493, in _get_system_network_packets
    for metric in self._config["system.network.dropped.packets"]:
KeyError: 'system.network.dropped.packets'
Callback failed for instrument system.network.errors.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 508, in _get_system_network_errors
    for metric in self._config["system.network.errors"]:
KeyError: 'system.network.errors'
Callback failed for instrument system.network.io.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 524, in _get_system_network_io
    for metric in self._config["system.network.dropped.packets"]:
KeyError: 'system.network.dropped.packets'
Callback failed for instrument system.network.connections.
Traceback (most recent call last):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/instrument.py", line 122, in callback
    for api_measurement in callback(callback_options):
  File "/home/ocelotl/github/ocelotl/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py", line 544, in _get_system_network_connections
    for metric in self._config["system.network.connections"]:
KeyError: 'system.network.connections'

[aabmass]

@ocelotl I think we should remove the instrumentation level configuration altogether in favor of views

[ocelotl]

@ocelotl I think we should remove the instrumentation level configuration altogether in favor of views

That would be a breaking change 🤷

[lzchen]

@aabmass
Probably would have to add some documentation on this as the metric names are not immediately available to the user.

@ocelotl
Shouldn't that be fine since this is experimental?

[ocelotl]

@aabmass Probably would have to add some documentation on this as the metric names are not immediately available to the user.

@ocelotl Shouldn't that be fine since this is experimental?

Right, my bad. @aabmass I see your point, it would be better to provide as little instrumentation-specific configuration mechanisms as possible, ideally zero. Nevertheless, for this particular case, even when we can select the instruments that are to be used with the configuration dictionary keys and views, we can only do the configuration of specific system metrics with the configuration dictionary values, which I think is hard to do with views only.

I think this is a good point that will lead us into further discussion regarding how specific instrumentations are to be configured. I think views can be used but I am not sure that they will be able to do all the fine tuning certain instrumentations need, and for that we will have to use instrumentation specific mechanisms like this one or extend views somehow.

For the time being, this PR solves a bug now, in my opinion we are better having this merged to fix it and we can discuss the configuration topic later.

[lzchen]

@ocelotl
I'm ok with fixing the bug for now. Maybe add it to this week's agenda topic just to get concensus and start the conversation on how we should handle configuration?