✨ Add exemplars to prometheus metrics

[bisgaard-itis]

What do these changes do?

• Add Prometheus exemplars to our Prometheus metrics. This will allow us (in a couple of mouse clicks) to navigate from the Grafana dashboard associated with a given simcore service to the trace of a relevant request. I.e. if we see a that response latency is very slow, we can navigate to the trace of one of the requests for which the server is so slow to respond. Since our logs are also instrumented we can use the trace to navigate to all the logs emitted during handling that request. One can then to the same for another similar request (same endpoint) at a point in time when response latency was not so slow and compare the traces/logs. This workflow is inspired by https://github.com/blueswen/fastapi-observability
• Since https://github.com/trallnag/prometheus-fastapi-instrumentator doesn't support adding exemplars and we have already hit the limitations of that library I decided to consolidate the aiohttp and fastapi middleware we use to collect prometheus metrics. Hence, I will also update our Grafana dashboards accordingly.

Related issue/s

• closes Instrument prometheus metrics via exemplars #7635
• Improve/Setup osparc-simcore observability #6831

How to test

Dev-ops

• Master: prometheus enable exemplars osparc-ops-environments#1040

[codecov]

Codecov Report

Attention: Patch coverage is 89.38547% with 19 lines in your changes missing coverage. Please review.

Project coverage is 87.22%. Comparing base (97fafc4) to head (ef82189).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7644      +/-   ##
==========================================
- Coverage   87.51%   87.22%   -0.30%     
==========================================
  Files        1811     1427     -384     
  Lines       70308    59370   -10938     
  Branches     1143      604     -539     
==========================================
- Hits        61533    51787    -9746     
+ Misses       8462     7398    -1064     
+ Partials      313      185     -128     

Flag	Coverage Δ
integrationtests	64.41% <100.00%> (+<0.01%)	⬆️
unittests	86.29% <89.38%> (-0.44%)	⬇️

Components	Coverage Δ
api	∅ <ø> (∅)
pkg_aws_library	∅ <ø> (∅)
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	∅ <ø> (∅)
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	∅ <ø> (∅)
pkg_service_library	72.36% <87.31%> (+0.72%)	⬆️
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	85.07% <ø> (ø)
agent	96.46% <100.00%> (ø)
api_server	91.62% <100.00%> (ø)
autoscaling	96.07% <100.00%> (-0.01%)	⬇️
catalog	92.64% <83.33%> (ø)
clusters_keeper	99.25% <100.00%> (ø)
dask_sidecar	89.86% <ø> (ø)
datcore_adapter	98.12% <100.00%> (ø)
director	76.78% <100.00%> (-0.03%)	⬇️
director_v2	91.12% <100.00%> (-0.05%)	⬇️
dynamic_scheduler	96.76% <100.00%> (ø)
dynamic_sidecar	90.18% <ø> (ø)
efs_guardian	89.79% <ø> (ø)
invitations	93.28% <100.00%> (ø)
payments	92.63% <100.00%> (ø)
resource_usage_tracker	89.13% <ø> (ø)
storage	87.49% <80.00%> (-0.32%)	⬇️
webclient	∅ <ø> (∅)
webserver	85.82% <ø> (+0.02%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 97fafc4...ef82189. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[GitHK]

Thanks please consider my comments

[matusdrobuliak66]

👍 thanks

[sanderegg]

Am I correct in thinking that you removed the fastapi-instrumentation library usage?
If yes should you not also remove its dependency?

[YuryHrytsuk]

I decided to consolidate the aiohttp and fastapi middleware we use to collect prometheus metrics.

What does it mean? This is not clear

[bisgaard-itis]

I decided to consolidate the aiohttp and fastapi middleware we use to collect prometheus metrics.

What does it mean? This is not clear

Before we used our own custom implementation middleware for collecting prometheus metrics for aiohttp services and a pip installed tool for fastapi services. Now I have removed that tool and refactored the aiohttp middleware to have an aiohttp middleware and a separate fastapi middleware (custom implementation) which call the same module for collecting prometheus metrics

[bisgaard-itis]

Also I do not see any test

tests have been added.

[sanderegg]

thanks

[pcrespov]

Very nice! I would like to ask you some specific details about it sometime offline

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud