webserver's healthcheck monitors and diagnoses slow callbacks as unhealthy

[pcrespov]

What do these changes do?

Introduces some diagnostics in webserver that monitors slow callbacks. Under certain conditions () the diagnose can determine that the service in unhealthy and swarm will automatically restart it.

• servicelib.monitor_slow_callbacks: hooks look event handler and registers an incident when there is a slow callback
• webserver new diagnostics modules that keeps track of incidents
• webserver healthcheck now returns 503 when it is overloaded to long-delayed callbacks. Then swarm will restart the webserver.

Related issue number

Platform Stability
Split from PR #1401

How to test

cd package/service-library
make install-dev
make tests

cd ../services/web/server
make install-dev
make tests-unit

Manual Test: Artificially enforced slowdown and diagnostics flagged the service as unhealthy. Swarm reacted by restarting it

Checklist

• Did you change any service's API? Then make sure to bundle document and upgrade version (make openapi-specs, git commit ... and then make version-*)
• Unit tests for the changes exist
• Runs in the swarm
• Documentation reflects the changes
• New module? Add your github username to .github/CODEOWNERS

[codecov]

Codecov Report

Merging #1406 into master will increase coverage by 0.99%.
The diff coverage is 93.75%.

@@            Coverage Diff             @@
##           master    #1406      +/-   ##
==========================================
+ Coverage   69.76%   70.76%   +0.99%     
==========================================
  Files         222      225       +3     
  Lines        8818     8913      +95     
  Branches      968      979      +11     
==========================================
+ Hits         6152     6307     +155     
+ Misses       2383     2324      -59     
+ Partials      283      282       -1

Flag	Coverage Δ
#integrationtests	57.16% <48.83%> (-0.09%)	⬇️
#unittests	65.33% <93.75%> (-2.01%)	⬇️

Impacted Files	Coverage Δ
...ver/src/simcore_service_webserver/rest_handlers.py	87.5% <100%> (+3.12%)	⬆️
...erver/src/simcore_service_webserver/application.py	93.65% <100%> (+0.2%)	⬆️
...e-library/src/servicelib/monitor_slow_callbacks.py	100% <100%> (ø)
...ckages/service-library/src/servicelib/incidents.py	88.23% <88.23%> (ø)
...erver/src/simcore_service_webserver/diagnostics.py	93.75% <93.75%> (ø)
.../director/src/simcore_service_director/producer.py	67.01% <0%> (+0.25%)	⬆️
...core-sdk/src/simcore_sdk/node_ports/_items_list.py	90.69% <0%> (+2.32%)	⬆️
... and 7 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d2b3e6b...95ec007. Read the comment docs.

[odeimaiz]

Yeah, no more every second day manual restarting!

[sanderegg]

awesome! just an empty test to clean away