ensure only one connection acquisition per request

[sanderegg]

What do these changes do?

Some requests were acquiring several connections
This leads to an issue where a burst of requests (concurrent requests) might end in a dead-lock in case where several repositories are needed at the same time by the request and the amount of connection is limited.

For example:

• a request that needs 2 repositories (both needed 1 connection -> 2 connections)
• we have 2 available connections
• 10 requests come at the same time or more or less
• 1st request acquire one connection for one of the repos and then try to get another for the second repo
• 2nd request acquire one connection for one of the repos and then try to get another for the second repo
  --> request 1 and 2 are locked... since there is no connection available anymore + the 8 remaining requests are waiting forever...

The solution here changes the following:

• the connection is acquired once per request and released at the end of the request.

Pros: no dead-lock, it keeps the nice concept
Cons: in case of bursts, the requests have to wait until a connection is available even though maybe the request would not need to block the connection all the time... to be continued...

BONUS:

• migrate projects which workbench contains FAILURE state to FAILED...

Related issue number

How to test

Checklist

• Did you change any service's API? Then make sure to bundle document and upgrade version (make openapi-specs, git commit ... and then make version-*)
• Unit tests for the changes exist
• Runs in the swarm
• Documentation reflects the changes
• New module? Add your github username to .github/CODEOWNERS

[codecov]

Codecov Report

Merging #1966 (b48c55a) into master (cae0146) will increase coverage by 0.0%.
The diff coverage is 51.5%.

@@          Coverage Diff           @@
##           master   #1966   +/-   ##
======================================
  Coverage    73.2%   73.2%           
======================================
  Files         372     372           
  Lines       13805   13817   +12     
  Branches     1408    1409    +1     
======================================
+ Hits        10114   10125   +11     
- Misses       3317    3320    +3     
+ Partials      374     372    -2     

Flag	Coverage Δ
integrationtests	63.3% <81.8%> (+<0.1%)	⬆️
unittests	66.7% <36.3%> (-0.1%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...re_service_api_server/api/dependencies/database.py	60.0% <36.3%> (-15.0%)	⬇️
...mcore_service_catalog/api/dependencies/database.py	60.0% <36.3%> (-1.2%)	⬇️
...e_service_director_v2/api/dependencies/database.py	90.0% <81.8%> (+1.1%)	⬆️
.../director/src/simcore_service_director/producer.py	60.9% <0.0%> (+0.4%)	⬆️
...webserver/computation_comp_tasks_listening_task.py	91.0% <0.0%> (+3.3%)	⬆️

[pcrespov]

The solution is very elegant but subtle. If the FastAPI dependency cache gets deactivated or the policy changes it is difficult to notice that all repositories in a request is using the same connection. Add at least a note or a reference to this PR so that in the future we could understand the rationale.

[GitHK]

OK, for now. This could also cause issues of requests timing out, because it is taking too long to get a free connection.