♻️ Use catalog rpc client in api-server

[bisgaard-itis]

What do these changes do?

• Introduce SolverService, ProgramsService and JobService in the api-server. These parts of the service layer in the api-server are new in the sense that they build on top of the "raw" clients to the backend. They make use of the FastAPI dependency injection system for handling the their dependency on the underlying CatalogService. This reduces boilerplate code and simplifies the app initialization phase.
• Make use of new catalog rpc client within the api-server
• Add several tests where they were missing
• Along the way I had to expose the legacy long running tasks in the openapi specs of the webserver. I hid them myself when exposing the async jobs interface in the webserver. But it seems clear that the legacy endpoints should also be exposed in the OAS.
• This almost completes Use catalog RPC client instead of the http client. #7525, but still a few changes are to be done once Support filtering of dynamic/computational services in catalog rpc client #7532 has been completed

Related issue/s

• Use catalog RPC client instead of the http client. #7525

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 99.30070% with 1 line in your changes missing coverage. Please review.

Project coverage is 87.56%. Comparing base (9a00aa7) to head (06a98bc).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7541      +/-   ##
==========================================
+ Coverage   87.49%   87.56%   +0.06%     
==========================================
  Files        1743     1738       -5     
  Lines       67497    67364     -133     
  Branches     1144     1144              
==========================================
- Hits        59054    58984      -70     
+ Misses       8122     8059      -63     
  Partials      321      321              

Flag	Coverage Δ
integrationtests	65.22% <ø> (-0.02%)	⬇️
unittests	86.74% <99.30%> (+0.07%)	⬆️

Components	Coverage Δ
api	∅ <ø> (∅)
pkg_aws_library	93.91% <ø> (ø)
pkg_dask_task_models_library	97.09% <ø> (ø)
pkg_models_library	92.00% <ø> (ø)
pkg_notifications_library	85.26% <ø> (ø)
pkg_postgres_database	88.18% <ø> (ø)
pkg_service_integration	70.03% <ø> (ø)
pkg_service_library	72.53% <ø> (ø)
pkg_settings_library	90.79% <ø> (ø)
pkg_simcore_sdk	85.40% <ø> (ø)
agent	96.46% <ø> (ø)
api_server	91.23% <99.30%> (+0.97%)	⬆️
autoscaling	96.08% <ø> (ø)
catalog	92.52% <ø> (ø)
clusters_keeper	99.24% <ø> (ø)
dask_sidecar	91.29% <ø> (ø)
datcore_adapter	98.12% <ø> (ø)
director	76.87% <ø> (ø)
director_v2	91.30% <ø> (ø)
dynamic_scheduler	97.40% <ø> (ø)
dynamic_sidecar	90.11% <ø> (ø)
efs_guardian	89.79% <ø> (ø)
invitations	93.28% <ø> (ø)
payments	92.66% <ø> (ø)
resource_usage_tracker	89.12% <ø> (-0.06%)	⬇️
storage	87.59% <ø> (+0.07%)	⬆️
webclient	∅ <ø> (∅)
webserver	86.00% <ø> (-0.05%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a00aa7...06a98bc. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pcrespov]

Super nice work! I specially enjoyed the technique of using dependencies in the constructor.

Left some suggestions for your consideration

[matusdrobuliak66]

Thanks a lot!

THOUGHT: @bisgaard-itis @pcrespov - From a design perspective, I think I’d prefer if all the logic were built into the CORE service rather than the API server.
Reason? Reusability.

Right now, you’re introducing a new concept "programs" from the perspective of the public API, but eventually, some UI elements will be needed (btw, even now, just creating an API key requires going through the UI — so these things are coupled).

I believe the API gateway or web API should act purely as proxies/gateways to the CORE application - which currently lives in the webserver container. This core service owns almost everything important: projects, users, access rights, wallets, etc.

[bisgaard-itis]

Thanks a lot!

THOUGHT: @bisgaard-itis @pcrespov - From a design perspective, I think I’d prefer if all the logic were built into the CORE service rather than the API server. Reason? Reusability.

Right now, you’re introducing a new concept "programs" from the perspective of the public API, but eventually, some UI elements will be needed (btw, even now, just creating an API key requires going through the UI — so these things are coupled).

I believe the API gateway or web API should act purely as proxies/gateways to the CORE application - which currently lives in the webserver container. This core service owns almost everything important: projects, users, access rights, wallets, etc.

I see your point. I would argue though, that the issue is that we are coupling concepts in the webserver and the api-server. I don't think that's the way to go. We don't need to do this coupling except maybe for some technical details (like the new projects_to_jobs table).
I don't think a program should really be a concept in the core, because from the point of view of the core it is just another project (which turns out to have some special properties). The logic in the api-server is essentially a reduced interface to interact with projects that fulfill certain conditions. For me it would make more sense that when the frontend wants to know about solver jobs it would ask the public api.

[matusdrobuliak66]

Thanks a lot!
THOUGHT: @bisgaard-itis @pcrespov - From a design perspective, I think I’d prefer if all the logic were built into the CORE service rather than the API server. Reason? Reusability.
Right now, you’re introducing a new concept "programs" from the perspective of the public API, but eventually, some UI elements will be needed (btw, even now, just creating an API key requires going through the UI — so these things are coupled).
I believe the API gateway or web API should act purely as proxies/gateways to the CORE application - which currently lives in the webserver container. This core service owns almost everything important: projects, users, access rights, wallets, etc.

I see your point. I would argue though, that the issue is that we are coupling concepts in the webserver and the api-server. I don't think that's the way to go. We don't need to do this coupling except maybe for some technical details (like the new projects_to_jobs table). I don't think a program should really be a concept in the core, because from the point of view of the core it is just another project (which turns out to have some special properties). The logic in the api-server is essentially a reduced interface to interact with projects that fulfill certain conditions. For me it would make more sense that when the frontend wants to know about solver jobs it would ask the public api.

@bisgaard-itis Yes, I see your point 👍 As always, both approaches have their pros and cons. :)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud