🐛 Fixes catalog's synchronization background task continues errors due to faulty service info

[pcrespov]

What do these changes do?

Prevents the background sync task in catalog from continuously "stopping and restarting" due to errors in the director's response for certain items in the list. This change ensures that the task can still complete for the remaining items despite errors.

This is the error recurrently logged by the background sync task:

log_level=DEBUG | log_timestamp=2024-09-10 13:40:01,893 | log_source=httpcore.http11:atrace(85) | log_uid=None | log_msg=response_closed.complete
log_level=ERROR | log_timestamp=2024-09-10 13:40:01,893 | log_source=simcore_service_catalog.core.background_tasks:_sync_services_task(190) | log_uid=None | log_msg=Unexpected error while syncing registry entries, restarting now...
Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_catalog/core/background_tasks.py", line 177, in _sync_services_task
    await _run_sync_services(app)
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_catalog/core/background_tasks.py", line 165, in _run_sync_services
    await _ensure_registry_and_database_are_synced(app)
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_catalog/core/background_tasks.py", line 114, in _ensure_registry_and_database_are_synced
    await _create_services_in_database(
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_catalog/core/background_tasks.py", line 74, in _create_services_in_database
    ) = await access_rights.evaluate_default_policy(app, service_metadata)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_catalog/services/access_rights.py", line 73, in evaluate_default_policy
    if _is_frontend_service(service) or await _is_old_service(app, service):
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_catalog/services/access_rights.py", line 43, in _is_old_service
    await client.get(
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_catalog/services/director.py", line 94, in request_wrapper
    return _unenvelope_or_raise_error(resp)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_catalog/services/director.py", line 71, in _unenvelope_or_raise_error
    raise HTTPException(resp.status_code, detail=msg)
fastapi.exceptions.HTTPException: 404

We have improved the resilience of these routines by catching expected exceptions, issuing a warning, and continuing to process the remaining elements in the loop without interruption.

Misc

• extra tmp timeout for @mguidon until issue is fully resolved

Related issue/s

• fixes Director stops synchronizing with registry if there is a problem #6318
• part of Optimization and wrap-up of catalog services web-api #6201
• resolves part of Catalog background task fails to add access rights to a project not listed in the services_meta_data #1854

How to test

cd services/catalog
make install-dev
# when parameter `director_fails=True`
pytest -vv tests/unit/with_dbs -k test_registry_sync_task

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.3%. Comparing base (cafbf96) to head (a10025e).
Report is 529 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #6344      +/-   ##
=========================================
- Coverage    84.5%   82.3%    -2.3%     
=========================================
  Files          10    1103    +1093     
  Lines         214   48647   +48433     
  Branches       25     925     +900     
=========================================
+ Hits          181   40068   +39887     
- Misses         23    8408    +8385     
- Partials       10     171     +161     

Flag	Coverage Δ
integrationtests	64.3% <ø> (?)
unittests	79.9% <100.0%> (-4.7%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...icelib/rabbitmq/rpc_interfaces/catalog/services.py	0.0% <ø> (ø)
...c/simcore_service_catalog/core/background_tasks.py	92.3% <100.0%> (ø)
...g/src/simcore_service_catalog/exceptions/errors.py	93.3% <100.0%> (ø)
.../simcore_service_catalog/services/access_rights.py	98.7% <100.0%> (ø)

... and 1109 files with indirect coverage changes

[matusdrobuliak66]

Nice, thanks.

[sanderegg]

perfect, thanks!

[GitHK]

👍

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud