Test services resilience

[pcrespov]

Found some situations in which service suddenly crashes and the swarm constantly restarts it ... and enters in an endless stop-restart loop.

This happens either because code is broken (normally caught by pylint but not always) or a faulty design

The fact is: All services should guarantee certain level of resilience.

• only situation it stops is if configuration is wrong?
• does not crash upon startup (ensure app is running after startup and container does not restart)
• if a dependent backend service is down, it should NOT stop but rather disable the affected functionality

Examples

• If specs validation fails, webserver will retry ... as if he failed to connect apihub!
• If minio/s3 access is not available. services should not collapse but just report error
  - storage GET /locations/?user_id ... gets an error in db (connection drops) and
• shall be able to start service from CLI w/o any service around up
• if redis fails to start, webserver fails
• Migration service takes some time to upgrade. Webserver and other might be resilient to failure until migration is achieved! This happens e.g. in the e2e where migration takes a some time to start ... and in the meantime the webserver has failed and restarted. Notice that startup cannot wait eternaly because we also want a fast startup ... to achieve zero-downtime!

TODO

• define resilience guarantee levels (analogous to exceptions guarantees )
• create generic tests. Should involve starting a service within its container (see ideas on /docker-makefile-x-ops)
• add to cookiecutters

Related with PBI

• Software stability

[pcrespov]

Can use scripts in ITISFoundation/osparc-op to start stacks with failures or stress test the system

[pcrespov]

Case: A connection dropped unexpectedly

In this test:

• front-end hits run pipeline
• webserver -> (start pipeline) -> director_v2
• client in webserver raised aiohttp.client_exceptions.ServerDisconnectedError and was translated into 500
• backend responded with 500
• the front-end did not handle it

-> client retry policy
-> front-end should handle it since a simply ask the user to retry.