Description
Currently a gateway can work with at most one server instance due to the server using a fixed 8001 port for the gateway-server reverse tunnel. (The server establishes a reverse tunnel for the gateway, making the server API accessible to the gateway on port 8001. This is currently used for user token verification. However, since the port is fixed, two server replicas cannot create a tunnel on port 8001.) This is a blocker for #1632.
A possible solution to this problem is to establish the reverse tunnel via a Unix domain socket instead of 8001 port. This is how the direct server->gateway tunnel works. So, the server replicas will be accessible on the gateway through paths like /somepath/server1, /somepath/server2, etc, and the gateway will be able to choose any server.
Another limitation related to gateways is that the server maintains an in-memory pool of gateway connections that initialized once at server startup. New connections are added when processing submitted gateways. Since it takes place on one server replica, other replicas won't establish the connection. Submitting a service would fail on those replicas. Perhaps the easiest solution would be to establish the connection when submitting a service if it's not in the connection pool yet. There should also be a background task to keep the pool in sync with the db state to remove connections for deleted gateways.
We also need to be mindful of gateway updates compatibility to support zero-downtime (e.g. blue-green) deployments. The server updates the gateways to the latest version on startup. With a zero-downtime deployment, there will still be old server replicas running when a new one starts, so the new version of gateway should always be compatible with the old server. If incompatible changes are required, introduce them in two releases as with the incompatible db changes: 1) add and adapt code 2) remove. Gateway update process should be safe to run concurrently (e.g. take advisory locks).
To sum up:
- Use Unix domain sockets for gateway-server communication.
- Adapt gateway connections management to work with multi-replica server.
- Ensure the gateway update process works correctly with multi-replica server.