[Feature]: RPS autoscaling for services without a gateway

[jvstme]

Problem

RPS auto-scaling is only supported when running services with a gateway.
It is not possible to run a service without a gateway with RPS auto-scaling enabled.

> cat .dstack.yml
type: service
image: nginx
port: 80
gateway: false
replicas: 1..2
scaling:
  metric: rps
  target: 50

> dstack apply
Auto-scaling is not yet supported when running services without a gateway. Please configure a gateway or set `replicas` to a fixed value in the service configuration

Solution

Support RPS auto-scaling for services without a gateway.

Implementation note

The main difficulty in implementing this feature is synchronizing the request metrics between multiple dstack-server replicas. We can try to store the metrics in the database, but this may impact performance when running services with high RPS.

Workaround

Use a gateway.

Would you like to help us implement this feature by sending a PR?

Yes

[github-actions]

This issue is stale because it has been open for 30 days with no activity.