[serve] Log rejected requests at router side (#51346)

## Why are these changes needed?

Router side logs (made less alarming, made clear that request will be
retried):
```
INFO 2025-03-13 13:42:35,298 serve 40047 -- Replica(id='7mqhdb0d', deployment='Model', app='default') rejected request because it is at max capacity of 1 ongoing requests. Retrying request 4a843e03-e1c7-47a2-be9d-6c0224108f42.
INFO 2025-03-13 13:42:35,298 serve 40047 -- Replica(id='7mqhdb0d', deployment='Model', app='default') rejected request because it is at max capacity of 1 ongoing requests. Retrying request 57d94c8a-13b4-4ea2-a628-75d566ef29e5.
INFO 2025-03-13 13:42:35,301 serve 40047 -- Replica(id='7mqhdb0d', deployment='Model', app='default') rejected request because it is at max capacity of 1 ongoing requests. Retrying request 4a843e03-e1c7-47a2-be9d-6c0224108f42.
```

Replica side logs about rejected requests are now DEBUG logs only.

This is to make the logs appear less alarming for users who are not
familiar with the request lifecycle. The way the logs are now, the user
can get confused reading the replica-side logs and think requests got
dropped.
https://anyscale1.atlassian.net/browse/SERVE-659

---------

Signed-off-by: Cindy Zhang <[email protected]>

Author: zcin

Diff for: python/ray/serve/_private/replica.py

@@ -624,7 +624,7 @@ async def handle_request_with_rejection(
         limit = self._deployment_config.max_ongoing_requests
         num_ongoing_requests = self.get_num_ongoing_requests()
         if num_ongoing_requests >= limit:
-            logger.warning(
+            logger.debug(
                 f"Replica at capacity of max_ongoing_requests={limit}, "
                 f"rejecting request {request_metadata.request_id}.",
                 extra={"log_to_stderr": False},

Diff for: python/ray/serve/_private/router.py

@@ -541,6 +541,14 @@ async def schedule_and_send_request(
                 self._replica_scheduler.on_new_queue_len_info(r.replica_id, queue_info)
                 if queue_info.accepted:
                     return result, r.replica_id
+                else:
+                    logger.info(
+                        f"{r.replica_id} rejected request because it is at max "
+                        f"capacity of {r.max_ongoing_requests} ongoing request"
+                        f"{'s' if r.max_ongoing_requests > 1 else ''}. "
+                        f"Retrying request {pr.metadata.request_id}.",
+                        extra={"log_to_stderr": False},
+                    )
             except asyncio.CancelledError:
                 # NOTE(edoakes): this is not strictly necessary because there are
                 # currently no `await` statements between getting the ref and returning,

Diff for: python/ray/serve/tests/unit/test_router.py

@@ -90,6 +90,10 @@ def __init__(
     def replica_id(self) -> ReplicaID:
         return self._replica_id
 
+    @property
+    def max_ongoing_requests(self) -> int:
+        return 5
+
     @property
     def is_cross_language(self) -> bool:
         return self._is_cross_language