Disable buffered response body processing mode #110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

ahg-g opened this issue Dec 19, 2024 · 0 comments · Fixed by #121

Assignees

Contributor

ahg-g commented Dec 19, 2024

A significant portion of LLM requests are streaming requests, and therefore we can't use buffered processing mode for responses.

/assign @liu-cong

k8s-ci-robot assigned liu-cong

liu-cong mentioned this issue

Disable response body processing #121

Merged

k8s-ci-robot closed this as completed in #121

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment