-
Notifications
You must be signed in to change notification settings - Fork 768
feat: block stream write #17744
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: block stream write #17744
Conversation
a0e48a0
to
b6b2b41
Compare
Docker Image for PR
|
Docker Image for PR
|
Docker Image for PR
|
Docker Image for PR
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Docker Image for PR
|
Confilicts: src/query/storages/fuse/src/io/write/block_writer.rs |
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
This PR introduces block stream writing support in Databend, which enables writing data in a streaming manner during insert operations.
Key highlights:
TransformBlockWriter
, designed to stream and flush data blocks efficiently based on uncompressed size and compressed size thresholds.INSERT
statementsCOPY INTO
operationsTo enable block stream write:
This improvement lays the foundation for better memory control and performance in large-scale data writing scenarios.
Tests
Type of change
This change is