Skip to content

Partition temporary (and resulting) vector index tables before uploading #17166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Tracked by #8967
kunga opened this issue Apr 14, 2025 · 1 comment
Open
Tracked by #8967

Comments

@kunga
Copy link
Member

kunga commented Apr 14, 2025

Because we use (parentClusterId, ...) keys in build and resulting tables we may evenly them create with necessary number of partitions before upload to speed up build index process

It seems fine to make 1 shard is twice bigger than expected evenly distributed rows among clusters

@kunga kunga mentioned this issue Apr 14, 2025
48 tasks
@azevaykin azevaykin changed the title Split resulting tables before uploading Partition temporary vector index tables before uploading Apr 14, 2025
@kunga kunga changed the title Partition temporary vector index tables before uploading Partition temporary (and resulting) vector index tables before uploading Apr 14, 2025
@kunga
Copy link
Member Author

kunga commented Apr 22, 2025

Seems already done for temp build tables here

But need to be improved for posting table with covered columns data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant