Reindex - provide option to specify batch size in bytes #90195
Labels
:Distributed Indexing/Reindex
Issues relating to reindex that are not caused by issues further down
>enhancement
Team:Distributed (Obsolete)
Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Description
The current reindex API provides only the batch size as number of documents in a batch, but documents size can vary and as of now i have to account for worst case scenario (getting largest docs in one batch) to be able to finish the reindex task.
size
{Optional, integer) The number of documents to index per batch. Use when indexing from remote to ensure that the batches fit within the on-heap buffer, which defaults to a maximum size of 100 MB.
By default _reindex uses scroll batches of 1000. You can change the batch size with the size field in the source element.
But I cannot say max batch size is 100MB for example.
I'm requesting to be able to specify the size in bytes for each batch. Also it should be limited to max 10% of heap to avoid exceptions by exceeding the limit when indexing.
The text was updated successfully, but these errors were encountered: