Can not Reindex from remote even with 1 batch size - Remote responded with a chunk that was too large. Use a smaller batch size.

[phinzin]

Elasticsearch version (bin/elasticsearch --version): old: 5.6.3 new: 6.8.9

Plugins installed: None

JVM version (java -version): Open JDK 1.8

OS version (uname -a if on a Unix-like system): window 10

Description of the problem including expected versus actual behavior:
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Remote responded with a chunk that was too large. Use a smaller batch size."
}
],
"type": "illegal_argument_exception",
"reason": "Remote responded with a chunk that was too large. Use a smaller batch size.",
"caused_by": {
"type": "content_too_long_exception",
"reason": "entity content is too long [255064703] for the configured buffer limit [104857600]"
}
},
"status": 400
}

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) index creation, mappings, settings, query etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. Reindex with batch size 1
   POST http://dest_host:9200/_reindex
   {
   "source": {
   "remote": {
   "host": "source_host:9200"
   },
   "index": "index name",
   "size": 5000
   },
   "dest": {
   "index": "index name"
   }
   }

Provide logs (if relevant):

[pgomulka]

I don't think we would be able to reproduce this with so little information
can you share the logs? the stacktrace from the cluster? elasticserach yaml configuration?
mappings from the source and dest indices?

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[nik9000]

If the document is very very very large reindex won't be able to store it in memory. It looks like the response from the remote host was 250mb. You probably should exclude that document. We probably should change the error message. In the past we'd resisted making the buffer size configurable. I wonder if it'd make sense to do so now. Or make it a portion of the heap or something. That buffer is part of the apache async http client - I don't think we could easily move it off heap.

[henningandersen]

@phinzin in your reproduction example (and title of this issue), you mention:

Reindex with batch size 1

But the batch size is set to 5000:

"size": 5000

did you try it out with "size": 1 too?

[phinzin]

@phinzin in your reproduction example (and title of this issue), you mention:

Reindex with batch size 1

But the batch size is set to 5000:

"size": 5000

did you try it out with "size": 1 too?

@henningandersen yes i tried already, the only thing strange here is that even i changed size to 1,
still same error message i got with same number "entity content is too long [255064703] for the configured buffer limit [104857600]"

[henningandersen]

@phinzin is a document of this size normal for your deployment? Would you perhaps be willing to find that document and figure out how it got in to ES to begin with?

For one, our http handler has a default limit of 100MB, so plain indexing such a document should fail unless that setting has been tweaked.

It could surely be updated enough times to grow that big but I would be interested in the background here before we decide on whether and/or how we will want to address this.

[phinzin]

some index contains incorrect data format and causing the issue