Reindex from remote cluster

[clintongormley]

The reindex API should be able to pull data from an index in a remote cluster. This would allow importing an index in a 1.x cluster into a cluster running Elasticsearch 5 or later.

The syntax could look like this:

POST /_reindex
{
  "source": {
    "remote": {
      "host": "192.168.1.2:9200",
      "username": "foo",
      "password": "bar"
    },
    "index": "twitter",
    "query": {
      "match": {
        "user": "kimchy"
      }
    }
  },
  "dest": {
    "index": "new_twitter"
  }
}

The username and password and any other options within remote would be used to configure the HTTP client contacting the remote server, and the query etc would be passed to the remote server without parsing (which deals with any API changes).

[nik9000]

I like the syntax. I think we'll need the http based java API for it though.

[clintongormley]

Not necessarily. A simple HTTP client should be sufficient for this.

[ayushsangani]

+1 @nik9000 eagerly waiting for this to be released in 1.x

[nik9000]

+1 @nik9000 eagerly waiting for this to be released in 1.x

It'll be a 5.0 or 5.1 feature depending on when I get time to really start it. You should be able to connect to 1.x with it though.

[rgb4268]

Is the _source._ingest feature working to get metadata fields from a remote source? I can't get it to work. Have tried posting in forum, no response.

[ghost]

Could you tell what would happen to data that is created or updated when performing reindex api?

[nik9000]

Could you tell what would happen to data that is created or updated when performing reindex api?

Reindex sees a consistent view of the data. It won't notice the changes.

[ghost]

So how to add that changes.

[nik9000]

So how to add that changes.

We don't have native support for that. If you ask around on discuss.elastic.co you might find people that have some solutions.

[kirvk548]

Hey Guys,

Have you tried changing the remote host from this format "host": "192.168.1.2:9200", to this format "host": "https://something.com" ?

Thanks for the help!