Snapshot of ES data based on node.attr.box_type field option

[jayannah]

/

Snapshot of ES data based on node.attr.box_type field option

Elasticsearch version: 5.4.2
Plugins installed: [] S3-repository, X-pack

JVM version (java -version): 1.8.73

OS version (uname -a if on a Unix-like system): 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Scenario:
We have ES cluster deployed in 3 AWS availability Zones (AZs). We are using node.attr.box_type in elasticsearch.yml file to differentiate the ES node in each AZs. We would like to schedule separate backup calls for each AZs..

Hence we are looking for option to specify "node.attr.box_type" with value "us-east-1a" so that it will backup only the data from ES node set to "node.attr.box_type" as "us-east-1a"

We didn't find such options in the documentation https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

Steps to reproduce:

1. Create ES cluster in us-east-1a and us-east-1b
2. 50% of data node will be set "node.attr.box_type" as "us-east-1a" and rest with "node.attr.box_type" as "us-east-1b"
3. Would like to create separate s3 repository for "us-east-1a" and "us-east-1b"
4. Take backup from ES nodes in "us-east-1a" and "us-east-1b" separately and store them in different S3 bucket if required..

Note: Our usecase is more complex, just made it simple to understand the requirement.

[rjernst]

Snapshots are of an index, not a node. So you would not "take backup of ES nodes", you create a snapshot of a particular index within the cluster. Given that, I don't see this feature likely to happen, but I will mark for discussion to let others respond with their thoughts.

[jayannah]

Hi All - Any other suggestions?

[jasontedor]

I agree with @rjernst, we are not going to add a perspective of snapshots that would be node-based.