repository-s3 plugin breaking changes in 5.4.0: can't see oldest snapshots

[MaxymVlasov]

Elasticsearch version (bin/elasticsearch --version):
Version: 5.4.3, Build: eed30a8/2017-06-22T00:34:03.743Z, JVM: 1.8.0_171

Plugins installed: []
repository-s3

JVM version (java -version):
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux ip-10-0-6-252 4.4.0-1052-aws #61-Ubuntu SMP Mon Feb 12 23:05:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
We have ES 5.2, make snapshot per index everyday and store them to S3.
Now we prepare migration to ES 6.2 and find next issue: ES 6.2 didn't see ES 5.2 S3 snapshots.
Within 5 days of debug, I found next things:
ES 5.4.0 and newest:

• see 5.2 FS snapshots
• don't see 5.2 S3 snapshots
• see 6.2 S3 snapshots
• Restored 5.2 snapshot/index can be stored as 5.4 S3 snapshot, and 6.2 their see

ES 5.3.3 and oldest:

• see 5.2 FS snapshots
• see 5.2 S3 snapshots
• don't see 5.4/6.2 S3 snapshots

looks like something goes wrong on split AWS Cloud plugin to 2 plugins (git diff 65e26e66dad..eed30a8e9bb plugins/repository-s3/)

Steps to reproduce:

AWS

1. Register on AWS
2. create EC2 instance with ubuntu 16.04 and attach Elastic IP to instance
3. Create user with full access to S3 or with recommended settings
4. Create S3 bucket and add Bucket policy (your guide)
5. Go to your EC2 by ssh

EC2 instance:

1. Download and ES 5.3 using tar.gz
2. Install plugin repository-s3
3. Add settings like this to config/elasticsearch.yml

cloud:
  aws:
    access_key: XXX
    secret_key: XXXXXX
    s3:
      signer: S3SignerType

path.repo: ["/home/ubuntu/local_backup"]

path:
  data: /home/ubuntu/elasticsearch-5.2.0/data

4. Run ES and create some index if you haven't, for example, using Kibana.
5. Create snapshot repos (S3 & FS) like this:

curl -X PUT "http://127.0.0.1:9200/_snapshot/s3_elastic_log" -H 'Content-Type: application/json' -d'
{
    "type": "s3",
    "settings": {
        "bucket": "BUCKET",
        "region": "REGION",
        "base_path": "/path/to/logs/"
    }
}'

curl -X PUT "localhost:9200/_snapshot/local_backup" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/home/ubuntu/local_backup"
  }
}
'

6. Create snapshots to S3 & FS.
7. Make Rolling upgrade to 5.4. And repeat steps 1-3 so that ES 5.4 look at ES 5.3 data and settings.
   
   Also you need export ES_PATH_CONF like this export ES_PATH_CONF=/home/ubuntu/elasticsearch-5.2.0
8. After upgrade to 5.4 you can see FS snapshot and can't see S3 snapshot
   Using curl localhost:9200/_snapshot/s3_elastic_lo/_all and curl localhost:9200/_snapshot/local_backup/_all

[elasticmachine]

Pinging @elastic/es-distributed

[tlrx]

@MaxymVlasov Thanks for reporting.

I tried to reproduce locally (ie, not on an EC2 instance) and snapshots created on 5.2.0 can be listed and restored correctly on 6.2.3.

I have some questions:

• can you try to reproduce the behavior but not using EC2?
• do you have any error in logs or in the HTTP responses?

Also, it's very important to NOT have multiple clusters in different versions writing to the same repository. It means that if you created snapshots on 5.2.0 and later upgraded your cluster to 5.4.0 you should never create or delete a snapshot using a 5.2.0 version as it corrupts the repository files.

[MaxymVlasov]

@tlrx Thanks for answer.
Are you using AWS S3 for snapshoting?

OK, I will try repoduce on local VM and then write what I got.

do you have any error in logs or in the HTTP responses

No.

Thanks for warning. But snapshots <5.4 in test looks like isolated from >=5.4 so all works fine.

[tlrx]

@MaxymVlasov Thanks. I'm wondering if your repository got corrupted at some point because of different versions writing to it. I'm going to close this issue for now, but please feel free to reopen if it appears again.