Skip to content

Add API to mount a snapshot as a searchable index #53084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

dakrone
Copy link
Member

@dakrone dakrone commented Mar 3, 2020

This commit adds an API for mounting a snapshot as a searchable index.

Internally, a request is made to the repository to fetch metadata information necessary from the
snapshot in question, which is then passed to the regular snapshot restore mechanism with some
specialized settings.

Assume you have an existing snapshot repository and snapshot, the API looks like:

POST /my-new-index/_snapshot/mount?wait_for_completion=true
{
  "repository": "my-s3-repo",
  "snapshot": "production-2019-12-11-aoeufao",
  "snapshot_index": "important-data",
  "index_settings": {
    "index.number_of_replicas": 2
  },
  "ignore_index_settings": ["index.refresh_interval"]
}

While the request may appear similar to a snapshot restore request, it has slight differences. The
repository, snapshot, and snapshot_index fields are required (all other fields are optional).

The response is identical to the response when restoring a snapshot. In the above example the newly
restored/mounted index will be called my-new-index.

Relates to #50999

This commit adds an API for mounting a snapshot as a searchable index.

Internally, a request is made to the repository to fetch metadata information necessary from the
snapshot in question, which is then passed to the regular snapshot restore mechanism with some
specialized settings.

Assume you have an existing snapshot repository and snapshot, the API looks like:

```json
POST /my-new-index/_snapshot/mount?wait_for_completion=true
{
  "repository": "my-s3-repo",
  "snapshot": "production-2019-12-11-aoeufao",
  "snapshot_index": "important-data",
  "index_settings": {
    "index.number_of_replicas": 2
  },
  "ignore_index_settings": ["index.refresh_interval"]
}
```

While the request may appear similar to a snapshot restore request, it has slight differences. The
`repository`, `snapshot`, and `snapshot_index` fields are required (all other fields are optional).

The response is identical to the response when restoring a snapshot.

Relates to elastic#50999
@dakrone dakrone added the :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs label Mar 3, 2020
@dakrone dakrone requested review from tlrx and DaveCTurner March 3, 2020 21:16
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

@dakrone
Copy link
Member Author

dakrone commented Mar 3, 2020

Note for reviewers: this adds the API, but I tried to make as minimal changes as possible to the rest of the searchable snapshot infrastructure. This does not remove the "searchable" repository type, and only changes a few assertions made about the repository type where necessary to have mounting work with no issues.

Copy link
Member

@tlrx tlrx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks@dakrone, and thanks for having minimized the changes. I made a first bunch of comments (I didn't look at the tests yet).

Copy link
Contributor

@DaveCTurner DaveCTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dakrone, I left a few small comments too.

@DaveCTurner DaveCTurner requested a review from tlrx March 6, 2020 13:58
@DaveCTurner DaveCTurner mentioned this pull request Mar 6, 2020
19 tasks
@DaveCTurner DaveCTurner self-assigned this Mar 6, 2020
@DaveCTurner
Copy link
Contributor

@elasticmachine please run elasticsearch-ci/bwc
@elasticmachine please run elasticsearch-ci/default-distro

Copy link
Member

@tlrx tlrx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, thanks for picking this work David. I've left some comments, but nothing to worry about.

@DaveCTurner
Copy link
Contributor

Everything addressed @tlrx, I think, this is ready for another look.

@DaveCTurner DaveCTurner requested a review from tlrx March 9, 2020 11:43
Copy link
Member

@tlrx tlrx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - if CI is happy, I'll be happy

@DaveCTurner DaveCTurner merged commit 88b01a2 into elastic:feature/searchable-snapshots Mar 10, 2020
@DaveCTurner
Copy link
Contributor

Thanks @dakrone and @tlrx 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants