Skip to content

Rollover During Partial Snapshot of a Datastream Should not Exclude the DS from the Snapshot #68536

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
original-brownbear opened this issue Feb 4, 2021 · 2 comments · Fixed by #69364
Closed

Rollover During Partial Snapshot of a Datastream Should not Exclude the DS from the Snapshot #68536

original-brownbear opened this issue Feb 4, 2021 · 2 comments · Fixed by #69364
Assignees
@original-brownbear
Labels
:Data Management/Data streams Data streams and their lifecycles :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement Team:Data Management Meta label for data/management team Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.

Comments

@original-brownbear
Copy link
Member

Background: #68258 fixed the issue that rolling over a datastream during a snapshot would break the snapshot finalization because the snapshot would not include the newest index of the datastream during finalization but also would not have snapshotted the datastream metadata before rollover, leaving the cluster without a consistent pair of metadata and indices.
The current fix to this situation was to leave datastreams out of a partial snapshot if they were rolled over during the snapshot completely.

We discussed this in the core-features meeting and we decided that this behavior needs to be improved given that data-streams will be used more and more and start to work with time-based rollovers, increasing the chance of this type of collision between snapshot and rollover and thus the chance of never getting a snapshot for a certain datastream if partial snapshots are used.
I see a fairly direct fix to this and I'll open a PR for it.
@original-brownbear original-brownbear added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs :Data Management/Data streams Data streams and their lifecycles labels Feb 4, 2021
@original-brownbear original-brownbear self-assigned this Feb 4, 2021
@elasticmachine elasticmachine added Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. Team:Data Management Meta label for data/management team labels Feb 4, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (Team:Core/Features)

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Feb 22, 2021
@original-brownbear
Improve Partial Snapshot Rollover Behavior
023716f 
Using new reconciliation functionality to not needlessly drop rolling over
data streams from the final snapshot.

closes elastic#68536
@original-brownbear original-brownbear mentioned this issue Feb 22, 2021
Merged
original-brownbear added a commit that referenced this issue Feb 22, 2021
@original-brownbear
Improve Partial Snapshot Rollover Behavior (#69364)
d334e3b 
Using new reconciliation functionality to not needlessly drop rolling over
data streams from the final snapshot.

closes #68536
@original-brownbear original-brownbear mentioned this issue Feb 22, 2021
Merged
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Feb 22, 2021
@original-brownbear
Improve Partial Snapshot Rollover Behavior (elastic#69364)
3828bc0 
Using new reconciliation functionality to not needlessly drop rolling over
data streams from the final snapshot.

closes elastic#68536
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Feb 22, 2021
@original-brownbear
Improve Partial Snapshot Rollover Behavior (elastic#69364)
6589ece 
Using new reconciliation functionality to not needlessly drop rolling over
data streams from the final snapshot.

closes elastic#68536
@original-brownbear original-brownbear mentioned this issue Feb 22, 2021
Merged
original-brownbear added a commit that referenced this issue Feb 23, 2021
@original-brownbear
Improve Partial Snapshot Rollover Behavior (#69364) (#69370)
b41784f 
Using new reconciliation functionality to not needlessly drop rolling over
data streams from the final snapshot.

closes #68536
original-brownbear added a commit that referenced this issue Feb 23, 2021
@original-brownbear
Improve Partial Snapshot Rollover Behavior (#69364) (#69369)
e2530b6 
Using new reconciliation functionality to not needlessly drop rolling over
data streams from the final snapshot.

closes #68536
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
:Data Management/Data streams Data streams and their lifecycles :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement Team:Data Management Meta label for data/management team Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improve Partial Snapshot Rollover Behavior original-brownbear/elasticsearch
2 participants
@original-brownbear @elasticmachine