Skip to content

Allow date math for naming newly-created snapshots (#7939) #30479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
May 16, 2018
Merged

Allow date math for naming newly-created snapshots (#7939) #30479

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4055c34
Automatic snapshot naming (#7939)
May 8, 2018
4b20260
Automatic snapshot naming #7939
May 11, 2018
b168ec8
Automatic snapshot naming #7939
May 11, 2018
256c3c7
#7939 dropped unnecessary new lines in SnapshotBlocksIT
May 15, 2018
117495f
#7939 use simple get() instead of execute().actionGet()
May 15, 2018
ff244bd
#7939 fix test taking into account potential day rollover
May 15, 2018
4648145
#7939 doc adjusted
May 15, 2018
04404d0
Merge remote-tracking branch 'remotes/origin/master' into 7939
May 15, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/reference/modules/snapshots.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,20 @@ By setting `include_global_state` to false it's possible to prevent the cluster
the snapshot. By default, the entire snapshot will fail if one or more indices participating in the snapshot don't have
all primary shards available. This behaviour can be changed by setting `partial` to `true`.

Snapshot names can be automatically derived using <<date-math-index-names,date math expressions>>, similarly as when creating
new indices. Note that special characters need to be URI encoded.

For example, creating a snapshot with the current day in the name, like `snapshot-2018.05.11`, can be achieved with
the following command:
[source,js]
-----------------------------------
# PUT /_snapshot/my_backup/<snapshot-{now/d}>
PUT /_snapshot/my_backup/%3Csnapshot-%7Bnow%2Fd%7D%3E
-----------------------------------
// CONSOLE
// TEST[continued]


The index snapshot process is incremental. In the process of making the index snapshot Elasticsearch analyses
the list of the index files that are already stored in the repository and copies only files that were created or
changed since the last snapshot. That allows multiple snapshots to be preserved in the repository in a compact form.
Expand Down
7 changes: 4 additions & 3 deletions ...rg/elasticsearch/action/admin/cluster/snapshots/create/TransportCreateSnapshotAction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,9 @@ protected ClusterBlockException checkBlock(CreateSnapshotRequest request, Cluste

@Override
protected void masterOperation(final CreateSnapshotRequest request, ClusterState state, final ActionListener<CreateSnapshotResponse> listener) {
final String snapshotName = indexNameExpressionResolver.resolveDateMathExpression(request.snapshot());
SnapshotsService.SnapshotRequest snapshotRequest =
new SnapshotsService.SnapshotRequest(request.repository(), request.snapshot(), "create_snapshot [" + request.snapshot() + "]")
new SnapshotsService.SnapshotRequest(request.repository(), snapshotName, "create_snapshot [" + snapshotName + "]")
.indices(request.indices())
.indicesOptions(request.indicesOptions())
.partial(request.partial())
Expand All @@ -87,7 +88,7 @@ public void onResponse() {
@Override
public void onSnapshotCompletion(Snapshot snapshot, SnapshotInfo snapshotInfo) {
if (snapshot.getRepository().equals(request.repository()) &&
snapshot.getSnapshotId().getName().equals(request.snapshot())) {
snapshot.getSnapshotId().getName().equals(snapshotName)) {
listener.onResponse(new CreateSnapshotResponse(snapshotInfo));
snapshotsService.removeListener(this);
}
Expand All @@ -96,7 +97,7 @@ public void onSnapshotCompletion(Snapshot snapshot, SnapshotInfo snapshotInfo) {
@Override
public void onSnapshotFailure(Snapshot snapshot, Exception e) {
if (snapshot.getRepository().equals(request.repository()) &&
snapshot.getSnapshotId().getName().equals(request.snapshot())) {
snapshot.getSnapshotId().getName().equals(snapshotName)) {
listener.onFailure(e);
snapshotsService.removeListener(this);
}
Expand Down
38 changes: 38 additions & 0 deletions server/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse;
import org.elasticsearch.action.index.IndexRequestBuilder;
import org.elasticsearch.action.support.ActiveShardCount;
import org.elasticsearch.client.AdminClient;
import org.elasticsearch.client.Client;
import org.elasticsearch.client.node.NodeClient;
import org.elasticsearch.cluster.ClusterState;
Expand All @@ -41,6 +42,7 @@
import org.elasticsearch.cluster.SnapshotDeletionsInProgress;
import org.elasticsearch.cluster.SnapshotsInProgress;
import org.elasticsearch.cluster.health.ClusterHealthStatus;
import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver;
import org.elasticsearch.cluster.metadata.MetaData;
import org.elasticsearch.cluster.metadata.MetaDataIndexStateService;
import org.elasticsearch.cluster.routing.allocation.decider.EnableAllocationDecider;
Expand All @@ -49,13 +51,15 @@
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.ParseField;
import org.elasticsearch.common.Priority;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.io.stream.NamedWriteableRegistry;
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.common.io.stream.Writeable;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.settings.SettingsFilter;
import org.elasticsearch.common.unit.ByteSizeUnit;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.common.util.set.Sets;
import org.elasticsearch.common.xcontent.NamedXContentRegistry;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.discovery.zen.ElectMasterService;
Expand All @@ -68,6 +72,7 @@
import org.elasticsearch.rest.RestController;
import org.elasticsearch.rest.RestRequest;
import org.elasticsearch.rest.RestResponse;
import org.elasticsearch.rest.RestStatus;
import org.elasticsearch.rest.action.admin.cluster.RestClusterStateAction;
import org.elasticsearch.rest.action.admin.cluster.RestGetRepositoriesAction;
import org.elasticsearch.snapshots.mockstore.MockRepository;
Expand Down Expand Up @@ -96,6 +101,7 @@
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.greaterThan;
import static org.hamcrest.Matchers.greaterThanOrEqualTo;
import static org.hamcrest.Matchers.hasSize;
import static org.hamcrest.Matchers.lessThan;
import static org.hamcrest.Matchers.not;
import static org.hamcrest.Matchers.notNullValue;
Expand Down Expand Up @@ -981,6 +987,38 @@ public void testRestoreShrinkIndex() throws Exception {
ensureYellow();
}

public void testSnapshotWithDateMath() {
final String repo = "repo";
final AdminClient admin = client().admin();

final IndexNameExpressionResolver nameExpressionResolver = new IndexNameExpressionResolver(Settings.EMPTY);
final String snapshotName = "<snapshot-{now/d}>";

logger.info("--> creating repository");
assertAcked(admin.cluster().preparePutRepository(repo).setType("fs")
.setSettings(Settings.builder().put("location", randomRepoPath())
.put("compress", randomBoolean())));

final String expression1 = nameExpressionResolver.resolveDateMathExpression(snapshotName);
logger.info("--> creating date math snapshot");
CreateSnapshotResponse snapshotResponse =
admin.cluster().prepareCreateSnapshot(repo, snapshotName)
.setIncludeGlobalState(true)
.setWaitForCompletion(true)
.get();
assertThat(snapshotResponse.status(), equalTo(RestStatus.OK));
// snapshot could be taken before or after a day rollover
final String expression2 = nameExpressionResolver.resolveDateMathExpression(snapshotName);

SnapshotsStatusResponse response = admin.cluster().prepareSnapshotStatus(repo)
.setSnapshots(Sets.newHashSet(expression1, expression2).toArray(Strings.EMPTY_ARRAY))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a varargs method, so just .setSnapshots(expression1, expression2) is good enough.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

w/o a day rollover it requests two statuses with two identical names - and the result is 2 snapshot statuses - that's why I used set here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

.setIgnoreUnavailable(true)
.get();
List<SnapshotStatus> snapshots = response.getSnapshots();
assertThat(snapshots, hasSize(1));
assertThat(snapshots.get(0).getState().completed(), equalTo(true));
}

public static class SnapshottableMetadata extends TestCustomMetaData {
public static final String TYPE = "test_snapshottable";

Expand Down