Frozen default cache size #71844

henningandersen · 2021-04-19T13:46:28Z

This commit adds a default cache size to frozen tier of the greater of
90% and total disk size minus 100 GB.

Will follow-up with two other items:

Fail nodes on startup when frozen cache has a size on multiple data paths. The tricky bit here is mainly the test framework. (added in Reject multiple data paths on frozen capable nodes #71896)
Add a frozen specific flood stage setting that has a max headroom of 20GB (added in Add separate flood stage limit for frozen #71855)

This commit adds a default cache size to frozen tier of the greater of 90% and total disk size minus 100 GB. Additionally, configuring a frozen cache is now warned against on nodes with multiple data paths.

elasticmachine · 2021-04-19T13:46:32Z

Pinging @elastic/es-distributed (Team:Distributed)

Dedicated frozen nodes can survive less headroom than other data nodes. This commits introduces a separate flood stage threshold for frozen as well as an accompanying max_headroom setting that caps the amount of free space necessary on frozen. Relates elastic#71844

…ult_cache_size

Some functionality will no longer work with multiple data paths and in order to run integration tests for that, we need the capability to force a single data path for those tests. Relates elastic#71844

henningandersen · 2021-04-20T05:17:55Z

@elasticmachine run elasticsearch-ci/2

Some functionality will no longer work with multiple data paths and in order to run integration tests for that, we need the capability to force a single data path for those tests. Relates #71844

Nodes with a frozen cache no longer supports multiple data paths. This simplifies cache sizing and avoids the need to support multiple cache files. Relates elastic#71844

ywelsch

I've left only minor comments, o.w. looking good.

ywelsch · 2021-04-20T08:12:22Z

docs/reference/searchable-snapshots/index.asciidoc

 <<data-frozen-node,`data_frozen`>> role.

+IMPORTANT: You can only configure these settings on nodes that use a single


Perhaps more of a question to the docs team: I wonder if we should have two "important" notices following each other, or whether it should be all folded into one.

I tried to find other places with two, but found none and did find some where it could have been split into two. So I collapsed it to one in 1b8ede4

ywelsch · 2021-04-20T08:22:36Z

...c/main/java/org/elasticsearch/xpack/searchablesnapshots/cache/shared/FrozenCacheService.java

+        Setting.Property.NodeScope
+    );
+
+    private static boolean isDedicatedFrozen(Settings settings) {


I'm surprised this method did not already exist.

Maybe add this method to DiscoveryNode?

I centralized this to DiscoveryNode in 616940c

ywelsch · 2021-04-20T08:26:50Z

...c/main/java/org/elasticsearch/xpack/searchablesnapshots/cache/shared/FrozenCacheService.java

-        this.cacheSize = SNAPSHOT_CACHE_SIZE_SETTING.get(settings).getBytes();
+        FsInfo.Path pathInfo;
+        try {
+            pathInfo = FsProbe.getFSInfo(environment.nodePaths()[0]);


environment.nodeDataPaths()[0] here

Fixed in 7280ae8

tlrx

LGTM

tlrx · 2021-04-20T10:09:24Z

docs/reference/searchable-snapshots/index.asciidoc

+must have one or more nodes with a shared cache available. By default,
+dedicated frozen data tier nodes (nodes with the `data_frozen` role and no other
+data roles) have a shared cache configured using the greater of 90% of total
+disk space and total disk space subtracted a headroom of 100GB.


I wonder if inserting an example here would help.

Good idea, added examples in 53f5982

tlrx · 2021-04-20T10:11:16Z

docs/reference/searchable-snapshots/index.asciidoc

 <<data-frozen-node,`data_frozen`>> role.

+IMPORTANT: You can only configure these settings on nodes that use a single
+<<path-settings,data path>>


Suggested change

<<path-settings,data path>>

<<path-settings,data path>>.

Fixed this when turning it into one IMPORTANT note in 1b8ede4

tlrx · 2021-04-20T10:14:43Z

server/src/test/java/org/elasticsearch/common/unit/RelativeByteSizeValueTests.java

+        double value = (double) randomIntBetween(1, 100) / 100;
+        RelativeByteSizeValue parsed = RelativeByteSizeValue.parseRelativeByteSizeValue(Double.toString(value), "test");
+        assertThat(parsed.getRatio().getAsRatio(),
+            equalTo(value));


nit: fits on the same line as the previous one

Thanks, see 3160c5f

tlrx · 2021-04-20T10:24:45Z

...c/main/java/org/elasticsearch/xpack/searchablesnapshots/cache/shared/FrozenCacheService.java

+        Setting.Property.NodeScope
+    );
+
+    private static boolean isDedicatedFrozen(Settings settings) {


Maybe add this method to DiscoveryNode?

ywelsch

LGTM

ywelsch · 2021-04-20T11:58:02Z

...c/main/java/org/elasticsearch/xpack/searchablesnapshots/cache/shared/FrozenCacheService.java

+        try {
+            totalFsSize = FsProbe.getTotal(Environment.getFileStore(environment.nodeDataPaths()[0]));
+        } catch (IOException e) {
+            throw new IllegalStateException("unable to probe size of filesystem [" + environment.nodePaths()[0] + "]");


also change to environment.nodeDataPaths

Thanks, sorry for missing that one 1b3e031

Nodes with a frozen cache no longer supports multiple data paths. This simplifies cache sizing and avoids the need to support multiple cache files. Relates #71844

Nodes with a frozen cache no longer supports multiple data paths. This simplifies cache sizing and avoids the need to support multiple cache files. Relates elastic#71844

Dedicated frozen nodes can survive less headroom than other data nodes. This commits introduces a separate flood stage threshold for frozen as well as an accompanying max_headroom setting that caps the amount of free space necessary on frozen. Relates #71844

Dedicated frozen nodes can survive less headroom than other data nodes. This commits introduces a separate flood stage threshold for frozen as well as an accompanying max_headroom setting that caps the amount of free space necessary on frozen. Relates elastic#71844

…ult_cache_size

Nodes with a frozen cache no longer supports multiple data paths. This simplifies cache sizing and avoids the need to support multiple cache files. Relates #71844

Dedicated frozen nodes can survive less headroom than other data nodes. This commits introduces a separate flood stage threshold for frozen as well as an accompanying max_headroom setting that caps the amount of free space necessary on frozen. Relates #71844

This commit adds a default cache size to frozen tier of the greater of 90% and total disk size minus 100 GB.

Changes: * Renames 'full copy searchable snapshot' to 'fully mounted index.' * Renames 'shared cache searchable snapshot' to 'partially mounted index.' * Removes some unneeded cache setup instructions for the frozen tier. We added a default cache size with #71844.

Frozen default cache size

1f35be1

This commit adds a default cache size to frozen tier of the greater of 90% and total disk size minus 100 GB. Additionally, configuring a frozen cache is now warned against on nodes with multiple data paths.

henningandersen added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.0.0 v7.13.1 labels Apr 19, 2021

elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Apr 19, 2021

henningandersen requested a review from ywelsch April 19, 2021 15:00

henningandersen mentioned this pull request Apr 19, 2021

Add separate flood stage limit for frozen #71855

Merged

Merge remote-tracking branch 'origin/master' into enhance_frozen_defa…

3fa17eb

…ult_cache_size

henningandersen mentioned this pull request Apr 19, 2021

Add force single data path option for integ tests #71868

Merged

henningandersen mentioned this pull request Apr 20, 2021

Reject multiple data paths on frozen capable nodes #71896

Merged

henningandersen added 2 commits April 20, 2021 09:40

Remove overlap to elastic#71896

e7b5ab6

Slightly more friendly on large full disks.

9831d9d

henningandersen requested a review from tlrx April 20, 2021 07:53

henningandersen added a commit to henningandersen/elasticsearch that referenced this pull request Apr 20, 2021

Remove most overlap to elastic#71844

4d168b4

ywelsch reviewed Apr 20, 2021

View reviewed changes

henningandersen added 4 commits April 20, 2021 10:58

Use nodeDataPaths()

7280ae8

Centralize isDedicatedFrozenNode

616940c

Only one IMPORTANT note.

1b8ede4

checkstyle

2c7f194

tlrx approved these changes Apr 20, 2021

View reviewed changes

One line

3160c5f

Add examples.

53f5982

henningandersen requested a review from ywelsch April 20, 2021 11:46

ywelsch approved these changes Apr 20, 2021

View reviewed changes

Also use nodeDataPaths in message.

1b3e031

henningandersen mentioned this pull request Apr 20, 2021

Reject multiple data paths on frozen capable nodes (#71896) #71939

Merged

henningandersen mentioned this pull request Apr 20, 2021

Add separate flood stage limit for frozen (#71855) #71941

Merged

henningandersen added 2 commits April 20, 2021 16:27

Merge remote-tracking branch 'origin/master' into enhance_frozen_defa…

9165ac7

…ult_cache_size

spotless

3b40f76

henningandersen merged commit c57fbe8 into elastic:master Apr 20, 2021

henningandersen added a commit to henningandersen/elasticsearch that referenced this pull request Apr 20, 2021

Frozen default cache size (elastic#71844)

2767df3

This commit adds a default cache size to frozen tier of the greater of 90% and total disk size minus 100 GB.

henningandersen added the backport pending label Apr 20, 2021

henningandersen added a commit that referenced this pull request Apr 20, 2021

Frozen default cache size (#71844) (#71967)

17fcd8c

This commit adds a default cache size to frozen tier of the greater of 90% and total disk size minus 100 GB.

henningandersen removed the backport pending label Apr 21, 2021

This was referenced May 3, 2021

[DOCS] Update self-managed frozen tier instructions #72626

Closed

[DOCS] Rename mount types for searchable snapshots #72699

Merged

mark-vieira added v7.13.0 and removed v7.13.1 labels May 5, 2021

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

		<<data-frozen-node,`data_frozen`>> role.

		IMPORTANT: You can only configure these settings on nodes that use a single

Frozen default cache size #71844

Frozen default cache size #71844

Uh oh!

Conversation

henningandersen commented Apr 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Apr 19, 2021

Uh oh!

henningandersen commented Apr 20, 2021

Uh oh!

ywelsch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tlrx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ywelsch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

henningandersen commented Apr 19, 2021 •

edited

Loading