Add `index.mode: time_series` #77626

nik9000 · 2021-09-13T13:45:13Z

This adds a setting to enable time series mode that is hidden by the
es.index_mode_feature_flag_registered feature flag. All it does right
now is make sure you haven't configured index sorting and partitioning.
This gives us a place to "hang" all of our further work on time series
mode.

Time series mode will entirely take over index sorting. We don't believe
you'll ever be able to configure sorting yourself when you are in time
series mode. We don't expect time series mode to support index
partitioning when we first build it but we'd like to get there.

elasticmachine · 2021-09-13T13:45:17Z

Pinging @elastic/es-analytics-geo (Team:Analytics)

nik9000 · 2021-09-13T13:49:32Z

This is a tiny tiny tiny part of the rebuild of #75638. But opening it is a good sign! I feel like I really can get to the meat of the rebuild soon.

This adds a setting to enable time series mode that is hidden by the `es.index_mode_feature_flag_registered` feature flag. All it does right now is make sure you haven't configured index sorting and partitioning. This gives us a place to "hang" all of our further work on time series mode. Time series mode will entirely take over index sorting. We don't believe you'll ever be able to configure sorting yourself when you are in time series mode. We don't expect time series mode to support index partitioning when we first build it but we'd like to get there.

mark-vieira

LGTM. I assume you ran a test build locally with -Dbuild.snapshot=false to see the tests were ok? We don't actually run in that configuration in PR or intake jobs.

nik9000 · 2021-09-13T16:14:41Z

LGTM. I assume you ran a test build locally with -Dbuild.snapshot=false to see the tests were ok? We don't actually run in that configuration in PR or intake jobs.

I hadn't. Let me give that a go.

nik9000 · 2021-09-13T16:49:28Z

I hadn't. Let me give that a go.

Well first time trying it everything crashed. I've stared with --max-workers 2 to see how that goes.

nik9000 · 2021-09-13T16:50:47Z

Well first time trying it everything crashed. I've stared with --max-workers 2 to see how that goes.

Yeah, the load average was 130 and the machine has 12 cores....

With -max-workers 2 I'm getting load average of 10.

nik9000 · 2021-09-13T17:44:35Z

After a few false starts I have some progress!

<====---------> 34% EXECUTING [50m 57s]

It's been a long, long, long time since I ran the entire suite locally.

nik9000 · 2021-09-13T20:28:34Z

server/src/main/java/org/elasticsearch/index/IndexSettings.java

+                return IndexMode.VALIDATE_WITH_SETTINGS.iterator();
+            }
+        },
+        Property.IndexScope


This should probably also have Property.Final.

Indeed Property.Final seems like that way to go here.

Should INDEX_ROUTING_PARTITION_SIZE_SETTING be Final as well? It is not marked Final but it seems like it shouldn't be set after index creation.

I opened #77669 to talk about that.

imotov

LGTM.

imotov · 2021-09-13T21:50:44Z

server/src/main/java/org/elasticsearch/index/IndexSettings.java

+                return IndexMode.VALIDATE_WITH_SETTINGS.iterator();
+            }
+        },
+        Property.IndexScope


Indeed Property.Final seems like that way to go here.

imotov · 2021-09-13T21:59:59Z

server/src/main/java/org/elasticsearch/index/IndexSettings.java

+                return IndexMode.VALIDATE_WITH_SETTINGS.iterator();
+            }
+        },
+        Property.IndexScope


Should INDEX_ROUTING_PARTITION_SIZE_SETTING be Final as well? It is not marked Final but it seems like it shouldn't be set after index creation.

nik9000 · 2021-09-14T13:39:46Z

Lovely:

$  ./gradlew --max-workers 2 -Dbuild.snapshot=false -Dlicense.key="x-pack/license-tools/src/test/resources/public.key" -p qa check
...

BUILD FAILED in 1h 46m 30s

Failed overnight. :qa:snapshot-based-recoveries:gcs:integTest got an empty reply from the server. Checking it again locally.

I think, maybe, we need a way to make CI do this for me. Or we need some better way for me to do it locally.

nik9000 · 2021-09-14T13:44:21Z

Failed overnight. :qa:snapshot-based-recoveries:gcs:integTest got an empty reply from the server. Checking it again locally.

Didn't fail locally on second run, either with build.snapshot=false or without it. Gremlins killed my build.

nik9000 · 2021-09-14T16:50:10Z

I've tested this a bunch locally and it looks ok. I've merged to master and manually kicked off a non-snapshot build using it. I'm going to wait to backport until it passes.

This adds a setting to enable time series mode that is hidden by the `es.index_mode_feature_flag_registered` feature flag. All it does right now is make sure you haven't configured index sorting and partitioning. This gives us a place to "hang" all of our further work on time series mode. Time series mode will entirely take over index sorting. We don't believe you'll ever be able to configure sorting yourself when you are in time series mode. We don't expect time series mode to support index partitioning when we first build it but we'd like to get there.

Now that elastic#77626 has landed we can test tsdb *things* against 7.16.

nik9000 · 2021-10-01T18:08:59Z

I'm pulling this back out of 7.16 because we don't really have a chance of making a useful tsdb in time. So there isn't really any need to have the setting in that version.

nik9000 · 2021-10-01T18:09:06Z

#78570 is the revert PR

nik9000 added >non-issue v8.0.0 :StorageEngine/TSDB You know, for Metrics v7.16.0 labels Sep 13, 2021

nik9000 requested review from imotov and csoulios September 13, 2021 13:45

elasticmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Sep 13, 2021

nik9000 added 3 commits September 13, 2021 09:50

Unit test

be395ff

License header

aec26ea

mark-vieira approved these changes Sep 13, 2021

View reviewed changes

nik9000 added 2 commits September 13, 2021 12:22

Merge branch 'master' into time_series_mode_empty

645d920

Merge branch 'master' into time_series_mode_empty

89c6fc7

nik9000 added 3 commits September 13, 2021 14:55

Merge branch 'master' into time_series_mode_empty

f9149d4

Merge branch 'master' into time_series_mode_empty

bb1fdca

Merge branch 'master' into time_series_mode_empty

99ed3b3

nik9000 commented Sep 13, 2021

View reviewed changes

imotov approved these changes Sep 13, 2021

View reviewed changes

nik9000 added 4 commits September 13, 2021 18:07

Merge branch 'master' into time_series_mode_empty

dd9b276

Multinode

64e4175

Runtime fields tests

7630a33

Final

f77d271

Merge branch 'master' into time_series_mode_empty

2140d88

nik9000 added the backport pending label Sep 14, 2021

nik9000 merged commit 962976b into elastic:master Sep 14, 2021

nik9000 removed the backport pending label Sep 15, 2021

nik9000 added a commit to nik9000/elasticsearch that referenced this pull request Sep 20, 2021

Update skip after backport of elastic#77626

f367e0a

Now that elastic#77626 has landed we can test tsdb *things* against 7.16.

nik9000 mentioned this pull request Sep 20, 2021

Update skip after backport of #77626 #78014

Closed

mark-vieira mentioned this pull request Sep 22, 2021

Mixed cluster tests failing in release builds due to feature flag #78219

Closed

nik9000 removed the v7.16.0 label Oct 1, 2021

jakelandis added v8.0.0-beta1 and removed v8.0.0 labels Oct 27, 2021

Add index.mode: time_series #77626

Add index.mode: time_series #77626

Uh oh!

Conversation

nik9000 commented Sep 13, 2021

Uh oh!

elasticmachine commented Sep 13, 2021

Uh oh!

nik9000 commented Sep 13, 2021

Uh oh!

mark-vieira left a comment

Choose a reason for hiding this comment

Uh oh!

nik9000 commented Sep 13, 2021

Uh oh!

nik9000 commented Sep 13, 2021

Uh oh!

nik9000 commented Sep 13, 2021

Uh oh!

nik9000 commented Sep 13, 2021

Uh oh!

nik9000 Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

imotov Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

imotov Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

nik9000 Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

imotov left a comment

Choose a reason for hiding this comment

Uh oh!

imotov Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

imotov Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

nik9000 commented Sep 14, 2021

Uh oh!

nik9000 commented Sep 14, 2021

Uh oh!

nik9000 commented Sep 14, 2021

Uh oh!

nik9000 commented Oct 1, 2021

Uh oh!

nik9000 commented Oct 1, 2021

Uh oh!

Uh oh!

Add `index.mode: time_series` #77626

Add `index.mode: time_series` #77626