Skip to content

Avoid warnings for non-system indexes using fast_refresh #124294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Mar 11, 2025
Merged

Avoid warnings for non-system indexes using fast_refresh #124294

merged 3 commits into from
Mar 11, 2025

Conversation

kkrik-es
Copy link
Contributor

@kkrik-es kkrik-es commented Mar 7, 2025

LogsdbIndexModeSettingsProvider creates a transient MappingService to check the index mappings for information relevant to logsdb mode. This may trigger unrelated warnings, e.g. for non-system indexes configured with [fast_refresh=true]. To avoid the noise, we can always unset this setting as it's not relevant to the logsdb functionality.

Fixes #124284

@kkrik-es kkrik-es added >non-issue auto-backport Automatically create backport pull requests when merged Team:StorageEngine :StorageEngine/Logs You know, for Logs v8.18.1 v9.0.1 labels Mar 7, 2025
@kkrik-es kkrik-es self-assigned this Mar 7, 2025
@kkrik-es kkrik-es marked this pull request as ready for review March 7, 2025 08:47
@kkrik-es kkrik-es requested review from martijnvg and ywangd March 7, 2025 08:47
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

martijnvg
martijnvg reviewed Mar 7, 2025
View reviewed changes
...gin/logsdb/src/main/java/org/elasticsearch/xpack/logsdb/LogsdbIndexModeSettingsProvider.java
@@ -309,7 +309,8 @@ private IndexMetadata buildIndexMetadataForMapperService(
.put(indexTemplateAndCreateRequestSettings)
.put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, dummyShards)
.put(IndexMetadata.SETTING_NUMBER_OF_REPLICAS, shardReplicas)
.put(IndexMetadata.SETTING_INDEX_UUID, UUIDs.randomBase64UUID());
.put(IndexMetadata.SETTING_INDEX_UUID, UUIDs.randomBase64UUID())
.put(IndexSettings.INDEX_FAST_REFRESH_SETTING.getKey(), false); // Avoid warnings for non-system indexes.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a relative easy way to test the problem reported in the issue?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a warning.. I tried to reproduce but it probably needs more than just the setting. @ywangd too for ideas.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The warning is for serverless deployments since fast_refresh was a feature for serverless only. In a serverless deployment, you can trigger this warning by just creating a related system index, e.g.

PUT .security-profile-8

In theory, the alternative fix can be removing the fast_refresh setting from all system indices since the setting is a noop today. But I am not sure whether that could still have other implications. Overall fixing it here feels safer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I stand corrected, it's not just a warning - the provider doesn't work correctly when this trips. Not a real issue as system indexes are not in logsdb mode, but still easy to test.

martijnvg
martijnvg approved these changes Mar 11, 2025
View reviewed changes
Copy link
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks Kostas!

ywangd
ywangd approved these changes Mar 11, 2025
View reviewed changes
Copy link
Member

@ywangd ywangd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM2

@kkrik-es kkrik-es merged commit 04917c5 into elastic:main Mar 11, 2025
17 checks passed
@kkrik-es kkrik-es mentioned this pull request Mar 11, 2025
Closed
kkrik-es added a commit to kkrik-es/elasticsearch that referenced this pull request Mar 11, 2025
@kkrik-es
Avoid warnings for non-system indexes using fast_refresh (elastic#1…
ae5f7e1 
…24294)

* Update LogsdbIndexModeSettingsProvider.java

* add test
This was referenced Mar 11, 2025
Merged
Merged
@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
✅ 8.18
✅ 8.x
✅ 9.0

kkrik-es added a commit to kkrik-es/elasticsearch that referenced this pull request Mar 11, 2025
@kkrik-es
Avoid warnings for non-system indexes using fast_refresh (elastic#1…
03bc3c4 
…24294)

* Update LogsdbIndexModeSettingsProvider.java

* add test
kkrik-es added a commit to kkrik-es/elasticsearch that referenced this pull request Mar 11, 2025
@kkrik-es
Avoid warnings for non-system indexes using fast_refresh (elastic#1…
3d02758 
…24294)

* Update LogsdbIndexModeSettingsProvider.java

* add test
elasticsearchmachine pushed a commit that referenced this pull request Mar 11, 2025
@kkrik-es
[9.0] Avoid warnings for non-system indexes using fast_refresh (#12…
8b78ee2 
…4294) (#124547)

* Avoid warnings for non-system indexes using `fast_refresh` (#124294)

* Update LogsdbIndexModeSettingsProvider.java

* add test

* Update LogsdbIndexModeSettingsProviderTests.java
elasticsearchmachine pushed a commit that referenced this pull request Mar 11, 2025
@kkrik-es
[8.x] Avoid warnings for non-system indexes using fast_refresh (#12…
9ddcf24 
…4294) (#124546)

* Avoid warnings for non-system indexes using `fast_refresh` (#124294)

* Update LogsdbIndexModeSettingsProvider.java

* add test

* Update LogsdbIndexModeSettingsProviderTests.java
@kkrik-es kkrik-es deleted the fix/124284 branch March 11, 2025 17:14
albertzaharovits pushed a commit to albertzaharovits/elasticsearch that referenced this pull request Mar 13, 2025
@kkrik-es @albertzaharovits
Avoid warnings for non-system indexes using fast_refresh (elastic#1…
2c80241 
…24294)

* Update LogsdbIndexModeSettingsProvider.java

* add test
jfreden pushed a commit to jfreden/elasticsearch that referenced this pull request Mar 13, 2025
@kkrik-es @jfreden
Avoid warnings for non-system indexes using fast_refresh (elastic#1…
701c447 
…24294)

* Update LogsdbIndexModeSettingsProvider.java

* add test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@martijnvg martijnvg martijnvg approved these changes

@ywangd ywangd ywangd approved these changes

Assignees

@kkrik-es kkrik-es

Labels
auto-backport Automatically create backport pull requests when merged >non-issue :StorageEngine/Logs You know, for Logs Team:StorageEngine v8.18.1 v8.19.0 v9.0.1 v9.1.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

LogsdbIndexModeSettingsProvider generates unnecesary warnings for system indices
4 participants
@kkrik-es @elasticsearchmachine @martijnvg @ywangd