Skip to content

Commit c8f2089

Browse files
committed
Encourage using legacy template API (legacy and composable options) with custom template. ILM settings of config will be ignored when using with custom template.
1 parent 03db839 commit c8f2089

File tree

6 files changed

+96
-17
lines changed

6 files changed

+96
-17
lines changed

Diff for: CHANGELOG.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
## 11.12.3
2-
- Legacy template API with Elasticsearch 8.x (or up) version and installing custom template fixes. [#1102](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1102)
3-
- When using legacy template API with ES8+ versions, default template installation failed. Now the plugin encourages using custom template.
4-
- The plugin failed when it doesn't resolve ILM settings from the given custom template. Now, the plugin tries to install provided custom template as it is without touching ILM settings.
2+
- Legacy template API usage custom template installation when ILM enabled improvements. [#1102](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1102)
3+
- Use cases defined and improved when using legacy template API with any ES versions. Now the plugin encourages using custom template when except `template_api` => `auto`.
4+
- The plugin failed when it doesn't resolve ILM settings from the given custom template. Now, the plugin warns when ILM is enabled with a custom template and uses ILM settings of the template if it exists.
55

66
## 11.12.2
77
- [Doc] Fixes the broken apache http client link [#1101](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1101)

Diff for: docs/index.asciidoc

+3-2
Original file line numberDiff line numberDiff line change
@@ -1079,6 +1079,8 @@ the *$JDK_HOME/conf/security/java.security* configuration file. That is, `TLSv1.
10791079
* There is no default value for this setting.
10801080

10811081
You can set the path to your own template here, if you so desire.
1082+
When a custom template is used, the limitation is that ILM setups will be ignored. It is recommended to add ILM settings in the template, manually manage rollover and policy.
1083+
10821084
If not set, the included template will be used.
10831085

10841086
[id="plugins-{type}s-{plugin}-template_api"]
@@ -1093,9 +1095,8 @@ Elasticsearch cluster is running Elasticsearch version `8.0.0` or higher,
10931095
and use {ref}/indices-templates-v1.html[legacy template API] otherwise.
10941096

10951097
Setting this flag to `legacy` will use legacy template API to create index template.
1096-
When this option is used with Elasticsearch 8.x or higher versions, the plugin requires custom template. Make sure to provide custom template by setting `template` path, otherwise installing default template will be failed.
1097-
10981098
Setting this flag to `composable` will use index template API to create index template.
1099+
When this option, except `auto`, is used with any Elasticsearch versions, the plugin requires a custom template. Make sure to provide a custom template by setting `template` path, otherwise installing default template will be failed.
10991100

11001101
NOTE: The format of template provided to <<plugins-{type}s-{plugin}-template>> needs to match the template API being used.
11011102

Diff for: lib/logstash/outputs/elasticsearch.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -347,7 +347,7 @@ def finish_register
347347
assert_es_version_supports_data_streams if data_stream_config?
348348
discover_cluster_uuid
349349
install_template
350-
setup_ilm if ilm_in_use?
350+
setup_ilm if !@template && ilm_in_use?
351351
super
352352
end
353353

Diff for: lib/logstash/outputs/elasticsearch/template_manager.rb

+11-4
Original file line numberDiff line numberDiff line change
@@ -15,16 +15,23 @@ def self.install_template(plugin)
1515
"The legacy template API is slated for removal in Elasticsearch 9.")
1616
end
1717

18-
# we need to guide customers using their own (custom) template when they want to use legacy template API with Elasticsearch 8.x (or up) versions,
19-
# otherwise `load_default_template` method loads the template based on ES 8.x versions which fits the `_index_template` index mapping but not `_template`.
20-
if !plugin.template && plugin.template_api == 'legacy' && plugin.maximum_seen_major_version >= 8
21-
raise LogStash::ConfigurationError, "Plugin cannot resolve the default template when using legacy template API with Elasticsearch 8.x (or higher) versions. Please consider using a `template_api => composable` option or set the custom template by providing `template` path."
18+
# we need to guide customers using their own (custom) template when they want to use legacy template API (except auto) with Elasticsearch (any versions).
19+
# 1. `template_api` => `legacy` with ES 8.x, `load_default_template` method loads the template based on ES 8.x versions which fits the `_index_template` index mapping but not `_template`.
20+
# 2. `template_api` => `legacy` with ES 7.x, surprisingly this works as `load_default_template` method loads suitable default template for `_template`. But there is a big risk to crash all pipelines when customer upgrades to ES 8.x.
21+
# 3. `template_api` => `composable` with ES 7.x, `load_default_template` method loads non-suitable default template for `_template`.
22+
if plugin.template_api != 'auto' && !plugin.template
23+
raise LogStash::ConfigurationError, "Custom template has to be set When using `template_api` => #{plugin.template_api}. Please consider using a `template_api => auto` or set the custom template by providing `template` path."
2224
end
2325

2426
if plugin.template
2527
plugin.logger.info("Using mapping template from", :path => plugin.template)
2628
# we let customers to place ILM setting in their custom templates
2729
template = read_template_file(plugin.template)
30+
31+
# we need to encourage users to manually set ILM in their templates, ILM settings in the config will be ignored.
32+
if plugin.ilm_in_use?
33+
plugin.logger.warn("ILM is enabled but it will be ignored because custom template is provided. Please make sure to set the ILM settings in the custom template.")
34+
end
2835
else
2936
plugin.logger.info("Using a default mapping template", :es_version => plugin.maximum_seen_major_version,
3037
:ecs_compatibility => plugin.ecs_compatibility)

Diff for: logstash-output-elasticsearch.gemspec

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
Gem::Specification.new do |s|
22
s.name = 'logstash-output-elasticsearch'
3-
s.version = '11.12.2'
3+
s.version = '11.12.3'
44
s.licenses = ['apache-2.0']
55
s.summary = "Stores logs in Elasticsearch"
66
s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"

Diff for: spec/unit/outputs/elasticsearch/template_manager_spec.rb

+77-6
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
require "logstash/devutils/rspec/spec_helper"
1+
require_relative '../../../../spec/spec_helper'
22
require "logstash/outputs/elasticsearch/template_manager"
33

44
describe LogStash::Outputs::ElasticSearch::TemplateManager do
@@ -96,11 +96,6 @@
9696
endpoint = described_class.template_endpoint(plugin)
9797
expect(endpoint).to be_equal(LogStash::Outputs::ElasticSearch::TemplateManager::LEGACY_TEMPLATE_ENDPOINT)
9898
end
99-
100-
it "should raise an error when installing the default template" do
101-
expect(plugin).to receive(:maximum_seen_major_version).at_least(:once).and_return(8)
102-
expect{ described_class.install_template(plugin) }.to raise_error(LogStash::ConfigurationError, a_string_including("Plugin cannot resolve the default template when using legacy template API with Elasticsearch 8.x (or higher) versions. Please consider using a `template_api => composable` option or set the custom template by providing `template` path."))
103-
end
10499
end
105100
end
106101

@@ -117,4 +112,80 @@
117112
end
118113
end
119114
end
115+
116+
describe "installing default template" do
117+
118+
describe "when 'template_api' => 'auto'" do
119+
# by default the `template_api` config is `auto`
120+
let(:plugin_settings) { {"manage_template" => true } }
121+
let(:plugin) { LogStash::Outputs::ElasticSearch.new(plugin_settings) }
122+
123+
it "should not raise an error without custom template with ES 7.x" do
124+
expect(plugin).to receive(:client).at_least(:once).and_return(Object.new)
125+
expect(plugin.client).to receive(:template_install).at_least(:once).and_return({})
126+
expect(plugin).to receive(:maximum_seen_major_version).at_least(:once).and_return(7)
127+
expect{ described_class.install_template(plugin) }.not_to raise_error
128+
end
129+
130+
it "should not raise an error with custom template with ES 7.x" do
131+
file_path = described_class.default_template_path(7)
132+
plugin = LogStash::Outputs::ElasticSearch.new(plugin_settings.merge({"template" => file_path}))
133+
expect(plugin).to receive(:client).at_least(:once).and_return(Object.new)
134+
expect(plugin.client).to receive(:template_install).at_least(:once).and_return({})
135+
expect(plugin).to receive(:maximum_seen_major_version).at_least(:once).and_return(7)
136+
expect{ described_class.install_template(plugin) }.not_to raise_error
137+
end
138+
139+
it "should not raise an error without custom template with ES 8.x" do
140+
expect(plugin).to receive(:client).at_least(:once).and_return(Object.new)
141+
expect(plugin.client).to receive(:template_install).at_least(:once).and_return({})
142+
expect(plugin).to receive(:maximum_seen_major_version).at_least(:once).and_return(8)
143+
expect{ described_class.install_template(plugin) }.not_to raise_error
144+
end
145+
146+
it "should not raise an error with custom template with ES 8.x" do
147+
file_path = described_class.default_template_path(8)
148+
plugin = LogStash::Outputs::ElasticSearch.new(plugin_settings.merge({"template" => file_path}))
149+
expect(plugin).to receive(:client).at_least(:once).and_return(Object.new)
150+
expect(plugin.client).to receive(:template_install).at_least(:once).and_return({})
151+
expect(plugin).to receive(:maximum_seen_major_version).at_least(:once).and_return(8)
152+
expect{ described_class.install_template(plugin) }.not_to raise_error
153+
end
154+
end
155+
156+
# 'template_api' => 'composable' behaves same
157+
describe "when 'template_api' => 'legacy'" do
158+
159+
let(:plugin_settings) { {"manage_template" => true, "template_api" => 'legacy'} }
160+
let(:plugin) { LogStash::Outputs::ElasticSearch.new(plugin_settings) }
161+
162+
it "should raise an error without custom template with ES 7.x" do
163+
expect(plugin).to receive(:maximum_seen_major_version).at_least(:once).and_return(7)
164+
expect{ described_class.install_template(plugin) }.to raise_error(LogStash::ConfigurationError, a_string_including("Custom template has to be set When using `template_api` => legacy. Please consider using a `template_api => auto` or set the custom template by providing `template` path."))
165+
end
166+
167+
it "should raise an error without custom template with ES 8.x" do
168+
expect(plugin).to receive(:maximum_seen_major_version).at_least(:once).and_return(8)
169+
expect{ described_class.install_template(plugin) }.to raise_error(LogStash::ConfigurationError, a_string_including("Custom template has to be set When using `template_api` => legacy. Please consider using a `template_api => auto` or set the custom template by providing `template` path."))
170+
end
171+
172+
it "should not raise an error with custom template with ES 7.x" do
173+
file_path = described_class.default_template_path(7)
174+
plugin = LogStash::Outputs::ElasticSearch.new(plugin_settings.merge({"template" => file_path}))
175+
expect(plugin).to receive(:maximum_seen_major_version).at_least(:once).and_return(7)
176+
expect(plugin).to receive(:client).at_least(:once).and_return(Object.new)
177+
expect(plugin.client).to receive(:template_install).at_least(:once).and_return({})
178+
expect{ described_class.install_template(plugin) }.not_to raise_error
179+
end
180+
181+
it "should not raise an error with custom template with ES 8.x" do
182+
file_path = described_class.default_template_path(8)
183+
plugin = LogStash::Outputs::ElasticSearch.new(plugin_settings.merge({"template" => file_path}))
184+
expect(plugin).to receive(:maximum_seen_major_version).at_least(:once).and_return(8)
185+
expect(plugin).to receive(:client).at_least(:once).and_return(Object.new)
186+
expect(plugin.client).to receive(:template_install).at_least(:once).and_return({})
187+
expect{ described_class.install_template(plugin) }.not_to raise_error
188+
end
189+
end
190+
end
120191
end

0 commit comments

Comments
 (0)