Mark previously deprecated SSL settings as obsolete (#1197)

* Mark previously deprecated SSL settings as obsolete

 - SSL settings that were marked deprecated in version `11.14.0` are now marked
   obsolete, and will prevent the plugin from starting.
  - These settings are:
    - `cacert`, which should be replaced by `ssl_certificate_authorities`
    - `keystore`, which should be replaced by `ssl_keystore_path`
    - `keystore_password`, which should be replaced by `ssl_keystore_password`
    - `ssl`, which should be replaced by `ssl_enabled`
    - `ssl_certificate_verification`, which should be replaced by
      `ssl_verification_mode`
    - `truststore`, which should be replaced by `ssl_truststore_path`
    - `truststore_password`, which should be replaced by
      `ssl_truststore_password`

* Restore deleted footer

Restore the deleted EOF as it is required for docs.

* Remove reference to deprecated options

* Remove characters copied from a diff view

* Fix reference to old removed ssl setting

Replace ssl with ssl_enabled.

* Remove whitespace

* Restore deleted pipe character

* Add a note detailing removal

Co-authored-by: Karen Metts <[email protected]>

---------

Co-authored-by: Karen Metts <[email protected]>

Author: donoghuc

Diff for: CHANGELOG.md

@@ -1,3 +1,15 @@
+## 12.0.0
+  - SSL settings that were marked deprecated in version `11.14.0` are now marked obsolete, and will prevent the plugin from starting.
+  - These settings are:
+    - `cacert`, which should be replaced by `ssl_certificate_authorities`
+    - `keystore`, which should be replaced by `ssl_keystore_path`
+    - `keystore_password`, which should be replaced by `ssl_keystore_password`
+    - `ssl`, which should be replaced by `ssl_enabled`
+    - `ssl_certificate_verification`, which should be replaced by `ssl_verification_mode`
+    - `truststore`, which should be replaced by `ssl_truststore_path`
+    - `truststore_password`, which should be replaced by `ssl_truststore_password`
+    - [#1197](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1197)
+
 ## 11.22.10
  - Add `x-elastic-product-origin` header to Elasticsearch requests [#1195](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1195)
 

Diff for: docs/index.asciidoc

@@ -325,8 +325,10 @@ When a string value on an event contains one or more byte sequences that are not
 [id="plugins-{type}s-{plugin}-options"]
 ==== Elasticsearch Output Configuration Options
 
-This plugin supports the following configuration options plus the
-<<plugins-{type}s-{plugin}-common-options>> and the <<plugins-{type}s-{plugin}-deprecated-options>> described later.
+This plugin supports these configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+
+NOTE: As of version 12.0.0 of this plugin, a number of previously deprecated SSL settings have been removed.
+Please check out <<plugins-{type}s-{plugin}-obsolete-options>> for details.
 
 [cols="<,<,<",options="header",]
 |=======================================================================
@@ -441,7 +443,7 @@ For more details on actions, check out the {ref}/docs-bulk.html[Elasticsearch bu
   * There is no default value for this setting.
 
 Authenticate using Elasticsearch API key.
-Note that this option also requires SSL/TLS, which can be enabled by supplying a <<plugins-{type}s-{plugin}-cloud_id>>, a list of HTTPS <<plugins-{type}s-{plugin}-hosts>>, or by setting <<plugins-{type}s-{plugin}-ssl,`ssl_enabled => true`>>.
+Note that this option also requires SSL/TLS, which can be enabled by supplying a <<plugins-{type}s-{plugin}-cloud_id>>, a list of HTTPS <<plugins-{type}s-{plugin}-hosts>>, or by setting <<plugins-{type}s-{plugin}-ssl_enabled,`ssl_enabled => true`>>.
 
 Format is `id:api_key` where `id` and `api_key` are as returned by the
 Elasticsearch {ref}/security-api-create-api-key.html[Create API key API].
@@ -1324,98 +1326,24 @@ https://www.elastic.co/blog/elasticsearch-versioning-support[versioning support
 blog] and {ref}/docs-index_.html#_version_types[Version types] in the
 Elasticsearch documentation.
 
-[id="plugins-{type}s-{plugin}-deprecated-options"]
-==== Elasticsearch Output Deprecated Configuration Options
-
-This plugin supports the following deprecated configurations.
+[id="plugins-{type}s-{plugin}-obsolete-options"]
+==== Elasticsearch Output Obsolete Configuration Options
 
-WARNING: Deprecated options are subject to removal in future releases.
+WARNING: As of version `12.0.0` of this plugin, some configuration options have been replaced.
+The plugin will fail to start if it contains any of these obsolete options.
 
-[cols="<,<,<",options="header",]
+[cols="<,<",options="header",]
 |=======================================================================
-|Setting|Input type|Replaced by
-| <<plugins-{type}s-{plugin}-cacert>> |a valid filesystem path|<<plugins-{type}s-{plugin}-ssl_certificate_authorities>>
-| <<plugins-{type}s-{plugin}-keystore>> |a valid filesystem path|<<plugins-{type}s-{plugin}-ssl_keystore_path>>
-| <<plugins-{type}s-{plugin}-keystore_password>> |<<password,password>>|<<plugins-{type}s-{plugin}-ssl_keystore_password>>
-| <<plugins-{type}s-{plugin}-ssl>> |<<boolean,boolean>>|<<plugins-{type}s-{plugin}-ssl_enabled>>
-| <<plugins-{type}s-{plugin}-ssl_certificate_verification>> |<<boolean,boolean>>|<<plugins-{type}s-{plugin}-ssl_verification_mode>>
-| <<plugins-{type}s-{plugin}-truststore>> |a valid filesystem path|<<plugins-{type}s-{plugin}-ssl_truststore_path>>
-| <<plugins-{type}s-{plugin}-truststore_password>> |<<password,password>>|<<plugins-{type}s-{plugin}-ssl_truststore_password>>
+|Setting|Replaced by
+| cacert | <<plugins-{type}s-{plugin}-ssl_certificate_authorities>>
+| keystore | <<plugins-{type}s-{plugin}-ssl_keystore_path>>
+| keystore_password | <<plugins-{type}s-{plugin}-ssl_keystore_password>>
+| ssl | <<plugins-{type}s-{plugin}-ssl_enabled>>
+| ssl_certificate_verification | <<plugins-{type}s-{plugin}-ssl_verification_mode>>
+| truststore | <<plugins-{type}s-{plugin}-ssl_truststore_path>>
+| truststore_password | <<plugins-{type}s-{plugin}-ssl_truststore_password>>
 |=======================================================================
 
-
-[id="plugins-{type}s-{plugin}-cacert"]
-===== `cacert`
-deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_certificate_authorities>>]
-
-* Value type is a list of <<path,path>>
-* There is no default value for this setting.
-
-The .cer or .pem file to validate the server's certificate.
-
-[id="plugins-{type}s-{plugin}-keystore"]
-===== `keystore`
-deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_keystore_path>>]
-
-* Value type is <<path,path>>
-* There is no default value for this setting.
-
-The keystore used to present a certificate to the server.
-It can be either .jks or .p12
-
-NOTE: You cannot use this setting and <<plugins-{type}s-{plugin}-ssl_certificate>> at the same time.
-
-[id="plugins-{type}s-{plugin}-keystore_password"]
-===== `keystore_password`
-deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_keystore_password>>]
-
-* Value type is <<password,password>>
-* There is no default value for this setting.
-
-Set the keystore password
-
-[id="plugins-{type}s-{plugin}-ssl"]
-===== `ssl`
-deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_enabled>>]
-
-* Value type is <<boolean,boolean>>
-* There is no default value for this setting.
-
-Enable SSL/TLS secured communication to Elasticsearch cluster.
-Leaving this unspecified will use whatever scheme is specified in the URLs listed in <<plugins-{type}s-{plugin}-hosts>> or extracted from the <<plugins-{type}s-{plugin}-cloud_id>>.
-If no explicit protocol is specified plain HTTP will be used.
-
-[id="plugins-{type}s-{plugin}-ssl_certificate_verification"]
-===== `ssl_certificate_verification`
-deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_verification_mode>>]
-
-* Value type is <<boolean,boolean>>
-* Default value is `true`
-
-Option to validate the server's certificate. Disabling this severely compromises security.
-For more information on disabling certificate verification please read
-https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
-
-[id="plugins-{type}s-{plugin}-truststore"]
-===== `truststore`
-deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_truststore_path>>]
-
-* Value type is <<path,path>>
-* There is no default value for this setting.
-
-The truststore to validate the server's certificate.
-It can be either `.jks` or `.p12`.
-Use either `:truststore` or `:cacert`.
-
-[id="plugins-{type}s-{plugin}-truststore_password"]
-===== `truststore_password`
-deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_truststore_password>>]
-
-* Value type is <<password,password>>
-* There is no default value for this setting.
-
-Set the truststore password
-
 [id="plugins-{type}s-{plugin}-common-options"]
 include::{include_path}/{type}.asciidoc[]
 

Diff for: lib/logstash/outputs/elasticsearch.rb

@@ -275,7 +275,6 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
   def initialize(*params)
     super
     setup_ecs_compatibility_related_defaults
-    setup_ssl_params!
     setup_compression_level!
   end
 
@@ -694,52 +693,6 @@ def setup_template_manager_defaults(data_stream_enabled)
     end
   end
 
-  def setup_ssl_params!
-    @ssl_enabled = normalize_config(:ssl_enabled) do |normalize|
-      normalize.with_deprecated_alias(:ssl)
-    end
-
-    @ssl_certificate_authorities = normalize_config(:ssl_certificate_authorities) do |normalize|
-      normalize.with_deprecated_mapping(:cacert) do |cacert|
-        [cacert]
-      end
-    end
-
-    @ssl_keystore_path =  normalize_config(:ssl_keystore_path) do |normalize|
-      normalize.with_deprecated_alias(:keystore)
-    end
-
-    @ssl_keystore_password = normalize_config(:ssl_keystore_password) do |normalize|
-      normalize.with_deprecated_alias(:keystore_password)
-    end
-
-    @ssl_truststore_path = normalize_config(:ssl_truststore_path) do |normalize|
-      normalize.with_deprecated_alias(:truststore)
-    end
-
-    @ssl_truststore_password =  normalize_config(:ssl_truststore_password) do |normalize|
-      normalize.with_deprecated_alias(:truststore_password)
-    end
-
-    @ssl_verification_mode = normalize_config(:ssl_verification_mode) do |normalize|
-      normalize.with_deprecated_mapping(:ssl_certificate_verification) do |ssl_certificate_verification|
-        if ssl_certificate_verification == true
-          "full"
-        else
-          "none"
-        end
-      end
-    end
-
-    params['ssl_enabled'] = @ssl_enabled unless @ssl_enabled.nil?
-    params['ssl_certificate_authorities'] = @ssl_certificate_authorities unless @ssl_certificate_authorities.nil?
-    params['ssl_keystore_path'] = @ssl_keystore_path unless @ssl_keystore_path.nil?
-    params['ssl_keystore_password'] = @ssl_keystore_password unless @ssl_keystore_password.nil?
-    params['ssl_truststore_path'] = @ssl_truststore_path unless @ssl_truststore_path.nil?
-    params['ssl_truststore_password'] = @ssl_truststore_password unless @ssl_truststore_password.nil?
-    params['ssl_verification_mode'] = @ssl_verification_mode unless @ssl_verification_mode.nil?
-  end
-
   def setup_compression_level!
     @compression_level = normalize_config(:compression_level) do |normalize|
       normalize.with_deprecated_mapping(:http_compression) do |http_compression|

Diff for: lib/logstash/plugin_mixins/elasticsearch/api_configs.rb

@@ -43,67 +43,40 @@ module APIConfigs
         # urls that already have query strings, the one specified here will be appended.
         :parameters => { :validate => :hash },
 
-        # Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever scheme
-        # is specified in the URLs listed in 'hosts'. If no explicit protocol is specified plain HTTP will be used.
-        # If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in 'hosts'
-        :ssl => { :validate => :boolean, :deprecated => "Set 'ssl_enabled' instead." },
-
         # Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever scheme
         # is specified in the URLs listed in 'hosts'. If no explicit protocol is specified plain HTTP will be used.
         # If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in 'hosts'
         :ssl_enabled => { :validate => :boolean },
 
-        # Option to validate the server's certificate. Disabling this severely compromises security.
-        # For more information on disabling certificate verification please read
-        # https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf
-        :ssl_certificate_verification => { :validate => :boolean, :default => true, :deprecated => "Set 'ssl_verification_mode' instead." },
-
         # Options to verify the server's certificate.
         # "full": validates that the provided certificate has an issue date that’s within the not_before and not_after dates;
         # chains to a trusted Certificate Authority (CA); has a hostname or IP address that matches the names within the certificate.
         # "none": performs no certificate validation. Disabling this severely compromises security (https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf)
         :ssl_verification_mode => { :validate => %w[full none], :default => 'full' },
 
-        # The .cer or .pem file to validate the server's certificate
-        :cacert => { :validate => :path, :deprecated => "Set 'ssl_certificate_authorities' instead." },
-
         # The .cer or .pem files to validate the server's certificate
         :ssl_certificate_authorities => { :validate => :path, :list => true },
 
         # One or more hex-encoded SHA256 fingerprints to trust as Certificate Authorities
         :ca_trusted_fingerprint => LogStash::PluginMixins::CATrustedFingerprintSupport,
 
-        # The JKS truststore to validate the server's certificate.
-        # Use either `:truststore` or `:cacert`
-        :truststore => { :validate => :path, :deprecated => "Set 'ssl_truststore_path' instead." },
-
         # The JKS truststore to validate the server's certificate.
         # Use either `:ssl_truststore_path` or `:ssl_certificate_authorities`
         :ssl_truststore_path => { :validate => :path },
 
         # The format of the truststore file. It must be either jks or pkcs12
         :ssl_truststore_type => { :validate => %w[pkcs12 jks] },
 
-        # Set the truststore password
-        :truststore_password => { :validate => :password, :deprecated => "Use 'ssl_truststore_password' instead." },
-
         # Set the truststore password
         :ssl_truststore_password => { :validate => :password },
 
-        # The keystore used to present a certificate to the server.
-        # It can be either .jks or .p12
-        :keystore => { :validate => :path, :deprecated => "Set 'ssl_keystore_path' instead." },
-
         # The keystore used to present a certificate to the server.
         # It can be either .jks or .p12
         :ssl_keystore_path => { :validate => :path },
 
         # The format of the keystore file. It must be either jks or pkcs12
         :ssl_keystore_type => { :validate => %w[pkcs12 jks] },
 
-        # Set the keystore password
-        :keystore_password => { :validate => :password, :deprecated => "Set 'ssl_keystore_password' instead." },
-
         # Set the keystore password
         :ssl_keystore_password => { :validate => :password },
 
@@ -229,7 +202,17 @@ module APIConfigs
         :dlq_custom_codes => { :validate => :number, :list => true, :default => [] },
 
         # if enabled, failed index name interpolation events go into dead letter queue.
-        :dlq_on_failed_indexname_interpolation => { :validate => :boolean, :default => true }
+        :dlq_on_failed_indexname_interpolation => { :validate => :boolean, :default => true },
+
+        # Obsolete Settings
+        :ssl => { :obsolete => "Set 'ssl_enabled' instead." },
+        :ssl_certificate_verification => { :obsolete => "Set 'ssl_verification_mode' instead." },
+        :cacert => { :obsolete => "Set 'ssl_certificate_authorities' instead." },
+        :truststore => { :obsolete => "Set 'ssl_truststore_path' instead." },
+        :keystore => { :obsolete => "Set 'ssl_keystore_path' instead." },
+        # Leave :validate to ensure obfuscation of sensitive setting for passwords
+        :truststore_password => { :validate => :password, :obsolete => "Use 'ssl_truststore_password' instead." },
+        :keystore_password => { :validate => :password, :obsolete => "Set 'ssl_keystore_password' instead." }
     }.freeze
 
     def self.included(base)
@@ -243,3 +226,4 @@ def self.included(base)
     end
   end
 end; end; end
+

Diff for: logstash-output-elasticsearch.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name            = 'logstash-output-elasticsearch'
-  s.version         = '11.22.10'
+  s.version         = '12.0.0'
   s.licenses        = ['apache-2.0']
   s.summary         = "Stores logs in Elasticsearch"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"

Diff for: spec/unit/outputs/elasticsearch_spec.rb

@@ -1125,81 +1125,6 @@
     end
   end
 
-  describe "SSL deprecated settings" do
-    let(:base_options) { {"ssl" => "true"} }
-
-    context "with client certificate" do
-      let(:do_register) { true }
-      let(:cacert) { Stud::Temporary.file.path }
-      let(:options) { base_options.merge(
-        "cacert" => cacert,
-        "ssl_certificate_verification" => false
-      ) }
-
-      after :each do
-        File.delete(cacert)
-      end
-
-      it "should map new configs into params" do
-        expect(subject.params).to match hash_including(
-                                          "ssl_enabled" => true,
-                                          "ssl_verification_mode" => "none",
-                                          "ssl_certificate_authorities" => [cacert]
-                                        )
-      end
-
-      it "should set new configs variables" do
-        expect(subject.instance_variable_get(:@ssl_enabled)).to eql(true)
-        expect(subject.instance_variable_get(:@ssl_verification_mode)).to eql("none")
-        expect(subject.instance_variable_get(:@ssl_certificate_authorities)).to eql([cacert])
-      end
-    end
-
-    context "with java stores" do
-      let(:do_register) { true }
-      let(:keystore) { Stud::Temporary.file.path }
-      let(:truststore) { Stud::Temporary.file.path }
-      let(:options) { base_options.merge(
-        "keystore" => keystore,
-        "keystore_password" => "keystore",
-        "truststore" => truststore,
-        "truststore_password" => "truststore",
-        "ssl_certificate_verification" => true
-      ) }
-
-      let(:spy_http_client_builder!) do
-        allow(described_class::HttpClientBuilder).to receive(:build).with(any_args).and_call_original
-        allow(described_class::HttpClientBuilder).to receive(:setup_ssl).with(any_args).and_return({})
-      end
-
-      after :each do
-        File.delete(keystore)
-        File.delete(truststore)
-      end
-
-      it "should map new configs into params" do
-        expect(subject.params).to match hash_including(
-                                          "ssl_enabled" => true,
-                                          "ssl_keystore_path" => keystore,
-                                          "ssl_truststore_path" => truststore,
-                                          "ssl_verification_mode" => "full"
-                                        )
-
-        expect(subject.params["ssl_keystore_password"].value).to eql("keystore")
-        expect(subject.params["ssl_truststore_password"].value).to eql("truststore")
-      end
-
-      it "should set new configs variables" do
-        expect(subject.instance_variable_get(:@ssl_enabled)).to eql(true)
-        expect(subject.instance_variable_get(:@ssl_keystore_path)).to eql(keystore)
-        expect(subject.instance_variable_get(:@ssl_keystore_password).value).to eql("keystore")
-        expect(subject.instance_variable_get(:@ssl_truststore_path)).to eql(truststore)
-        expect(subject.instance_variable_get(:@ssl_truststore_password).value).to eql("truststore")
-        expect(subject.instance_variable_get(:@ssl_verification_mode)).to eql("full")
-      end
-    end
-  end
-
   describe "retry_on_conflict" do
     let(:num_retries) { 123 }
     let(:event) { LogStash::Event.new("myactionfield" => "update", "message" => "blah") }