From 93a465b0dc7bc94dd2ac45d092c4ae1994271e16 Mon Sep 17 00:00:00 2001 From: Andrea Selva Date: Mon, 5 Oct 2020 14:20:55 +0200 Subject: [PATCH 1/2] [Doc] added the ILM pattern note ILM pattern number part is 0-padded 6 digits wide string --- docs/index.asciidoc | 110 ++++++++++++++++++++++---------------------- 1 file changed, 56 insertions(+), 54 deletions(-) diff --git a/docs/index.asciidoc b/docs/index.asciidoc index fc79526a3..d84b3edd7 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -23,14 +23,14 @@ include::{include_path}/plugin_header.asciidoc[] If you plan to use the Kibana web interface to analyze data transformed by Logstash, use the Elasticsearch output plugin to get your data into -Elasticsearch. +Elasticsearch. This output only speaks the HTTP protocol as it is the preferred protocol for interacting with Elasticsearch. In previous versions it was possible to communicate with Elasticsearch through the transport protocol, which is now reserved for internal cluster communication between nodes https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-transport.html[communication between nodes]. -Using the https://www.elastic.co/guide/en/elasticsearch/reference/current/java-clients.html[transport protocol] +Using the https://www.elastic.co/guide/en/elasticsearch/reference/current/java-clients.html[transport protocol] to communicate with the cluster has been deprecated in Elasticsearch 7.0.0 and will be removed in 8.0.0 @@ -76,7 +76,7 @@ and will correctly reject events with fields that conflict and cannot be coerced [NOTE] ================================================================================ -You cannot use dynamic variable substitution when `ilm_enabled` is `true` and +You cannot use dynamic variable substitution when `ilm_enabled` is `true` and when using `ilm_rollover_alias`. ================================================================================ @@ -100,7 +100,7 @@ Example: index => "%{[some_field][sub_field]}-%{+YYYY.MM.dd}" } } - + **What to do in case there is no field in the event containing the destination index prefix?** You can use the `mutate` filter and conditionals to add a `[@metadata]` field (see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#metadata) to set @@ -136,7 +136,7 @@ HTTP requests to the bulk API are expected to return a 200 response code. All ot The following document errors are handled as follows: * 400 and 404 errors are sent to the dead letter queue (DLQ), if enabled. If a DLQ is not enabled, a log message will be emitted, and the event will be dropped. See <> for more info. - * 409 errors (conflict) are logged as a warning and dropped. + * 409 errors (conflict) are logged as a warning and dropped. Note that 409 exceptions are no longer retried. Please set a higher `retry_on_conflict` value if you experience 409 exceptions. It is more performant for Elasticsearch to retry these exceptions than this plugin. @@ -219,7 +219,7 @@ not reevaluate its DNS value while the keepalive is in effect. ==== HTTP Compression This plugin supports request and response compression. Response compression is -enabled by default for HTTP and for Elasticsearch versions 5.0 and later. +enabled by default for HTTP and for Elasticsearch versions 5.0 and later. You don't have to set any configs in Elasticsearch for it to send back a compressed response. For versions before 5.0, or if HTTPS is enabled, @@ -307,7 +307,7 @@ output plugins.   [id="plugins-{type}s-{plugin}-action"] -===== `action` +===== `action` * Value type is <> * Default value is `"index"` @@ -338,7 +338,7 @@ Authenticate using Elasticsearch API key. Note that this option also requires en Format is `id:api_key` where `id` and `api_key` are as returned by the Elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html[Create API key API]. [id="plugins-{type}s-{plugin}-bulk_path"] -===== `bulk_path` +===== `bulk_path` * Value type is <> * There is no default value for this setting. @@ -347,7 +347,7 @@ HTTP Path to perform the _bulk requests to this defaults to a concatenation of the path parameter and "_bulk" [id="plugins-{type}s-{plugin}-cacert"] -===== `cacert` +===== `cacert` * Value type is <> * There is no default value for this setting. @@ -375,7 +375,7 @@ Cloud ID, from the Elastic Cloud web console. If set `hosts` should not be used. For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html#_cloud_id[Logstash-to-Cloud documentation] [id="plugins-{type}s-{plugin}-doc_as_upsert"] -===== `doc_as_upsert` +===== `doc_as_upsert` * Value type is <> * Default value is `false` @@ -384,7 +384,7 @@ Enable `doc_as_upsert` for update mode. Create a new document with source if `document_id` doesn't exist in Elasticsearch [id="plugins-{type}s-{plugin}-document_id"] -===== `document_id` +===== `document_id` * Value type is <> * There is no default value for this setting. @@ -392,7 +392,7 @@ Create a new document with source if `document_id` doesn't exist in Elasticsearc The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID. [id="plugins-{type}s-{plugin}-document_type"] -===== `document_type` +===== `document_type` * Value type is <> * There is no default value for this setting. @@ -432,7 +432,7 @@ The value of this setting affects the _default_ values of: * <> [id="plugins-{type}s-{plugin}-failure_type_logging_whitelist"] -===== `failure_type_logging_whitelist` +===== `failure_type_logging_whitelist` * Value type is <> * Default value is `[]` @@ -453,7 +453,7 @@ an elasticsearch node. The headers will be used for any kind of request These custom headers will be overidden by settings like `http_compression`. [id="plugins-{type}s-{plugin}-healthcheck_path"] -===== `healthcheck_path` +===== `healthcheck_path` * Value type is <> * There is no default value for this setting. @@ -464,7 +464,7 @@ before it is once again eligible to service requests. If you have custom firewall rules you may need to change this [id="plugins-{type}s-{plugin}-hosts"] -===== `hosts` +===== `hosts` * Value type is <> * Default value is `[//127.0.0.1]` @@ -486,7 +486,7 @@ to prevent LS from sending bulk requests to the master nodes. So this parameter Any special characters present in the URLs here MUST be URL escaped! This means `#` should be put in as `%23` for instance. [id="plugins-{type}s-{plugin}-http_compression"] -===== `http_compression` +===== `http_compression` * Value type is <> * Default value is `false` @@ -523,6 +523,8 @@ NOTE: Updating the pattern will require the index template to be rewritten NOTE: The pattern must finish with a dash and a number that will be automatically incremented when indices rollover. +NOTE: The pattern number is a string zero-padded 6 digits wide, see {ref}/indices-rollover-index.html#rollover-index-api-path-params[Rollover path parameters API docs] for details + [id="plugins-{type}s-{plugin}-ilm_policy"] ===== `ilm_policy` @@ -551,7 +553,7 @@ NOTE: Updating the rollover alias will require the index template to be rewritte NOTE: `ilm_rollover_alias` does NOT support dynamic variable substitution as `index` does. [id="plugins-{type}s-{plugin}-index"] -===== `index` +===== `index` * Value type is <> * Default value depends on whether <> is enabled: @@ -567,7 +569,7 @@ LS uses Joda to format the index pattern from event timestamp. Joda formats are defined http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html[here]. [id="plugins-{type}s-{plugin}-keystore"] -===== `keystore` +===== `keystore` * Value type is <> * There is no default value for this setting. @@ -576,7 +578,7 @@ The keystore used to present a certificate to the server. It can be either .jks or .p12 [id="plugins-{type}s-{plugin}-keystore_password"] -===== `keystore_password` +===== `keystore_password` * Value type is <> * There is no default value for this setting. @@ -584,7 +586,7 @@ It can be either .jks or .p12 Set the keystore password [id="plugins-{type}s-{plugin}-manage_template"] -===== `manage_template` +===== `manage_template` * Value type is <> * Default value is `true` @@ -604,7 +606,7 @@ field names) you should set `manage_template` to false and use the REST API to apply your templates manually. [id="plugins-{type}s-{plugin}-parameters"] -===== `parameters` +===== `parameters` * Value type is <> * There is no default value for this setting. @@ -614,7 +616,7 @@ to every host listed in the 'hosts' configuration. If the 'hosts' list contains urls that already have query strings, the one specified here will be appended. [id="plugins-{type}s-{plugin}-parent"] -===== `parent` +===== `parent` * Value type is <> * Default value is `nil` @@ -623,7 +625,7 @@ For child documents, ID of the associated parent. This can be dynamic using the `%{foo}` syntax. [id="plugins-{type}s-{plugin}-password"] -===== `password` +===== `password` * Value type is <> * There is no default value for this setting. @@ -631,7 +633,7 @@ This can be dynamic using the `%{foo}` syntax. Password to authenticate to a secure Elasticsearch cluster [id="plugins-{type}s-{plugin}-path"] -===== `path` +===== `path` * Value type is <> * There is no default value for this setting. @@ -642,7 +644,7 @@ Note that if you use paths as components of URLs in the 'hosts' field you may not also set this field. That will raise an error at startup [id="plugins-{type}s-{plugin}-pipeline"] -===== `pipeline` +===== `pipeline` * Value type is <> * Default value is `nil` @@ -651,7 +653,7 @@ Set which ingest pipeline you wish to execute for an event. You can also use eve like `pipeline => "%{[@metadata][pipeline]}"`. The pipeline parameter won't be set if the value resolves to empty string (""). [id="plugins-{type}s-{plugin}-pool_max"] -===== `pool_max` +===== `pool_max` * Value type is <> * Default value is `1000` @@ -662,7 +664,7 @@ Setting this too low may mean frequently closing / opening connections which is bad. [id="plugins-{type}s-{plugin}-pool_max_per_route"] -===== `pool_max_per_route` +===== `pool_max_per_route` * Value type is <> * Default value is `100` @@ -673,7 +675,7 @@ Setting this too low may mean frequently closing / opening connections which is bad. [id="plugins-{type}s-{plugin}-proxy"] -===== `proxy` +===== `proxy` * Value type is <> * There is no default value for this setting. @@ -684,7 +686,7 @@ An empty string is treated as if proxy was not set. This is useful when using environment variables e.g. `proxy => '${LS_PROXY:}'`. [id="plugins-{type}s-{plugin}-resurrect_delay"] -===== `resurrect_delay` +===== `resurrect_delay` * Value type is <> * Default value is `5` @@ -694,7 +696,7 @@ Resurrection is the process by which backend endpoints marked 'down' are checked to see if they have come back to life [id="plugins-{type}s-{plugin}-retry_initial_interval"] -===== `retry_initial_interval` +===== `retry_initial_interval` * Value type is <> * Default value is `2` @@ -702,7 +704,7 @@ to see if they have come back to life Set initial interval in seconds between bulk retries. Doubled on each retry up to `retry_max_interval` [id="plugins-{type}s-{plugin}-retry_max_interval"] -===== `retry_max_interval` +===== `retry_max_interval` * Value type is <> * Default value is `64` @@ -710,7 +712,7 @@ Set initial interval in seconds between bulk retries. Doubled on each retry up t Set max interval in seconds between bulk retries. [id="plugins-{type}s-{plugin}-retry_on_conflict"] -===== `retry_on_conflict` +===== `retry_on_conflict` * Value type is <> * Default value is `1` @@ -720,7 +722,7 @@ See the https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-upda for more info [id="plugins-{type}s-{plugin}-routing"] -===== `routing` +===== `routing` * Value type is <> * There is no default value for this setting. @@ -729,7 +731,7 @@ A routing override to be applied to all processed events. This can be dynamic using the `%{foo}` syntax. [id="plugins-{type}s-{plugin}-script"] -===== `script` +===== `script` * Value type is <> * Default value is `""` @@ -745,7 +747,7 @@ Example: } [id="plugins-{type}s-{plugin}-script_lang"] -===== `script_lang` +===== `script_lang` * Value type is <> * Default value is `"painless"` @@ -754,7 +756,7 @@ Set the language of the used script. If not set, this defaults to painless in ES When using indexed (stored) scripts on Elasticsearch 6 and higher, you must set this parameter to `""` (empty string). [id="plugins-{type}s-{plugin}-script_type"] -===== `script_type` +===== `script_type` * Value can be any of: `inline`, `indexed`, `file` * Default value is `["inline"]` @@ -765,7 +767,7 @@ Define the type of script referenced by "script" variable file : "script" contains the name of script stored in elasticsearch's config directory [id="plugins-{type}s-{plugin}-script_var_name"] -===== `script_var_name` +===== `script_var_name` * Value type is <> * Default value is `"event"` @@ -773,7 +775,7 @@ Define the type of script referenced by "script" variable Set variable name passed to script (scripted update) [id="plugins-{type}s-{plugin}-scripted_upsert"] -===== `scripted_upsert` +===== `scripted_upsert` * Value type is <> * Default value is `false` @@ -781,7 +783,7 @@ Set variable name passed to script (scripted update) if enabled, script is in charge of creating non-existent document (scripted update) [id="plugins-{type}s-{plugin}-sniffing"] -===== `sniffing` +===== `sniffing` * Value type is <> * Default value is `false` @@ -791,7 +793,7 @@ For Elasticsearch 1.x and 2.x any nodes with `http.enabled` (on by default) will For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will be added to the hosts list, excluding master-only nodes. [id="plugins-{type}s-{plugin}-sniffing_delay"] -===== `sniffing_delay` +===== `sniffing_delay` * Value type is <> * Default value is `5` @@ -799,7 +801,7 @@ For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will How long to wait, in seconds, between sniffing attempts [id="plugins-{type}s-{plugin}-sniffing_path"] -===== `sniffing_path` +===== `sniffing_path` * Value type is <> * There is no default value for this setting. @@ -810,7 +812,7 @@ if sniffing_path is set it will be used as an absolute path do not use full URL here, only paths, e.g. "/sniff/_nodes/http" [id="plugins-{type}s-{plugin}-ssl"] -===== `ssl` +===== `ssl` * Value type is <> * There is no default value for this setting. @@ -820,7 +822,7 @@ is specified in the URLs listed in 'hosts'. If no explicit protocol is specified If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in 'hosts' [id="plugins-{type}s-{plugin}-ssl_certificate_verification"] -===== `ssl_certificate_verification` +===== `ssl_certificate_verification` * Value type is <> * Default value is `true` @@ -830,7 +832,7 @@ For more information on disabling certificate verification please read https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf [id="plugins-{type}s-{plugin}-template"] -===== `template` +===== `template` * Value type is <> * There is no default value for this setting. @@ -839,7 +841,7 @@ You can set the path to your own template here, if you so desire. If not set, the included template will be used. [id="plugins-{type}s-{plugin}-template_name"] -===== `template_name` +===== `template_name` * Value type is <> * Default value depends on whether <> is enabled: @@ -856,7 +858,7 @@ change this, you will need to prune the old template manually, e.g. where `OldTemplateName` is whatever the former setting was. [id="plugins-{type}s-{plugin}-template_overwrite"] -===== `template_overwrite` +===== `template_overwrite` * Value type is <> * Default value is `false` @@ -873,7 +875,7 @@ template (logstash), setting this to true will make Logstash to overwrite the "logstash" template (i.e. removing all customized settings) [id="plugins-{type}s-{plugin}-timeout"] -===== `timeout` +===== `timeout` * Value type is <> * Default value is `60` @@ -882,7 +884,7 @@ Set the timeout, in seconds, for network operations and requests sent Elasticsea a timeout occurs, the request will be retried. [id="plugins-{type}s-{plugin}-truststore"] -===== `truststore` +===== `truststore` * Value type is <> * There is no default value for this setting. @@ -892,7 +894,7 @@ It can be either .jks or .p12. Use either `:truststore` or `:cacert`. [id="plugins-{type}s-{plugin}-truststore_password"] -===== `truststore_password` +===== `truststore_password` * Value type is <> * There is no default value for this setting. @@ -900,7 +902,7 @@ Use either `:truststore` or `:cacert`. Set the truststore password [id="plugins-{type}s-{plugin}-upsert"] -===== `upsert` +===== `upsert` * Value type is <> * Default value is `""` @@ -909,7 +911,7 @@ Set upsert content for update mode. Create a new document with this parameter as json string if `document_id` doesn't exists [id="plugins-{type}s-{plugin}-user"] -===== `user` +===== `user` * Value type is <> * There is no default value for this setting. @@ -917,7 +919,7 @@ Create a new document with this parameter as json string if `document_id` doesn' Username to authenticate to a secure Elasticsearch cluster [id="plugins-{type}s-{plugin}-validate_after_inactivity"] -===== `validate_after_inactivity` +===== `validate_after_inactivity` * Value type is <> * Default value is `10000` @@ -932,7 +934,7 @@ have become stale (half-closed) while kept inactive in the pool.' See https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.html#setValidateAfterInactivity(int)[these docs for more info] [id="plugins-{type}s-{plugin}-version"] -===== `version` +===== `version` * Value type is <> * There is no default value for this setting. @@ -941,7 +943,7 @@ The version to use for indexing. Use sprintf syntax like `%{my_version}` to use See https://www.elastic.co/blog/elasticsearch-versioning-support. [id="plugins-{type}s-{plugin}-version_type"] -===== `version_type` +===== `version_type` * Value can be any of: `internal`, `external`, `external_gt`, `external_gte`, `force` * There is no default value for this setting. From 91f358b0f12842177f9c08eb18a8e58c94b0c462 Mon Sep 17 00:00:00 2001 From: Andrea Selva Date: Tue, 6 Oct 2020 10:36:12 +0200 Subject: [PATCH 2/2] [doc] reworded description for ILM pattern format Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> --- docs/index.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/index.asciidoc b/docs/index.asciidoc index d84b3edd7..bb160b465 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -523,7 +523,8 @@ NOTE: Updating the pattern will require the index template to be rewritten NOTE: The pattern must finish with a dash and a number that will be automatically incremented when indices rollover. -NOTE: The pattern number is a string zero-padded 6 digits wide, see {ref}/indices-rollover-index.html#rollover-index-api-path-params[Rollover path parameters API docs] for details +NOTE: The pattern is a 6-digit string padded by zeros, regardless of prior index name. Example: 000001. +See {ref}/indices-rollover-index.html#rollover-index-api-path-params[Rollover path parameters API docs] for details. [id="plugins-{type}s-{plugin}-ilm_policy"] ===== `ilm_policy`