From a61bf1d044c82a8bd891d804d6bb349567d52952 Mon Sep 17 00:00:00 2001 From: Jake Landis Date: Tue, 30 Oct 2018 10:45:48 -0500 Subject: [PATCH 1/3] ingest: add common options to each processor's documentation * adds `if`, `on_failure`, and `tag` to table for each processor part of #33188 --- .../ingest-node-common-processor.asciidoc | 4 ++ docs/reference/ingest/ingest-node.asciidoc | 37 ++++++++++++++++++- 2 files changed, 39 insertions(+), 2 deletions(-) create mode 100644 docs/reference/ingest/ingest-node-common-processor.asciidoc diff --git a/docs/reference/ingest/ingest-node-common-processor.asciidoc b/docs/reference/ingest/ingest-node-common-processor.asciidoc new file mode 100644 index 0000000000000..a2b12136bb119 --- /dev/null +++ b/docs/reference/ingest/ingest-node-common-processor.asciidoc @@ -0,0 +1,4 @@ +| `if` | no | - | Conditionally execute this processor. +| `on_failure` | no | - | Handle failures of this processor. See <>. +| `tag` | no | - | An identifier for this processor. Useful for debugging and metrics. +// TODO: See <>. <-- for the if description once PR 35044 is merged \ No newline at end of file diff --git a/docs/reference/ingest/ingest-node.asciidoc b/docs/reference/ingest/ingest-node.asciidoc index eeb914facc2c6..2d56000787845 100644 --- a/docs/reference/ingest/ingest-node.asciidoc +++ b/docs/reference/ingest/ingest-node.asciidoc @@ -778,6 +778,7 @@ Accepts a single value or an array of values. | Name | Required | Default | Description | `field` | yes | - | The field to be appended to. Supports <>. | `value` | yes | - | The value to be appended. Supports <>. +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -791,6 +792,8 @@ Accepts a single value or an array of values. -------------------------------------------------- // NOTCONSOLE + + [[bytes-processor]] === Bytes Processor Converts a human readable byte value (e.g. 1kb) to its value in bytes (e.g. 1024). @@ -806,6 +809,7 @@ the field is not a supported format or resultant value exceeds 2^63. | `field` | yes | - | The field to convert | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -844,6 +848,7 @@ still be updated with the unconverted field value. | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `type` | yes | - | The type to convert the existing value to | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -876,6 +881,7 @@ in the same order they were defined as part of the processor definition. | `formats` | yes | - | An array of the expected date formats. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. | `timezone` | no | UTC | The timezone to use when parsing the date. Supports <>. | `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days. Supports <>. +include::ingest-node-common-processor.asciidoc[] |====== Here is an example that adds the parsed date to the `timestamp` field based on the `initial_date` field: @@ -1065,6 +1071,7 @@ understands this to mean `2016-04-01` as is explained in the <>. +include::ingest-node-common-processor.asciidoc[] |====== [[dissect-processor]] @@ -1134,7 +1141,7 @@ See <> for more information. | `pattern` | yes | - | The pattern to apply to the field | `append_separator`| no | "" (empty string) | The character(s) that separate the appended fields. | `ignore_missing` | no | false | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document -| ` +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -1162,7 +1169,6 @@ modifiers. | `+` with `/n` | Append with order | left and right | `%{+keyname/2} %{+keyname/1}` | Appends two or more fields together in the order specified | <> | `?` | Named skip key | left | `%{?ignoreme}` | Skips the matched value in the output. Same behavior as `%{}`| <> | `*` and `&` | Reference keys | left | `%{*r1} %{&r1}` | Sets the output key as value of `*` and output value of `&` | <> -| ` |====== [[dissect-modifier-skip-right-padding]] @@ -1265,6 +1271,14 @@ Reference key modifier example Drops the document without raising any errors. This is useful to prevent the document from getting indexed based on some condition. +[[drop-options]] +.Drop Options +[options="header"] +|====== +| Name | Required | Default | Description +include::ingest-node-common-processor.asciidoc[] +|====== + [source,js] -------------------------------------------------- { @@ -1289,6 +1303,7 @@ Otherwise these <> can't be accessed by any | Name | Required | Default | Description | `field` | yes | - | The field to expand into an object field | `path` | no | - | The field that contains the field to expand. Only required if the field to expand is part another object field, because the `field` option can only understand leaf fields. +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -1406,6 +1421,7 @@ to the requester. |====== | Name | Required | Default | Description | `message` | yes | - | The error message thrown by the processor. Supports <>. +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -1452,6 +1468,7 @@ then it aborts the execution and leaves the array unmodified. | `field` | yes | - | The array field | `processor` | yes | - | The processor to execute against each field | `ignore_missing` | no | false | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== Assume the following document: @@ -1645,6 +1662,7 @@ Grok expression. | `pattern_definitions` | no | - | A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition. | `trace_match` | no | false | when true, `_ingest._grok_match_index` will be inserted into your matched document's metadata with the index into the pattern found in `patterns` that matched. | `ignore_missing` | no | false | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== Here is an example of using the provided patterns to extract out and name structured fields from a string field in @@ -1919,6 +1937,7 @@ If the field is not a string, the processor will throw an exception. | `replacement` | yes | - | The string to replace the matching patterns with | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -1946,6 +1965,7 @@ Throws an error when the field is not an array. | `field` | yes | - | The field to be separated | `separator` | yes | - | The separator character | `target_field` | no | `field` | The field to assign the joined value to, by default `field` is updated in-place +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -1971,6 +1991,7 @@ Converts a JSON string into a structured JSON object. | `field` | yes | - | The field to be parsed | `target_field` | no | `field` | The field to insert the converted structured object into | `add_to_root` | no | false | Flag that forces the serialized json to be injected into the top level of the document. `target_field` must not be set when this option is chosen. +include::ingest-node-common-processor.asciidoc[] |====== All JSON-supported types will be parsed (null, boolean, number, array, object, string). @@ -2082,6 +2103,7 @@ For example, if you have a log message which contains `ip=1.2.3.4 error=REFUSED` | `trim_key` | no | `null` | String of characters to trim from extracted keys | `trim_value` | no | `null` | String of characters to trim from extracted values | `strip_brackets` | no | `false` | If `true` strip brackets `()`, `<>`, `[]` as well as quotes `'` and `"` from extracted values +include::ingest-node-common-processor.asciidoc[] |====== @@ -2097,6 +2119,7 @@ Converts a string to its lowercase equivalent. | `field` | yes | - | The field to make lowercase | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2119,6 +2142,7 @@ Executes another pipeline. |====== | Name | Required | Default | Description | `name` | yes | - | The name of the pipeline to execute +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2234,6 +2258,7 @@ Removes existing fields. If one field doesn't exist, an exception will be thrown | Name | Required | Default | Description | `field` | yes | - | Fields to be removed. Supports <>. | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== Here is an example to remove a single field: @@ -2272,6 +2297,7 @@ Renames an existing field. If the field doesn't exist or the new name is already | `field` | yes | - | The field to be renamed. Supports <>. | `target_field` | yes | - | The new name of the field. Supports <>. | `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2305,6 +2331,7 @@ caching see <>. | `id` | no | - | The stored script id to refer to | `source` | no | - | An inline script to be executed | `params` | no | - | Script Parameters +include::ingest-node-common-processor.asciidoc[] |====== One of `id` or `source` options must be provided in order to properly reference a script to execute. @@ -2401,6 +2428,7 @@ its value will be replaced with the provided one. | `field` | yes | - | The field to insert, upsert, or update. Supports <>. | `value` | yes | - | The value to be set for the field. Supports <>. | `override` | no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched. +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2427,6 +2455,7 @@ Splits a field into an array using a separator character. Only works on string f | `separator` | yes | - | A regex which matches the separator, eg `,` or `\s+` | `target_field` | no | `field` | The field to assign the split value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2455,6 +2484,7 @@ Throws an error when the field is not an array. | `field` | yes | - | The field to be sorted | `order` | no | `"asc"` | The sort order to use. Accepts `"asc"` or `"desc"`. | `target_field` | no | `field` | The field to assign the sorted value to, by default `field` is updated in-place +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2482,6 +2512,7 @@ NOTE: This only works on leading and trailing whitespace. | `field` | yes | - | The string-valued field to trim whitespace from | `target_field` | no | `field` | The field to assign the trimmed value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2506,6 +2537,7 @@ Converts a string to its uppercase equivalent. | `field` | yes | - | The field to make uppercase | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2530,6 +2562,7 @@ URL-decodes a string | `field` | yes | - | The field to decode | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] From 03e16d79558f49156e33fbd88c5b4ff4423fcfd8 Mon Sep 17 00:00:00 2001 From: Jake Landis Date: Wed, 31 Oct 2018 10:02:18 -0500 Subject: [PATCH 2/3] added ingore_failure --- .../reference/ingest/ingest-node-common-processor.asciidoc | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/reference/ingest/ingest-node-common-processor.asciidoc b/docs/reference/ingest/ingest-node-common-processor.asciidoc index a2b12136bb119..dcf8b63630b4b 100644 --- a/docs/reference/ingest/ingest-node-common-processor.asciidoc +++ b/docs/reference/ingest/ingest-node-common-processor.asciidoc @@ -1,4 +1,5 @@ -| `if` | no | - | Conditionally execute this processor. -| `on_failure` | no | - | Handle failures of this processor. See <>. -| `tag` | no | - | An identifier for this processor. Useful for debugging and metrics. +| `if` | no | - | Conditionally execute this processor. +| `on_failure` | no | - | Handle failures for this processor. See <>. +| `ignore_failure` | no | `false` | Ignore failures for this processor. See <>. +| `tag` | no | - | An identifier for this processor. Useful for debugging and metrics. // TODO: See <>. <-- for the if description once PR 35044 is merged \ No newline at end of file From 3e7395b018d59ffcb68e872d11c56cedf92256f4 Mon Sep 17 00:00:00 2001 From: Jake Landis Date: Wed, 31 Oct 2018 13:41:23 -0500 Subject: [PATCH 3/3] fix whitespace noise --- docs/reference/ingest/ingest-node.asciidoc | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/reference/ingest/ingest-node.asciidoc b/docs/reference/ingest/ingest-node.asciidoc index 2d56000787845..ebcddd5afa55d 100644 --- a/docs/reference/ingest/ingest-node.asciidoc +++ b/docs/reference/ingest/ingest-node.asciidoc @@ -792,8 +792,6 @@ include::ingest-node-common-processor.asciidoc[] -------------------------------------------------- // NOTCONSOLE - - [[bytes-processor]] === Bytes Processor Converts a human readable byte value (e.g. 1kb) to its value in bytes (e.g. 1024).