Skip to content

[DOCS] Add data streams to bulk, delete, and index API docs #58340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jun 23, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 17 additions & 9 deletions docs/reference/docs/bulk.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ POST _bulk

`POST /_bulk`

`POST /<index>/_bulk`
`POST /<target>/_bulk`

[[docs-bulk-api-desc]]
==== {api-description-title}
Expand All @@ -47,8 +47,12 @@ optional_source\n

The `index` and `create` actions expect a source on the next line,
and have the same semantics as the `op_type` parameter in the standard index API:
create fails if a document with the same name already exists in the index,
index adds or replaces a document as necessary.
`create` fails if a document with the same ID already exists in the target,
`index` adds or replaces a document as necessary.

NOTE: <<data-streams,Data streams>> support only the `create` action. To update
or delete a document in a data stream, you must target the backing index
containing the document. See <<update-delete-docs-in-a-data-stream>>.

`update` expects that the partial doc, upsert,
and script and its options are specified on the next line.
Expand All @@ -67,8 +71,8 @@ the `Content-Type` header should be set to `application/x-ndjson`.
Because this format uses literal `\n`'s as delimiters,
make sure that the JSON actions and sources are not pretty printed.

If you specify an index in the request URI,
it is used for any actions that don't explicitly specify an index.
If you provide a `<target>` in the request path,
it is used for any actions that don't explicitly specify an `_index` argument.

A note on the format: The idea here is to make processing of this as
fast as possible. As some of the actions are redirected to other
Expand Down Expand Up @@ -192,8 +196,10 @@ See <<url-access-control>>.
[[docs-bulk-api-path-params]]
==== {api-path-parms-title}

`<index>`::
(Optional, string) Name of the index to perform the bulk actions against.
`<target>`::
(Optional, string)
Name of the data stream, index, or index alias to perform bulk actions
on.

[[docs-bulk-api-query-params]]
==== {api-query-parms-title}
Expand Down Expand Up @@ -321,7 +327,8 @@ operation.
=====
`_index`::
(string)
The index name or alias associated with the operation.
Name of the index associated with the operation. If the operation targeted a
data stream, this is the associated backing index.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: I wonder if it would be more clear to say something along the lines of "the backing index into which the document was written" instead of "associated backing index" since a data stream may have many associated backing indices.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I used this wording here and a modified version for failed operations.


`_id`::
(integer)
Expand Down Expand Up @@ -410,7 +417,8 @@ ID of the shard associated with the failed operation.

`index`::
(string)
The index name or alias associated with the failed operation.
Name of the index associated with the failed operation. If the operation
targeted a data stream, this is the associated backing index.
======
=====
====
Expand Down
14 changes: 8 additions & 6 deletions docs/reference/docs/delete.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,19 @@
<titleabbrev>Delete</titleabbrev>
++++

Removes a JSON document from the specified index.
Removes a JSON document from a specified data stream or index.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We disallow deletes directly on a data stream because it would require searching every backing index for the specified ID. If a document is to be deleted from a data stream, the relevant backing index must be targeted directly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:doh: Thanks for catching this. I think I got overzealous.

Fixed up here: https://github.com/elastic/elasticsearch/pull/58340/files#diff-907ef82eb2dfb22151e4312359f5aee4


[[docs-delete-api-request]]
==== {api-request-title}

`DELETE /<index>/_doc/<_id>`
`DELETE /<target>/_doc/<_id>`

[[docs-delete-api-desc]]
==== {api-description-title}

You use DELETE to remove a document from an index. You must specify the
index name and document ID.
You can use the delete API to remove a document from a data stream or index. You
must specify the document ID and the name of the data stream or index containing
the document.

[float]
[[optimistic-concurrency-control-delete]]
Expand Down Expand Up @@ -132,8 +133,9 @@ DELETE /twitter/_doc/1?timeout=5m
[[docs-delete-api-path-params]]
==== {api-path-parms-title}

`<index>`::
(Required, string) Name of the target index.
`<target>`::
(Required, string) Name of data stream or index containing the document to
delete.

`<_id>`::
(Required, string) Unique identifier for the document.
Expand Down
56 changes: 41 additions & 15 deletions docs/reference/docs/index_.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,27 +6,40 @@

IMPORTANT: See <<removal-of-types>>.

Adds a JSON document to the specified index and makes
it searchable. If the document already exists,
updates the document and increments its version.
Adds a JSON document to the specified data stream or index and makes
it searchable. If the target is an index and the document already exists,
the request updates the document and increments its version.

NOTE: You cannot send update requests for existing documents to a data stream.
To update a document in a data stream, you must target the backing index
containing the document. See <<update-delete-docs-in-a-data-stream>>.

[[docs-index-api-request]]
==== {api-request-title}

`PUT /<index>/_doc/<_id>`
`PUT /<target>/_doc/<_id>`

`POST /<index>/_doc/`
`POST /<target>/_doc/`

`PUT /<index>/_create/<_id>`
`PUT /<target>/_create/<_id>`

`POST /<index>/_create/<_id>`
`POST /<target>/_create/<_id>`

[[docs-index-api-path-params]]
==== {api-path-parms-title}

`<index>`::
(Required, string) Name of the target index. By default, the index is created
automatically if it doesn't exist. For more information, see <<index-creation>>.
`<target>`::
(Required, string) Name of the data stream or index to target.
+
If the target doesn't exist and matches the name or wildcard (`*`) pattern of a
<<create-a-data-stream-template,composable template with a `data_stream`
definition>>, this request creates the data stream. See
<<set-up-a-data-stream>>.
+
If the target doesn't exist and doesn't match a data stream template,
this request creates the index.
+
You can check for existing targets using the resolve index API.

`<_id>`::
(Optional, string) Unique identifier for the document. Required if you are
Expand All @@ -48,6 +61,9 @@ if it does not already exist (_put if absent_). If a document with the specified
`_id` already exists, the indexing operation will fail. Same as using the
`<index>/_create` endpoint. Valid values: `index`, `create`.
If document id is specified, it defaults to `index`. Otherwise, it defaults to `create`.
+
NOTE: If the request targets a data stream, an `op_type` of `create` is
required. See <<add-documents-to-a-data-stream>>.

include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=pipeline]

Expand Down Expand Up @@ -125,12 +141,19 @@ You can index a new JSON document with the `_doc` or `_create` resource. Using
exist. To update an existing document, you must use the `_doc` resource.

[[index-creation]]
===== Create indices automatically
===== Automatically create data streams and indices

If request's target doesn't exist and matches a
<<create-a-data-stream-template,composable template with a `data_stream`
definition>>, the index operation automatically creates the data stream. See
<<set-up-a-data-stream>>.

If the specified index does not already exist, by default the index operation
automatically creates it and applies any configured
<<indices-templates,index templates>>. If no mapping exists, the index operation
creates a dynamic mapping. By default, new fields and objects are
If the target doesn't exist and doesn't match a data stream template,
the operation automatically creates the index and applies any matching
<<indices-templates,index templates>>.

If no mapping exists, the index operation
creates a dynamic mapping. By default, new fields and objects are
automatically added to the mapping if needed. For more information about field
mapping, see <<mapping,mapping>> and the <<indices-put-mapping,put mapping>> API.

Expand All @@ -142,6 +165,9 @@ automatic creation of indices that match specified patterns, or set it to
comma-separated list of patterns you want to allow, or prefix each pattern with
`+` or `-` to indicate whether it should be allowed or blocked. When a list is
specified, the default behaviour is to disallow.
+
IMPORTANT: The `action.auto_create_index` setting only affects the automatic
creation of indices. It does not affect the creation of data streams.

[source,console]
--------------------------------------------------
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/rest-api/common-parms.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -430,8 +430,8 @@ end::index-total[]
tag::bulk-index[]
`_index`::
(Optional, string)
The name of the target index.
Required if not specified as a path parameter.
Name of the data stream, index, or index alias to perform the action on. This
parameter is required if a `<target>` is not specified in the request path.
end::bulk-index[]

tag::index-metric[]
Expand Down