Skip to content

[DOCS] Defines data frame transform stats API objects #44197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Feb 5, 2020
Merged

[DOCS] Defines data frame transform stats API objects #44197

Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
befca79
[DOCS] Defines data frame transform stats API objects
lcawl Jul 11, 2019
c95538d
[DOCS] Drafts response for get transform stats API
lcawl Jan 29, 2020
ee6097d
[DOCS] Removes unnecessary resource page
lcawl Jan 29, 2020
c07e37c
[DOCS] More draft info
lcawl Jan 31, 2020
92335b4
[DOCS] Addresses feedback
lcawl Feb 3, 2020
5e28e9d
[DOCS] Adds missing data types
lcawl Feb 4, 2020
29e27d9
[DOCS] Clarifies timestamp properties
lcawl Feb 4, 2020
fdd53d7
[DOCS] More timestamp clarifications
lcawl Feb 4, 2020
cb5a77b
[DOCS] Addresses more feedback
lcawl Feb 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 41 additions & 8 deletions docs/reference/transform/apis/get-transform-stats.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,11 +83,34 @@ informational; you cannot update their values.
`checkpointing`.`last`:::
(object) Contains statistics about the last completed checkpoint.
`checkpointing`.`last`.`checkpoint`::::
(TBD) A unique identifier for the checkpoint.
(integer) The sequence number for the checkpoint.
`checkpointing`.`last`.`time_upper_bound_millis`::::
(date) TBD
(date) When using time-based synchronization, this timestamp indicates the
upper bound of data that is included in the checkpoint. Typically, this value
is equal to the `checkpointing`.`last`.`time_upper_bound_millis` minus the
`sync`.`time`.`delay`, which is defined when you create the {transform}.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not include "Typically, this value is ..." This was just for your information, not meant to be put here.

`checkpointing`.`last`.`timestamp_millis`::::
(date) TBD
(date) The timestamp of the checkpoint, which indicates when the checkpoint
was created.
`checkpointing`.`next`:::
(object) Contains statistics about the next checkpoint that is currently in
progress. This object appears only if the {transform} is currently processing
data and only for the first checkpoint.
Copy link

@hendrikmuhs hendrikmuhs Feb 5, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry this is not quite correct yet: "and only for the first checkpoint" is only true for the checkpoint_progress nested object below.

checkpointing.next will always be there if the transform is actively doing something (when the state is indexing).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarifications! I've pushed another commit

`checkpointing`.`next`.`checkpoint`::::
(integer) The sequence number for the checkpoint.
`checkpointing`.`next`.`checkpoint_progress`::::
(object) Contains statistics about the progress of the checkpoint. For example,
it lists the `total_docs`, `docs_remaining`, `percent_complete`,
`docs_processed`, and `docs_indexed`. This information is available only for
batch {transforms} and the first checkpoint of {ctransforms}.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

`checkpointing`.`next`.`time_upper_bound_millis`::::
(date) When using time-based synchronization, this timestamp indicates the
upper bound of data that is included in the checkpoint. Typically, this value
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would omit "Typically..."

is equal to the `checkpointing`.`next`.`time_upper_bound_millis` minus the
`sync`.`time`.`delay`, which is defined when you create the {transform}.
`checkpointing`.`next`.`timestamp_millis`::::
(date) The timestamp of the checkpoint, which indicates when the checkpoint was
created.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checkpointing.next`:::
optional (object) Contains statistics about the next - currently in progress - checkpoint. This object only appears if the transform is currently processing data and only for the 1st checkpoint

It uses the same fields at last but has one more object:

checkpoint_progress::
(object) Contains statistics about the progress of the checkpoint.

Not sure how much we want to go into the details, the inner fields are:

  • total_docs
  • docs_remaining
  • percent_complete
  • docs_processed
  • docs_indexed

This information is only available for batch transforms and for the 1st checkpoint of a continuous transform.

`id`::
(string)
Expand All @@ -111,18 +134,24 @@ example, `127.0.0.1:9300`.
(string) The status of the {transform}, which can be one of the following values:
+
--
* `aborting`: The {transform} is aborting.
* `failed`: The {transform} failed. For more information about the failure,
check the reason field.
* `indexing`: The {transform} is actively processing data and creating new
documents.
* `started`: The {transform} is running but not actively indexing data.
* `stopped`: The {transform} is stopped.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • aborting The {transform} is aborting.
  • stopping The {transform} is stopping.
  • failed The {transform} has failed. Check the reason field for further information.

* `stopping`: The {transform} is stopping.
--
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • reason::
    (string) The reason of the failure if the transform is in failed state.


`stats`::
(object) An object that provides statistical information about the {transform}.
`stats`.`documents_indexed`:::
(TBD) The number of new documents that have been indexed.
(long) The number of documents that have been indexed into the destination index
for the {transform}.
`stats`.`documents_processed`:::
(TBD) The number of documents that have been processed.
(long) The number of documents that have been processed from the source index of
the {transform}.
`stats`.`exponential_avg_checkpoint_duration_ms`:::
(double) Exponential moving average of the duration of the checkpoint, in milliseconds.
`stats`.`exponential_avg_documents_indexed`:::
Expand All @@ -138,15 +167,19 @@ processed.
`stats`.`index_total`:::
(long) The number of indices created.
`stats`.`pages_processed`:::
(TBD) The number of pages processed.
(long) The number of search or bulk index operations processed. Documents are
processed in batches instead of individually.
`stats`.`search_failures`:::
(long) The number of search failures.
`stats`.`search_time_in_ms`:::
(long) The amount of time spent searching, in milliseconds.
`stats`.`search_total`:::
(long) TBD
(long) The number of search operations on the source index for the {transform}.
`stats`.`trigger_count`:::
(TBD) TBD
(long) The number of times the {transform} has been triggered by the scheduler.
For example, the scheduler triggers the {transform} indexer to check for updates
or ingest new data at an interval specified in the
<<put-transform-request-body,`frequency` property>>.

[[get-transform-stats-response-codes]]
==== {api-response-codes-title}
Expand Down