[DOCS] Defines data frame transform stats API objects #44197

lcawl · 2019-07-11T02:23:12Z

This PR drafts definitions for results from the "get data frame transform statistics" API (https://www.elastic.co/guide/en/elasticsearch/reference/master/get-transform-stats.html), equivalent to what we have for anomaly detector job statistics:
https://www.elastic.co/guide/en/elasticsearch/reference/master/ml-get-job-stats.html

Preview: http://elasticsearch_44197.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/get-transform-stats.html

elasticmachine · 2019-07-11T02:23:13Z

Pinging @elastic/es-docs

elasticmachine · 2019-07-11T02:23:14Z

Pinging @elastic/ml-core

droberts195 · 2019-07-11T14:04:04Z

Due to #43767 the format of stats will change between 7.2 and 7.3. Therefore it might be best just to document the new format post #43767 and not to backport this PR to the 7.2 branch as current labels suggest.

hendrikmuhs · 2020-01-28T08:02:39Z

@lcawl Can this PR be closed? Looks outdated to me.

lcawl · 2020-01-28T22:44:54Z

@elasticmachine update branch

elasticmachine · 2020-01-28T22:44:56Z

merge conflict between base and head

hendrikmuhs

Thanks for looking into this again.

I commented the TBD parts and added some explanations.

hendrikmuhs · 2020-02-03T08:15:58Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+`checkpointing`.`last`:::
+(object) Contains statistics about the last completed checkpoint. 
+`checkpointing`.`last`.`checkpoint`::::
+(TBD) A unique identifier for the checkpoint.


I suggest: "sequence number for the checkpoint")

hendrikmuhs · 2020-02-03T08:59:49Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+`checkpointing`.`last`.`checkpoint`::::
+(TBD) A unique identifier for the checkpoint.
+`checkpointing`.`last`.`time_upper_bound_millis`::::
+(date) TBD


optional, timestamp until data has been processed when using time-based synchronization

... timestamp until data has been processed...

Thanks for the feedback @hendrikmuhs ! I'm not sure I understand this description yet, however. Is it the duration of the checkpoint?

Think of a continuous transform, where you source indexing gets new data in, so the destination/transformed index runs always behind the source. time_upper_bound marks the timestamp until all data from source has been processed into dest. So it's not a duration its an endmarker until data has been processed.

There is also timestamp and it might seem like the same thing, but timestamp is the time the checkpoint has been created, time_upper_bound has to take the delay into account. So normally time_upper_bound = timestamp - delay. (However in future this might change, that's why timestamp and time_upper_bound are separate fields)

Thanks, I've drafted changes to those two descriptions. If they still need tweaking, please just let me know.

hendrikmuhs · 2020-02-03T09:00:19Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+`checkpointing`.`last`.`time_upper_bound_millis`::::
+(date) TBD
+`checkpointing`.`last`.`timestamp_millis`::::
+(date) TBD


timestamp of the checkpoint (when the checkpoint has been created)

hendrikmuhs · 2020-02-03T09:13:45Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+(date) TBD
+`checkpointing`.`last`.`timestamp_millis`::::
+(date) TBD
+


checkpointing.next`:::
optional (object) Contains statistics about the next - currently in progress - checkpoint. This object only appears if the transform is currently processing data and only for the 1st checkpoint

It uses the same fields at last but has one more object:

checkpoint_progress::
(object) Contains statistics about the progress of the checkpoint.

Not sure how much we want to go into the details, the inner fields are:

total_docs

docs_remaining

percent_complete

docs_processed

docs_indexed

This information is only available for batch transforms and for the 1st checkpoint of a continuous transform.

hendrikmuhs · 2020-02-03T09:20:13Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+* `indexing`: The {transform} is actively processing data and creating new
+documents.
+* `started`: The {transform} is running but not actively indexing data.
+* `stopped`: The {transform} is stopped.


aborting The {transform} is aborting.

stopping The {transform} is stopping.

failed The {transform} has failed. Check the reason field for further information.

hendrikmuhs · 2020-02-03T09:23:23Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+`stats`::
+(object) An object that provides statistical information about the {transform}.
+`stats`.`documents_indexed`:::
+(TBD) The number of new documents that have been indexed.


The number of documents that have been indexed into the transform dest index.

hendrikmuhs · 2020-02-03T09:23:49Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+`stats`.`documents_indexed`:::
+(TBD) The number of new documents that have been indexed.
+`stats`.`documents_processed`:::
+(TBD) The number of documents that have been processed.


The number of documents that have been processed from the transform source index.

hendrikmuhs · 2020-02-03T09:29:34Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+`stats`.`index_total`:::
+(long) The number of indices created.
+`stats`.`pages_processed`:::
+(TBD) The number of pages processed.


(long) The number of pages (number of search/bulk index operations) processed.

(I do not know if this need better explanation: In a nutshell documents are not processed one by one but always on batches. This happens both for search - a search page - as well as for indexing. There a "page" describes 1 bulk index operation that consists of a list of documents to be indexed.)

hendrikmuhs · 2020-02-03T09:30:46Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+`stats`.`search_time_in_ms`:::
+(long) The amount of time spent searching, in milliseconds.
+`stats`.`search_total`:::
+(long) TBD 


The number of search operations on the transform source index.

hendrikmuhs · 2020-02-03T09:35:25Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+`stats`.`search_total`:::
+(long) TBD 
+`stats`.`trigger_count`:::
+(TBD) TBD


The number of times the transform has been triggered by the scheduler.

(The scheduler triggers the transform indexer to e.g. check for updates / ingest new data, this can be controlled via the frequency parameter in the config: https://www.elastic.co/guide/en/elasticsearch/reference/master/put-transform.html#put-transform-request-body)

hendrikmuhs

Thanks! Looks great, I still suggest to remove some technical detail, those were only for you explaining how transform works internally.

hendrikmuhs · 2020-02-05T10:27:55Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+`checkpointing`.`next`:::
+(object) Contains statistics about the next checkpoint that is currently in
+progress. This object appears only if the {transform} is currently processing
+data and only for the first checkpoint.


sorry this is not quite correct yet: "and only for the first checkpoint" is only true for the checkpoint_progress nested object below.

checkpointing.next will always be there if the transform is actively doing something (when the state is indexing).

Thanks for the clarifications! I've pushed another commit

hendrikmuhs · 2020-02-05T10:29:17Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+(date) When using time-based synchronization, this timestamp indicates the
+upper bound of data that is included in the checkpoint. Typically, this value
+is equal to the `checkpointing`.`last`.`time_upper_bound_millis` minus the
+`sync`.`time`.`delay`, which is defined when you create the {transform}.


I would not include "Typically, this value is ..." This was just for your information, not meant to be put here.

hendrikmuhs · 2020-02-05T10:30:03Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+(object) Contains statistics about the progress of the checkpoint. For example,
+it lists the `total_docs`, `docs_remaining`, `percent_complete`,
+`docs_processed`, and `docs_indexed`. This information is available only for
+batch {transforms} and the first checkpoint of {ctransforms}. 


hendrikmuhs · 2020-02-05T10:30:55Z

docs/reference/transform/apis/get-transform-stats.asciidoc

+batch {transforms} and the first checkpoint of {ctransforms}. 
+`checkpointing`.`next`.`time_upper_bound_millis`::::
+(date) When using time-based synchronization, this timestamp indicates the
+upper bound of data that is included in the checkpoint. Typically, this value


I would omit "Typically..."

hendrikmuhs

LGTM!

lcawl added >docs General docs changes WIP v8.0.0 :ml/Transform Transform v7.2.1 v7.4.0 v7.3.1 labels Jul 11, 2019

lcawl removed the v7.2.1 label Jul 11, 2019

lcawl mentioned this pull request Jul 11, 2019

Roadmap for 7.3 data frame documentation elastic/stack-docs#406

Closed

10 tasks

jakelandis added v7.3.2 and removed v7.3.1 labels Aug 22, 2019

colings86 added v7.5.0 and removed v7.4.0 labels Aug 30, 2019

$@polyfractal$ polyfractal added v7.3.3 and removed v7.3.2 labels Sep 6, 2019

jimczi added v7.6.0 and removed v7.5.0 labels Nov 12, 2019

$@polyfractal$ polyfractal added v7.7.0 and removed v7.6.0 labels Jan 15, 2020

lcawl removed the v7.3.3 label Jan 28, 2020

lcawl added 3 commits January 28, 2020 16:13

[DOCS] Defines data frame transform stats API objects

befca79

[DOCS] Drafts response for get transform stats API

c95538d

[DOCS] Removes unnecessary resource page

ee6097d

lcawl force-pushed the df-stats branch from 7bdfb85 to ee6097d Compare January 29, 2020 01:55

[DOCS] More draft info

c07e37c

hendrikmuhs reviewed Feb 3, 2020

View reviewed changes

lcawl added 3 commits February 3, 2020 09:23

[DOCS] Addresses feedback

92335b4

[DOCS] Adds missing data types

5e28e9d

[DOCS] Clarifies timestamp properties

29e27d9

lcawl marked this pull request as ready for review February 4, 2020 17:13

lcawl removed the WIP label Feb 4, 2020

[DOCS] More timestamp clarifications

fdd53d7

hendrikmuhs reviewed Feb 5, 2020

View reviewed changes

[DOCS] Addresses more feedback

cb5a77b

hendrikmuhs approved these changes Feb 5, 2020

View reviewed changes

lcawl merged commit 14900c2 into elastic:master Feb 5, 2020

lcawl deleted the df-stats branch February 5, 2020 20:23

lcawl added a commit that referenced this pull request Feb 5, 2020

[DOCS] Defines data frame transform stats API objects (#44197)

ea1d9e0

lcawl added a commit that referenced this pull request Feb 5, 2020

[DOCS] Defines data frame transform stats API objects (#44197)

c0e5fca

lcawl added a commit that referenced this pull request Feb 5, 2020

[DOCS] Defines data frame transform stats API objects (#44197)

f226cba

lcawl added v7.5.3 v7.6.0 labels Feb 5, 2020

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOCS] Defines data frame transform stats API objects #44197

[DOCS] Defines data frame transform stats API objects #44197

lcawl commented Jul 11, 2019 •

edited

Loading

elasticmachine commented Jul 11, 2019

elasticmachine commented Jul 11, 2019

droberts195 commented Jul 11, 2019

hendrikmuhs commented Jan 28, 2020

lcawl commented Jan 28, 2020

elasticmachine commented Jan 28, 2020

hendrikmuhs left a comment

hendrikmuhs Feb 3, 2020

hendrikmuhs Feb 3, 2020

lcawl Feb 3, 2020

hendrikmuhs Feb 4, 2020 •

edited

Loading

lcawl Feb 4, 2020

hendrikmuhs Feb 3, 2020

hendrikmuhs Feb 3, 2020

hendrikmuhs Feb 3, 2020

hendrikmuhs Feb 3, 2020

hendrikmuhs Feb 3, 2020

hendrikmuhs Feb 3, 2020

hendrikmuhs Feb 3, 2020

hendrikmuhs Feb 3, 2020

hendrikmuhs left a comment

hendrikmuhs Feb 5, 2020 •

edited

Loading

lcawl Feb 5, 2020

hendrikmuhs Feb 5, 2020

hendrikmuhs Feb 5, 2020

hendrikmuhs Feb 5, 2020

hendrikmuhs left a comment

[DOCS] Defines data frame transform stats API objects #44197

[DOCS] Defines data frame transform stats API objects #44197

Conversation

lcawl commented Jul 11, 2019 • edited Loading

elasticmachine commented Jul 11, 2019

elasticmachine commented Jul 11, 2019

droberts195 commented Jul 11, 2019

hendrikmuhs commented Jan 28, 2020

lcawl commented Jan 28, 2020

elasticmachine commented Jan 28, 2020

hendrikmuhs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hendrikmuhs Feb 4, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hendrikmuhs left a comment

Choose a reason for hiding this comment

hendrikmuhs Feb 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hendrikmuhs left a comment

Choose a reason for hiding this comment

lcawl commented Jul 11, 2019 •

edited

Loading

hendrikmuhs Feb 4, 2020 •

edited

Loading

hendrikmuhs Feb 5, 2020 •

edited

Loading