From 04e40c105d9b45370a54560985f430c75df573ba Mon Sep 17 00:00:00 2001 From: Przemyslaw Witek Date: Thu, 9 Jan 2020 11:46:45 +0100 Subject: [PATCH 1/4] Make the order of evaluation fields consistent between the three types of evaluations --- .../ml/df-analytics/apis/evaluate-dfanalytics.asciidoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc b/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc index c331d9bda7aab..fa252cad4f099 100644 --- a/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc +++ b/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc @@ -137,10 +137,6 @@ belongs. (Required, string) The field of the `index` which contains the ground truth. The data type of this field must be keyword. -`metrics`:: - (Required, object) Specifies the metrics that are used for the evaluation. - Available metric is `multiclass_confusion_matrix`. - `predicted_field`:: (Required, string) The field in the `index` that contains the predicted value, in other words the results of the {classanalysis}. The data type of this field @@ -149,6 +145,10 @@ belongs. default value of the same field if you didn't specified explicitly). For example, `predicted_field` : `ml.animal_class_prediction.keyword`. +`metrics`:: + (Required, object) Specifies the metrics that are used for the evaluation. + Available metric is `multiclass_confusion_matrix`. + //// [[ml-evaluate-dfanalytics-results]] From 01a678479c4ed20b09a79b52d7989653971806a2 Mon Sep 17 00:00:00 2001 From: Przemyslaw Witek Date: Tue, 14 Jan 2020 15:29:08 +0100 Subject: [PATCH 2/4] Add evaluation metrics descriptions --- .../apis/evaluate-dfanalytics.asciidoc | 68 ++++++++++++------- 1 file changed, 45 insertions(+), 23 deletions(-) diff --git a/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc b/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc index fa252cad4f099..742ca7c5cd267 100644 --- a/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc +++ b/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc @@ -42,31 +42,32 @@ result field to be present. [[ml-evaluate-dfanalytics-request-body]] ==== {api-request-body-title} +`index`:: +(Required, object) Defines the `index` in which the evaluation will be +performed. + +`query`:: +(Optional, object) A query clause that retrieves a subset of data from the +source index. See <>. + `evaluation`:: -(Required, object) Defines the type of evaluation you want to perform. The -value of this object can be different depending on the type of evaluation you -want to perform. See <>. +(Required, object) Defines the type of evaluation you want to perform. +See <>. + -- Available evaluation types: + * `binary_soft_classification` * `regression` * `classification` --- -`index`:: -(Required, object) Defines the `index` in which the evaluation will be -performed. - -`query`:: -(Optional, object) A query clause that retrieves a subset of data from the -source index. See <>. +-- [[ml-evaluate-dfanalytics-resources]] ==== {dfanalytics-cap} evaluation resources [[binary-sc-resources]] -===== Binary soft classification configuration objects +===== Binary soft classification evaluation objects Binary soft classification evaluates the results of an analysis which outputs the probability that each document belongs to a certain class. For example, in @@ -87,20 +88,20 @@ document is an outlier. (Optional, object) Specifies the metrics that are used for the evaluation. Available metrics: - `auc_roc`:: + `auc_roc`::: (Optional, object) The AUC ROC (area under the curve of the receiver operating characteristic) score and optionally the curve. Default value is {"includes_curve": false}. - `precision`:: + `precision`::: (Optional, object) Set the different thresholds of the {olscore} at where the metric is calculated. Default value is {"at": [0.25, 0.50, 0.75]}. - `recall`:: + `recall`::: (Optional, object) Set the different thresholds of the {olscore} at where the metric is calculated. Default value is {"at": [0.25, 0.50, 0.75]}. - `confusion_matrix`:: + `confusion_matrix`::: (Optional, object) Set the different thresholds of the {olscore} at where the metrics (`tp` - true positive, `fp` - false positive, `tn` - true negative, `fn` - false negative) are calculated. Default value is @@ -122,9 +123,18 @@ which outputs a prediction of values. in other words the results of the {regression} analysis. `metrics`:: - (Required, object) Specifies the metrics that are used for the evaluation. - Available metrics are `r_squared` and `mean_squared_error`. - + (Optional, object) Specifies the metrics that are used for the evaluation. + Available metrics: + + `mean_squared_error`::: + (Optional, object) Average squared difference between the predicted values and the actual (`ground truth`) value. + Read more on https://en.wikipedia.org/wiki/Mean_squared_error[Wikipedia] + + `r_squared`::: + (Optional, object) Proportion of the variance in the dependent variable that is predictable from the independent variables. + Read more on https://en.wikipedia.org/wiki/Coefficient_of_determination[Wikipedia] + + [[classification-evaluation-resources]] ==== {classification-cap} evaluation objects @@ -134,8 +144,8 @@ outputs a prediction that identifies to which of the classes each document belongs. `actual_field`:: - (Required, string) The field of the `index` which contains the ground truth. - The data type of this field must be keyword. + (Required, string) The field of the `index` which contains the `ground truth`. + The data type of this field must be categorical. `predicted_field`:: (Required, string) The field in the `index` that contains the predicted value, @@ -146,8 +156,20 @@ belongs. example, `predicted_field` : `ml.animal_class_prediction.keyword`. `metrics`:: - (Required, object) Specifies the metrics that are used for the evaluation. - Available metric is `multiclass_confusion_matrix`. + (Optional, object) Specifies the metrics that are used for the evaluation. + Available metrics: + + `accuracy`::: + (Optional, object) Accuracy of predictions (per-class and overall) + + `precision`::: + (Optional, object) Precision of predictions (per-class and average) + + `recall`::: + (Optional, object) Recall of predictions (per-class and average) + + `multiclass_confusion_matrix`::: + (Optional, object) Multiclass confusion matrix //// From 9e605ccab713e99d0599cde137b424b3aabe257a Mon Sep 17 00:00:00 2001 From: Przemyslaw Witek Date: Wed, 15 Jan 2020 13:18:06 +0100 Subject: [PATCH 3/4] Apply review comments --- .../apis/evaluate-dfanalytics.asciidoc | 44 +++++++++---------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc b/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc index 742ca7c5cd267..97988fe39064f 100644 --- a/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc +++ b/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc @@ -42,14 +42,6 @@ result field to be present. [[ml-evaluate-dfanalytics-request-body]] ==== {api-request-body-title} -`index`:: -(Required, object) Defines the `index` in which the evaluation will be -performed. - -`query`:: -(Optional, object) A query clause that retrieves a subset of data from the -source index. See <>. - `evaluation`:: (Required, object) Defines the type of evaluation you want to perform. See <>. @@ -63,6 +55,14 @@ Available evaluation types: -- +`index`:: +(Required, object) Defines the `index` in which the evaluation will be +performed. + +`query`:: +(Optional, object) A query clause that retrieves a subset of data from the +source index. See <>. + [[ml-evaluate-dfanalytics-resources]] ==== {dfanalytics-cap} evaluation resources @@ -93,6 +93,12 @@ document is an outlier. operating characteristic) score and optionally the curve. Default value is {"includes_curve": false}. + `confusion_matrix`::: + (Optional, object) Set the different thresholds of the {olscore} at where + the metrics (`tp` - true positive, `fp` - false positive, `tn` - true + negative, `fn` - false negative) are calculated. Default value is + {"at": [0.25, 0.50, 0.75]}. + `precision`::: (Optional, object) Set the different thresholds of the {olscore} at where the metric is calculated. Default value is {"at": [0.25, 0.50, 0.75]}. @@ -100,12 +106,6 @@ document is an outlier. `recall`::: (Optional, object) Set the different thresholds of the {olscore} at where the metric is calculated. Default value is {"at": [0.25, 0.50, 0.75]}. - - `confusion_matrix`::: - (Optional, object) Set the different thresholds of the {olscore} at where - the metrics (`tp` - true positive, `fp` - false positive, `tn` - true - negative, `fn` - false negative) are calculated. Default value is - {"at": [0.25, 0.50, 0.75]}. [[regression-evaluation-resources]] @@ -128,11 +128,11 @@ which outputs a prediction of values. `mean_squared_error`::: (Optional, object) Average squared difference between the predicted values and the actual (`ground truth`) value. - Read more on https://en.wikipedia.org/wiki/Mean_squared_error[Wikipedia] + For more information, read https://en.wikipedia.org/wiki/Mean_squared_error[this wiki article]. `r_squared`::: (Optional, object) Proportion of the variance in the dependent variable that is predictable from the independent variables. - Read more on https://en.wikipedia.org/wiki/Coefficient_of_determination[Wikipedia] + For more information, read https://en.wikipedia.org/wiki/Coefficient_of_determination[this wiki article]. @@ -160,16 +160,16 @@ belongs. Available metrics: `accuracy`::: - (Optional, object) Accuracy of predictions (per-class and overall) + (Optional, object) Accuracy of predictions (per-class and overall). + + `multiclass_confusion_matrix`::: + (Optional, object) Multiclass confusion matrix. `precision`::: - (Optional, object) Precision of predictions (per-class and average) + (Optional, object) Precision of predictions (per-class and average). `recall`::: - (Optional, object) Recall of predictions (per-class and average) - - `multiclass_confusion_matrix`::: - (Optional, object) Multiclass confusion matrix + (Optional, object) Recall of predictions (per-class and average). //// From 0e244e93056edd5e34d8ad373954f050f51d0020 Mon Sep 17 00:00:00 2001 From: Przemyslaw Witek Date: Wed, 15 Jan 2020 13:53:26 +0100 Subject: [PATCH 4/4] Remove sentences suggesting the need to add `.keyword` suffix to field name --- .../df-analytics/apis/evaluate-dfanalytics.asciidoc | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc b/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc index 97988fe39064f..ad4f0467750fd 100644 --- a/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc +++ b/docs/reference/ml/df-analytics/apis/evaluate-dfanalytics.asciidoc @@ -149,11 +149,7 @@ belongs. `predicted_field`:: (Required, string) The field in the `index` that contains the predicted value, - in other words the results of the {classanalysis}. The data type of this field - is string. You need to add `.keyword` to the predicted field name (the name - you put in the {classanalysis} object as `prediction_field_name` or the - default value of the same field if you didn't specified explicitly). For - example, `predicted_field` : `ml.animal_class_prediction.keyword`. + in other words the results of the {classanalysis}. `metrics`:: (Optional, object) Specifies the metrics that are used for the evaluation. @@ -382,7 +378,7 @@ POST _ml/data_frame/_evaluate "evaluation": { "classification": { <1> "actual_field": "animal_class", <2> - "predicted_field": "ml.animal_class_prediction.keyword", <3> + "predicted_field": "ml.animal_class_prediction", <3> "metrics": { "multiclass_confusion_matrix" : {} <4> } @@ -396,8 +392,7 @@ POST _ml/data_frame/_evaluate <2> The field that contains the ground truth value for the actual animal classification. This is required in order to evaluate results. <3> The field that contains the predicted value for animal classification by -the {classanalysis}. Since the field storing predicted class is dynamically -mapped as text and keyword, you need to add the `.keyword` suffix to the name. +the {classanalysis}. <4> Specifies the metric for the evaluation.