[7.x] [ML] adding new flag exclude_generated that removes generated fields in GET config APIs (#63899)(#63092) (#63177)

* [ML] adding for_export flag for ml plugin GET resource APIs (#63092)

This adds the new `for_export` flag to the following APIs:

- GET _ml/anomaly_detection/<job_id>
- GET _ml/datafeeds/<datafeed_id>
- GET _ml/data_frame/analytics/<analytics_id>

The flag is designed for cloning or exporting configuration objects to later be put into the same cluster or a separate cluster.

The following fields are not returned in the objects:

- any field that is not user settable (e.g. version, create_time)
- any field that is a calculated default value (e.g. datafeed chunking_config)
- any field that would effectively require changing to be of use (e.g. datafeed job_id)
- any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by)

closes #63055

* [ML] adding new flag exclude_generated that removes generated fields in GET config APIs (#63899)

When exporting and cloning ml configurations in a cluster it can be
frustrating to remove all the fields that were generated by
the plugin. Especially as the number of these fields change
from version to version.

This flag, exclude_generated, allows the GET config APIs to return
configurations with these generated fields removed.

APIs supporting this flag:
- GET _ml/anomaly_detection/<job_id>
- GET _ml/datafeeds/<datafeed_id>
- GET _ml/data_frame/analytics/<analytics_id>

The following fields are not returned in the objects:

- any field that is not user settable (e.g. version, create_time)
- any field that is a calculated default value (e.g. datafeed chunking_config)
- any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by)

relates to #63055

Author: benwtrent

client/rest-high-level/src/main/java/org/elasticsearch/client/MLRequestConverters.java

@@ -121,7 +121,10 @@ static Request getJob(GetJobRequest getJobRequest) {
 
         RequestConverters.Params params = new RequestConverters.Params();
         if (getJobRequest.getAllowNoMatch() != null) {
-            params.putParam("allow_no_match", Boolean.toString(getJobRequest.getAllowNoMatch()));
+            params.putParam(GetJobRequest.ALLOW_NO_MATCH.getPreferredName(), Boolean.toString(getJobRequest.getAllowNoMatch()));
+        }
+        if (getJobRequest.getExcludeGenerated() != null) {
+            params.putParam(GetJobRequest.EXCLUDE_GENERATED, Boolean.toString(getJobRequest.getExcludeGenerated()));
         }
         request.addParameters(params.asMap());
         return request;
@@ -270,6 +273,9 @@ static Request getDatafeed(GetDatafeedRequest getDatafeedRequest) {
             params.putParam(GetDatafeedRequest.ALLOW_NO_MATCH.getPreferredName(),
                     Boolean.toString(getDatafeedRequest.getAllowNoMatch()));
         }
+        if (getDatafeedRequest.getExcludeGenerated() != null) {
+            params.putParam(GetDatafeedRequest.EXCLUDE_GENERATED, Boolean.toString(getDatafeedRequest.getExcludeGenerated()));
+        }
         request.addParameters(params.asMap());
         return request;
     }
@@ -645,7 +651,10 @@ static Request getDataFrameAnalytics(GetDataFrameAnalyticsRequest getRequest) {
             }
         }
         if (getRequest.getAllowNoMatch() != null) {
-            params.putParam(GetDataFrameAnalyticsRequest.ALLOW_NO_MATCH.getPreferredName(), Boolean.toString(getRequest.getAllowNoMatch()));
+            params.putParam(GetDataFrameAnalyticsRequest.ALLOW_NO_MATCH, Boolean.toString(getRequest.getAllowNoMatch()));
+        }
+        if (getRequest.getExcludeGenerated() != null) {
+            params.putParam(GetDataFrameAnalyticsRequest.EXCLUDE_GENERATED, Boolean.toString(getRequest.getExcludeGenerated()));
         }
         request.addParameters(params.asMap());
         return request;
@@ -786,8 +795,8 @@ static Request getTrainedModels(GetTrainedModelsRequest getTrainedModelsRequest)
         if (getTrainedModelsRequest.getTags() != null) {
             params.putParam(GetTrainedModelsRequest.TAGS, Strings.collectionToCommaDelimitedString(getTrainedModelsRequest.getTags()));
         }
-        if (getTrainedModelsRequest.getForExport() != null) {
-            params.putParam(GetTrainedModelsRequest.FOR_EXPORT, Boolean.toString(getTrainedModelsRequest.getForExport()));
+        if (getTrainedModelsRequest.getExcludeGenerated() != null) {
+            params.putParam(GetTrainedModelsRequest.EXCLUDE_GENERATED, Boolean.toString(getTrainedModelsRequest.getExcludeGenerated()));
         }
         Request request = new Request(HttpGet.METHOD_NAME, endpoint);
         request.addParameters(params.asMap());

client/rest-high-level/src/main/java/org/elasticsearch/client/ml/GetDataFrameAnalyticsRequest.java

@@ -23,7 +23,6 @@
 import org.elasticsearch.client.ValidationException;
 import org.elasticsearch.client.core.PageParams;
 import org.elasticsearch.common.Nullable;
-import org.elasticsearch.common.ParseField;
 
 import java.util.Arrays;
 import java.util.List;
@@ -32,11 +31,13 @@
 
 public class GetDataFrameAnalyticsRequest implements Validatable {
 
-    public static final ParseField ALLOW_NO_MATCH = new ParseField("allow_no_match");
+    public static final String ALLOW_NO_MATCH = "allow_no_match";
+    public static final String EXCLUDE_GENERATED = "exclude_generated";
 
     private final List<String> ids;
     private Boolean allowNoMatch;
     private PageParams pageParams;
+    private Boolean excludeGenerated;
 
     /**
      * Helper method to create a request that will get ALL Data Frame Analytics
@@ -58,6 +59,22 @@ public Boolean getAllowNoMatch() {
         return allowNoMatch;
     }
 
+    /**
+     * Setting this flag to `true` removes certain fields from the configuration on retrieval.
+     *
+     * This is useful when getting the configuration and wanting to put it in another cluster.
+     *
+     * Default value is false.
+     * @param excludeGenerated Boolean value indicating if certain fields should be removed
+     */
+    public void setExcludeGenerated(boolean excludeGenerated) {
+        this.excludeGenerated = excludeGenerated;
+    }
+
+    public Boolean getExcludeGenerated() {
+        return excludeGenerated;
+    }
+
     /**
      * Whether to ignore if a wildcard expression matches no data frame analytics.
      *
@@ -94,11 +111,12 @@ public boolean equals(Object o) {
         GetDataFrameAnalyticsRequest other = (GetDataFrameAnalyticsRequest) o;
         return Objects.equals(ids, other.ids)
             && Objects.equals(allowNoMatch, other.allowNoMatch)
+            && Objects.equals(excludeGenerated, other.excludeGenerated)
             && Objects.equals(pageParams, other.pageParams);
     }
 
     @Override
     public int hashCode() {
-        return Objects.hash(ids, allowNoMatch, pageParams);
+        return Objects.hash(ids, allowNoMatch, excludeGenerated, pageParams);
     }
 }

client/rest-high-level/src/main/java/org/elasticsearch/client/ml/GetDatafeedRequest.java

@@ -42,10 +42,12 @@ public class GetDatafeedRequest extends ActionRequest implements ToXContentObjec
 
     public static final ParseField DATAFEED_IDS = new ParseField("datafeed_ids");
     public static final ParseField ALLOW_NO_MATCH = new ParseField("allow_no_match");
+    public static final String EXCLUDE_GENERATED = "exclude_generated";
 
     private static final String ALL_DATAFEEDS = "_all";
     private final List<String> datafeedIds;
     private Boolean allowNoMatch;
+    private Boolean excludeGenerated;
 
     @SuppressWarnings("unchecked")
     public static final ConstructingObjectParser<GetDatafeedRequest, Void> PARSER = new ConstructingObjectParser<>(
@@ -101,14 +103,30 @@ public Boolean getAllowNoMatch() {
         return allowNoMatch;
     }
 
+    /**
+     * Setting this flag to `true` removes certain fields from the configuration on retrieval.
+     *
+     * This is useful when getting the configuration and wanting to put it in another cluster.
+     *
+     * Default value is false.
+     * @param excludeGenerated Boolean value indicating if certain fields should be removed
+     */
+    public void setExcludeGenerated(boolean excludeGenerated) {
+        this.excludeGenerated = excludeGenerated;
+    }
+
+    public Boolean getExcludeGenerated() {
+        return excludeGenerated;
+    }
+
     @Override
     public ActionRequestValidationException validate() {
         return null;
     }
 
     @Override
     public int hashCode() {
-        return Objects.hash(datafeedIds, allowNoMatch);
+        return Objects.hash(datafeedIds, excludeGenerated, allowNoMatch);
     }
 
     @Override
@@ -123,7 +141,8 @@ public boolean equals(Object other) {
 
         GetDatafeedRequest that = (GetDatafeedRequest) other;
         return Objects.equals(datafeedIds, that.datafeedIds) &&
-            Objects.equals(allowNoMatch, that.allowNoMatch);
+            Objects.equals(allowNoMatch, that.allowNoMatch) &&
+            Objects.equals(excludeGenerated, that.excludeGenerated);
     }
 
     @Override

client/rest-high-level/src/main/java/org/elasticsearch/client/ml/GetJobRequest.java

@@ -43,10 +43,12 @@ public class GetJobRequest extends ActionRequest implements ToXContentObject {
 
     public static final ParseField JOB_IDS = new ParseField("job_ids");
     public static final ParseField ALLOW_NO_MATCH = new ParseField("allow_no_match");
+    public static final String EXCLUDE_GENERATED = "exclude_generated";
 
     private static final String ALL_JOBS = "_all";
     private final List<String> jobIds;
     private Boolean allowNoMatch;
+    private Boolean excludeGenerated;
 
     @SuppressWarnings("unchecked")
     public static final ConstructingObjectParser<GetJobRequest, Void> PARSER = new ConstructingObjectParser<>(
@@ -101,14 +103,30 @@ public Boolean getAllowNoMatch() {
         return allowNoMatch;
     }
 
+    /**
+     * Setting this flag to `true` removes certain fields from the configuration on retrieval.
+     *
+     * This is useful when getting the configuration and wanting to put it in another cluster.
+     *
+     * Default value is false.
+     * @param excludeGenerated Boolean value indicating if certain fields should be removed
+     */
+    public void setExcludeGenerated(boolean excludeGenerated) {
+        this.excludeGenerated = excludeGenerated;
+    }
+
+    public Boolean getExcludeGenerated() {
+        return excludeGenerated;
+    }
+
     @Override
     public ActionRequestValidationException validate() {
         return null;
     }
 
     @Override
     public int hashCode() {
-        return Objects.hash(jobIds, allowNoMatch);
+        return Objects.hash(jobIds, excludeGenerated, allowNoMatch);
     }
 
     @Override
@@ -123,6 +141,7 @@ public boolean equals(Object other) {
 
         GetJobRequest that = (GetJobRequest) other;
         return Objects.equals(jobIds, that.jobIds) &&
+            Objects.equals(excludeGenerated, that.excludeGenerated) &&
             Objects.equals(allowNoMatch, that.allowNoMatch);
     }
 

client/rest-high-level/src/main/java/org/elasticsearch/client/ml/GetTrainedModelsRequest.java

@@ -39,7 +39,7 @@ public class GetTrainedModelsRequest implements Validatable {
     private static final String TOTAL_FEATURE_IMPORTANCE = "total_feature_importance";
     private static final String FEATURE_IMPORTANCE_BASELINE = "feature_importance_baseline";
     public static final String ALLOW_NO_MATCH = "allow_no_match";
-    public static final String FOR_EXPORT = "for_export";
+    public static final String EXCLUDE_GENERATED = "exclude_generated";
     public static final String DECOMPRESS_DEFINITION = "decompress_definition";
     public static final String TAGS = "tags";
     public static final String INCLUDE = "include";
@@ -48,7 +48,7 @@ public class GetTrainedModelsRequest implements Validatable {
     private Boolean allowNoMatch;
     private Set<String> includes = new HashSet<>();
     private Boolean decompressDefinition;
-    private Boolean forExport;
+    private Boolean excludeGenerated;
     private PageParams pageParams;
     private List<String> tags;
 
@@ -163,8 +163,8 @@ public GetTrainedModelsRequest setTags(String... tags) {
         return setTags(Arrays.asList(tags));
     }
 
-    public Boolean getForExport() {
-        return forExport;
+    public Boolean getExcludeGenerated() {
+        return excludeGenerated;
     }
 
     /**
@@ -173,10 +173,10 @@ public Boolean getForExport() {
      * This is useful when getting the model and wanting to put it in another cluster.
      *
      * Default value is false.
-     * @param forExport Boolean value indicating if certain fields should be removed from the mode on GET
+     * @param excludeGenerated Boolean value indicating if certain fields should be removed from the mode on GET
      */
-    public GetTrainedModelsRequest setForExport(Boolean forExport) {
-        this.forExport = forExport;
+    public GetTrainedModelsRequest setExcludeGenerated(Boolean excludeGenerated) {
+        this.excludeGenerated = excludeGenerated;
         return this;
     }
 
@@ -198,12 +198,12 @@ public boolean equals(Object o) {
             && Objects.equals(allowNoMatch, other.allowNoMatch)
             && Objects.equals(decompressDefinition, other.decompressDefinition)
             && Objects.equals(includes, other.includes)
-            && Objects.equals(forExport, other.forExport)
+            && Objects.equals(excludeGenerated, other.excludeGenerated)
             && Objects.equals(pageParams, other.pageParams);
     }
 
     @Override
     public int hashCode() {
-        return Objects.hash(ids, allowNoMatch, pageParams, decompressDefinition, includes, forExport);
+        return Objects.hash(ids, allowNoMatch, pageParams, decompressDefinition, includes, excludeGenerated);
     }
 }

client/rest-high-level/src/test/java/org/elasticsearch/client/documentation/MlClientDocumentationIT.java

@@ -343,6 +343,7 @@ public void testGetJob() throws Exception {
             // tag::get-job-request
             GetJobRequest request = new GetJobRequest("get-machine-learning-job1", "get-machine-learning-job*"); // <1>
             request.setAllowNoMatch(true); // <2>
+            request.setExcludeGenerated(false); // <3>
             // end::get-job-request
 
             // tag::get-job-execute
@@ -839,6 +840,7 @@ public void testGetDatafeed() throws Exception {
             // tag::get-datafeed-request
             GetDatafeedRequest request = new GetDatafeedRequest(datafeedId); // <1>
             request.setAllowNoMatch(true); // <2>
+            request.setExcludeGenerated(false); // <3>
             // end::get-datafeed-request
 
             // tag::get-datafeed-execute
@@ -2864,6 +2866,7 @@ public void testGetDataFrameAnalytics() throws Exception {
         {
             // tag::get-data-frame-analytics-request
             GetDataFrameAnalyticsRequest request = new GetDataFrameAnalyticsRequest("my-analytics-config"); // <1>
+            request.setExcludeGenerated(false); // <2>
             // end::get-data-frame-analytics-request
 
             // tag::get-data-frame-analytics-execute
@@ -3728,7 +3731,7 @@ public void testGetTrainedModels() throws Exception {
                 .setDecompressDefinition(false) // <6>
                 .setAllowNoMatch(true) // <7>
                 .setTags("regression") // <8>
-                .setForExport(false); // <9>
+                .setExcludeGenerated(false); // <9>
             // end::get-trained-models-request
             request.setTags((List<String>)null);
 

docs/java-rest/high-level/ml/get-data-frame-analytics.asciidoc

@@ -23,6 +23,9 @@ IDs, or the special wildcard `_all` to get all {dfanalytics-jobs}.
 include-tagged::{doc-tests-file}[{api}-request]
 --------------------------------------------------
 <1> Constructing a new GET request referencing an existing {dfanalytics-job}
+<2> Optional boolean value for requesting the {dfanalytics-job} in a format that can
+then be put into another cluster. Certain fields that can only be set when
+the {dfanalytics-job} is created are removed.
 
 include::../execution.asciidoc[]
 

docs/java-rest/high-level/ml/get-datafeed.asciidoc

@@ -25,6 +25,9 @@ include-tagged::{doc-tests-file}[{api}-request]
 contain wildcards.
 <2> Whether to ignore if a wildcard expression matches no datafeeds.
  (This includes `_all` string or when no datafeeds have been specified).
+<3> Optional boolean value for requesting the datafeed in a format that can
+then be put into another cluster. Certain fields that can only be set when
+the datafeed is created are removed.
 
 [id="{upid}-{api}-response"]
 ==== Get datafeeds response

docs/java-rest/high-level/ml/get-job.asciidoc

@@ -25,6 +25,9 @@ include-tagged::{doc-tests-file}[{api}-request]
 wildcards.
 <2> Whether to ignore if a wildcard expression matches no {anomaly-jobs}.
  (This includes `_all` string or when no jobs have been specified).
+<3> Optional boolean value for requesting the job in a format that can
+then be put into another cluster. Certain fields that can only be set when
+the job is created are removed.
 
 [id="{upid}-{api}-response"]
 ==== Get {anomaly-jobs} response

docs/reference/ml/anomaly-detection/apis/get-datafeed.asciidoc

@@ -19,7 +19,7 @@ Retrieves configuration information for {dfeeds}.
 
 `GET _ml/datafeeds/` +
 
-`GET _ml/datafeeds/_all` 
+`GET _ml/datafeeds/_all`
 
 [[ml-get-datafeed-prereqs]]
 == {api-prereq-title}
@@ -36,7 +36,7 @@ comma-separated list of {dfeeds} or a wildcard expression. You can get
 information for all {dfeeds} by using `_all`, by specifying `*` as the
 `<feed_id>`, or by omitting the `<feed_id>`.
 
-IMPORTANT: This API returns a maximum of 10,000 {dfeeds}. 
+IMPORTANT: This API returns a maximum of 10,000 {dfeeds}.
 
 [[ml-get-datafeed-path-parms]]
 == {api-path-parms-title}
@@ -57,6 +57,10 @@ all {dfeeds}.
 (Optional, boolean)
 include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=allow-no-datafeeds]
 
+`exclude_generated`::
+(Optional, boolean)
+include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=exclude-generated]
+
 [[ml-get-datafeed-results]]
 == {api-response-body-title}