Skip to content

[ML] Machine learning data frame analytics #43544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 110 commits into from
Jun 25, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
87e459a
[FEATURE][ML] ML data frame analytics
dimitris-athanasiou Oct 30, 2018
a52eab6
[FEATURE][ML] Inline empty string in data frame data extractor
dimitris-athanasiou Nov 21, 2018
3dff025
[FEATURE][ML] Add TODO regarding renaming the job thread pool
dimitris-athanasiou Nov 21, 2018
5e24b88
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Nov 22, 2018
7b47107
[FEATURE][ML] Remove settings from run analytics action
dimitris-athanasiou Nov 22, 2018
6e3f832
[FEATURE][ML] Reindex data frame before starting analytics (#35835)
dimitris-athanasiou Nov 26, 2018
3f49eef
[FEATURE][ML] Write data frame configuration to process (#35914)
dimitris-athanasiou Nov 27, 2018
801665a
[FEATURE][ML] Only write numeric fields to data frame (#35961)
dimitris-athanasiou Nov 28, 2018
68bc59c
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Nov 29, 2018
8cf5337
[FEATURE][ML] Skip rows that have missing data (#36067)
dimitris-athanasiou Nov 30, 2018
44491db
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Nov 30, 2018
88496c2
[FEATURE][ML] Write control message to signify end of data (#36158)
dimitris-athanasiou Dec 3, 2018
8457b81
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Dec 7, 2018
fd3e6a9
[FEATURE][ML] Parse results and join them in the data-frame copy inde…
dimitris-athanasiou Dec 11, 2018
5bc55e5
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Dec 12, 2018
9bb2a02
change the download location of the ml native code build (#36733)
Dec 18, 2018
e869b60
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Jan 11, 2019
371f5d7
[FEATURE][ML] Fix a few minor issues after merging master
dimitris-athanasiou Jan 11, 2019
9fd1d10
[FEATURE][ML] Add checksum checks on dataframe result joining (#37259)
dimitris-athanasiou Jan 11, 2019
1655264
[ML] Rename analytics package to dataframe (#37583)
dimitris-athanasiou Jan 21, 2019
aea2e2b
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Jan 31, 2019
c0eaba0
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Feb 4, 2019
551f9a8
[ML] Data frame analytics as persistent tasks (#37919)
dimitris-athanasiou Feb 4, 2019
ceab407
[FEATURE][ML] Allow parsing different types of results from analytics…
dimitris-athanasiou Feb 5, 2019
450d4e1
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Feb 11, 2019
267ca3a
[FEATURE][ML] Add query support for dataframe analytics config (#38576)
benwtrent Feb 12, 2019
2b6e6c0
Revert "change the download location of the ml native code build (#36…
dimitris-athanasiou Feb 12, 2019
911b805
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Feb 14, 2019
a2ab55b
[FEATURE][ML] User config appropriate permission checks on creating/r…
benwtrent Feb 15, 2019
deac81b
[FEATURE][ML] Wire in data frame analytics progress reporting (#38808)
dimitris-athanasiou Feb 18, 2019
e306cc8
[FEATURE][ML] Add ability to include/exclude fields to/from analyses …
benwtrent Feb 19, 2019
90eaf94
[FEATURE][ML] Add timeout to the start DF analytics request (#39177)
dimitris-athanasiou Feb 20, 2019
0ced42a
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Feb 27, 2019
ca306c5
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Feb 27, 2019
9faeb94
[DATA-FRAME-ANALYTICS] Add task recovery on node change (#39416)
benwtrent Feb 27, 2019
dd5241f
Merge branch 'master' into feature-ml-data-frame-analytics
droberts195 Mar 5, 2019
7637656
[FEATURE][ML] Add model_memory_limit to data frame analytics (#39561)
droberts195 Mar 5, 2019
9a20e29
Merge branch 'master' into feature-ml-data-frame-analytics
droberts195 Mar 6, 2019
dad41c8
Fix build failure after merging master (#39743)
droberts195 Mar 6, 2019
911cbea
Merge branch 'master' into feature-ml-data-frame-analytics
droberts195 Mar 15, 2019
142edfe
[FEATURE] Fix up feature branch after merging master (#40094)
droberts195 Mar 15, 2019
a5f2b58
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Mar 26, 2019
278021f
Merge branch 'master' into feature-ml-data-frame-analytics
droberts195 Mar 27, 2019
2f1a60e
[FEATURE] Adjust ML memory tracker to include data frame analytics jo…
droberts195 Mar 27, 2019
481b42e
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Mar 28, 2019
139b956
[FEATURE][ML] Implement data frame evaluate API (#40328)
dimitris-athanasiou Apr 1, 2019
7fcd771
[FEATURE][ML] Add source and dest as objects to analytics config (#40…
dimitris-athanasiou Apr 4, 2019
6ec2162
[FEATURE][ML] Fix results joining when rows are skipped (#40906)
dimitris-athanasiou Apr 9, 2019
0ca566a
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Apr 10, 2019
0bf8ede
[FEATURE][ML] Remove DataFrameAnalyticsManagerIT
dimitris-athanasiou Apr 11, 2019
bea87d3
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Apr 11, 2019
d230a07
[FEATURE][ML] Add setting for results field name (#40927)
dimitris-athanasiou Apr 11, 2019
ef71e45
[ML][FEATURE] Use memory limit to assign data frame analytics jobs (#…
droberts195 Apr 11, 2019
90d9165
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Apr 15, 2019
b2bcaf2
[FEATURE][ML] Change df-analytics config to accept single analysis (#…
dimitris-athanasiou Apr 16, 2019
d6e5975
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Apr 18, 2019
f1140d5
[FEATURE][ML] Rename number_neighbors to n_neighbors (#41242)
dimitris-athanasiou Apr 18, 2019
cefa32a
Merge branch 'master' into feature-ml-data-frame-analytics
davidkyle Apr 23, 2019
1cd470c
Fix failing test as a result of a bad merge
davidkyle Apr 23, 2019
07afb22
Merge branch 'master' into feature-ml-data-frame-analytics
davidkyle Apr 24, 2019
60378e0
[ML] Make a copy of unmodifiable sett (#41490)
davidkyle Apr 24, 2019
901bab2
[FEATURE][ML] Client-side data structures for DataFrame analytics con…
przemekwitek Apr 25, 2019
2c40ecf
[FEATURE][ML] Client-side API methods for DataFrame analytics config.…
przemekwitek Apr 26, 2019
ff4cf9c
Merge branch 'master' into feature-ml-data-frame-analytics
benwtrent Apr 29, 2019
efd1556
Implement GetDataFrameAnalyticsStats API method (#41637)
przemekwitek Apr 30, 2019
75c7380
[FEATURE][ML] Client-side "Start" API method for DataFrame analytics …
przemekwitek Apr 30, 2019
fa8d9f4
Implement integration test for StartDataFrameAnalytics API method (#4…
przemekwitek Apr 30, 2019
362ced3
Merge branch 'master' into feature-ml-data-frame-analytics
benwtrent May 2, 2019
8ace7a5
Merge branch 'master' of github.com:elastic/elasticsearch into featur…
May 3, 2019
cd27499
Merge branch 'master' of github.com:elastic/elasticsearch into featur…
May 6, 2019
5f4ceeb
Merge branch 'master' of github.com:elastic/elasticsearch into featur…
May 6, 2019
362775b
[ML-DataFrame] add sync api (#41800)
May 6, 2019
2854bbc
[ML] Refine client code for data frame analytics (#41665)
przemekwitek May 7, 2019
2d8e55f
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou May 8, 2019
dce6f50
[ML] Import PageParams correctly
dimitris-athanasiou May 8, 2019
0192c84
[FEATURE][ML] Remove evaluation result abstraction (#41937)
dimitris-athanasiou May 8, 2019
6aa3a97
[ML-DataFrame] timebased continuous dataframe (#41880)
May 9, 2019
dd52e88
Merge branch 'master' into use-hlrc-in-integ-tests
dimitris-athanasiou May 24, 2019
e07b78c
[ML] QueryProvider.fromStream should be public
dimitris-athanasiou May 24, 2019
6d05f8f
[FEATURE][ML] Allow configuring outlier score threshold for feature i…
dimitris-athanasiou May 27, 2019
57e3879
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou May 29, 2019
eed4319
[ML] Fix compilation after merging master
dimitris-athanasiou May 29, 2019
240e542
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou May 30, 2019
f27886a
[ML] Remove argument from request converter params constructor
dimitris-athanasiou May 30, 2019
62fdff9
[ML] And add those params the right way
dimitris-athanasiou May 30, 2019
d96df6b
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou May 30, 2019
8f91e61
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou May 31, 2019
ca29685
[FEATURE][ML] Data frame analytics stop API (#42624)
dimitris-athanasiou May 31, 2019
626cae5
[ML][FEATURE] Implement client-side evaluation API (#42058)
przemekwitek May 31, 2019
bec3b83
[FEATURE][ML] Add allow_no_match to df-analytics get and get-stats AP…
dimitris-athanasiou May 31, 2019
dc017a4
[FEATURE][ML] Client-side "Stop" API method for DataFrame analytics c…
przemekwitek May 31, 2019
7d43d6e
[FEATURE][ML] Ignore `index.version.upgraded` index setting (#42768)
dimitris-athanasiou Jun 3, 2019
0939119
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Jun 4, 2019
69f5652
[ML] Fix compilation error from upstream
dimitris-athanasiou Jun 4, 2019
44856a5
[FEATURE][ML] auc_roc cannot be calculated when there are no inliers/…
dimitris-athanasiou Jun 5, 2019
5efcead
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Jun 7, 2019
a2268a2
[ML] Fix compilation after upstream changes
dimitris-athanasiou Jun 7, 2019
0598fa0
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Jun 10, 2019
16a26a5
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Jun 11, 2019
521734d
[FEATURE][ML] Ensure data extractor is not leaking scroll contexts (#…
dimitris-athanasiou Jun 11, 2019
3783f87
Add missing argument to the instantiation of DataFrameTransformConfig…
przemekwitek Jun 11, 2019
c3c45c0
[ML] Documentation for Data Frame Analytics high-level REST client (#…
przemekwitek Jun 12, 2019
d14466b
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Jun 12, 2019
84b78e7
Merge branch 'master' into feature-ml-data-frame-analytics
dimitris-athanasiou Jun 14, 2019
cf8a8bd
[FEATURE][ML] Specify stability in df-analytics apis
dimitris-athanasiou Jun 14, 2019
eced353
[FEATURE][ML] Fetch from source when fields are more then docvalue li…
dimitris-athanasiou Jun 14, 2019
d86bea5
[FEATURE][ML] Stregthen source dest validations for DF analytics (#43…
dimitris-athanasiou Jun 24, 2019
02afdcf
Merge branch 'master' into feature-ml-data-frame-analytics
benwtrent Jun 24, 2019
d6fb2eb
post merge fixes
benwtrent Jun 24, 2019
bbe7a2c
Merge branch 'master' into feature-ml-data-frame-analytics
elasticmachine Jun 24, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -32,19 +32,23 @@
import org.elasticsearch.client.ml.DeleteCalendarEventRequest;
import org.elasticsearch.client.ml.DeleteCalendarJobRequest;
import org.elasticsearch.client.ml.DeleteCalendarRequest;
import org.elasticsearch.client.ml.DeleteDataFrameAnalyticsRequest;
import org.elasticsearch.client.ml.DeleteDatafeedRequest;
import org.elasticsearch.client.ml.DeleteExpiredDataRequest;
import org.elasticsearch.client.ml.DeleteFilterRequest;
import org.elasticsearch.client.ml.DeleteForecastRequest;
import org.elasticsearch.client.ml.DeleteJobRequest;
import org.elasticsearch.client.ml.DeleteModelSnapshotRequest;
import org.elasticsearch.client.ml.EvaluateDataFrameRequest;
import org.elasticsearch.client.ml.FindFileStructureRequest;
import org.elasticsearch.client.ml.FlushJobRequest;
import org.elasticsearch.client.ml.ForecastJobRequest;
import org.elasticsearch.client.ml.GetBucketsRequest;
import org.elasticsearch.client.ml.GetCalendarEventsRequest;
import org.elasticsearch.client.ml.GetCalendarsRequest;
import org.elasticsearch.client.ml.GetCategoriesRequest;
import org.elasticsearch.client.ml.GetDataFrameAnalyticsRequest;
import org.elasticsearch.client.ml.GetDataFrameAnalyticsStatsRequest;
import org.elasticsearch.client.ml.GetDatafeedRequest;
import org.elasticsearch.client.ml.GetDatafeedStatsRequest;
import org.elasticsearch.client.ml.GetFiltersRequest;
Expand All @@ -61,12 +65,15 @@
import org.elasticsearch.client.ml.PreviewDatafeedRequest;
import org.elasticsearch.client.ml.PutCalendarJobRequest;
import org.elasticsearch.client.ml.PutCalendarRequest;
import org.elasticsearch.client.ml.PutDataFrameAnalyticsRequest;
import org.elasticsearch.client.ml.PutDatafeedRequest;
import org.elasticsearch.client.ml.PutFilterRequest;
import org.elasticsearch.client.ml.PutJobRequest;
import org.elasticsearch.client.ml.RevertModelSnapshotRequest;
import org.elasticsearch.client.ml.SetUpgradeModeRequest;
import org.elasticsearch.client.ml.StartDataFrameAnalyticsRequest;
import org.elasticsearch.client.ml.StartDatafeedRequest;
import org.elasticsearch.client.ml.StopDataFrameAnalyticsRequest;
import org.elasticsearch.client.ml.StopDatafeedRequest;
import org.elasticsearch.client.ml.UpdateDatafeedRequest;
import org.elasticsearch.client.ml.UpdateFilterRequest;
Expand Down Expand Up @@ -581,6 +588,115 @@ static Request deleteCalendarEvent(DeleteCalendarEventRequest deleteCalendarEven
return new Request(HttpDelete.METHOD_NAME, endpoint);
}

static Request putDataFrameAnalytics(PutDataFrameAnalyticsRequest putRequest) throws IOException {
String endpoint = new EndpointBuilder()
.addPathPartAsIs("_ml", "data_frame", "analytics")
.addPathPart(putRequest.getConfig().getId())
.build();
Request request = new Request(HttpPut.METHOD_NAME, endpoint);
request.setEntity(createEntity(putRequest, REQUEST_BODY_CONTENT_TYPE));
return request;
}

static Request getDataFrameAnalytics(GetDataFrameAnalyticsRequest getRequest) {
String endpoint = new EndpointBuilder()
.addPathPartAsIs("_ml", "data_frame", "analytics")
.addPathPart(Strings.collectionToCommaDelimitedString(getRequest.getIds()))
.build();
Request request = new Request(HttpGet.METHOD_NAME, endpoint);
RequestConverters.Params params = new RequestConverters.Params();
if (getRequest.getPageParams() != null) {
PageParams pageParams = getRequest.getPageParams();
if (pageParams.getFrom() != null) {
params.putParam(PageParams.FROM.getPreferredName(), pageParams.getFrom().toString());
}
if (pageParams.getSize() != null) {
params.putParam(PageParams.SIZE.getPreferredName(), pageParams.getSize().toString());
}
}
if (getRequest.getAllowNoMatch() != null) {
params.putParam(GetDataFrameAnalyticsRequest.ALLOW_NO_MATCH.getPreferredName(), Boolean.toString(getRequest.getAllowNoMatch()));
}
request.addParameters(params.asMap());
return request;
}

static Request getDataFrameAnalyticsStats(GetDataFrameAnalyticsStatsRequest getStatsRequest) {
String endpoint = new EndpointBuilder()
.addPathPartAsIs("_ml", "data_frame", "analytics")
.addPathPart(Strings.collectionToCommaDelimitedString(getStatsRequest.getIds()))
.addPathPartAsIs("_stats")
.build();
Request request = new Request(HttpGet.METHOD_NAME, endpoint);
RequestConverters.Params params = new RequestConverters.Params();
if (getStatsRequest.getPageParams() != null) {
PageParams pageParams = getStatsRequest.getPageParams();
if (pageParams.getFrom() != null) {
params.putParam(PageParams.FROM.getPreferredName(), pageParams.getFrom().toString());
}
if (pageParams.getSize() != null) {
params.putParam(PageParams.SIZE.getPreferredName(), pageParams.getSize().toString());
}
}
if (getStatsRequest.getAllowNoMatch() != null) {
params.putParam(GetDataFrameAnalyticsStatsRequest.ALLOW_NO_MATCH.getPreferredName(),
Boolean.toString(getStatsRequest.getAllowNoMatch()));
}
request.addParameters(params.asMap());
return request;
}

static Request startDataFrameAnalytics(StartDataFrameAnalyticsRequest startRequest) {
String endpoint = new EndpointBuilder()
.addPathPartAsIs("_ml", "data_frame", "analytics")
.addPathPart(startRequest.getId())
.addPathPartAsIs("_start")
.build();
Request request = new Request(HttpPost.METHOD_NAME, endpoint);
RequestConverters.Params params = new RequestConverters.Params();
if (startRequest.getTimeout() != null) {
params.withTimeout(startRequest.getTimeout());
}
request.addParameters(params.asMap());
return request;
}

static Request stopDataFrameAnalytics(StopDataFrameAnalyticsRequest stopRequest) {
String endpoint = new EndpointBuilder()
.addPathPartAsIs("_ml", "data_frame", "analytics")
.addPathPart(stopRequest.getId())
.addPathPartAsIs("_stop")
.build();
Request request = new Request(HttpPost.METHOD_NAME, endpoint);
RequestConverters.Params params = new RequestConverters.Params();
if (stopRequest.getTimeout() != null) {
params.withTimeout(stopRequest.getTimeout());
}
if (stopRequest.getAllowNoMatch() != null) {
params.putParam(
StopDataFrameAnalyticsRequest.ALLOW_NO_MATCH.getPreferredName(), Boolean.toString(stopRequest.getAllowNoMatch()));
}
request.addParameters(params.asMap());
return request;
}

static Request deleteDataFrameAnalytics(DeleteDataFrameAnalyticsRequest deleteRequest) {
String endpoint = new EndpointBuilder()
.addPathPartAsIs("_ml", "data_frame", "analytics")
.addPathPart(deleteRequest.getId())
.build();
return new Request(HttpDelete.METHOD_NAME, endpoint);
}

static Request evaluateDataFrame(EvaluateDataFrameRequest evaluateRequest) throws IOException {
String endpoint = new EndpointBuilder()
.addPathPartAsIs("_ml", "data_frame", "_evaluate")
.build();
Request request = new Request(HttpPost.METHOD_NAME, endpoint);
request.setEntity(createEntity(evaluateRequest, REQUEST_BODY_CONTENT_TYPE));
return request;
}

static Request putFilter(PutFilterRequest putFilterRequest) throws IOException {
String endpoint = new EndpointBuilder()
.addPathPartAsIs("_ml")
Expand Down
Loading