[ML] Prepare parsing phase_progress from DFA process #55580
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/ml-core (:ml) |
przemekwitek
approved these changes
Apr 22, 2020
| @@ -58,6 +62,8 @@ | |||
| } | |||
|
|
|||
| private final RowResults rowResults; | |||
| // TODO remove after process is writing out phase_progress | |||
There was a problem hiding this comment.
Shouldn't this TODO be for line 67?
| @@ -164,11 +165,18 @@ private void processResult(AnalyticsResult result, DataFrameRowsJoiner resultsJo | |||
| if (rowResults != null) { | |||
| resultsJoiner.processRowResults(rowResults); | |||
| } | |||
| PhaseProgress phaseProgress = result.getPhaseProgress(); | |||
| if (phaseProgress != null) { | |||
| LOGGER.debug("[{}] Analyzing progress updated to [{}]", analytics.getId(), phaseProgress.getProgressPercent()); | |||
There was a problem hiding this comment.
Should we also print the phase?
Data frame analytics process currently reports progress as an integer `progress_percent`. We parse that and report it from the _stats API as the progress of the `analyzing` phase. However, we want to allow the DFA process to report progress for more than one phase. This commit prepares for this by parsing `phase_progress` from the process, an object that contains the `phase` name plus the `progress_percent` for that phase.
dimitris-athanasiou
force-pushed
the
parse-phase-progress-from-analytics-process
branch
from
8138f15 to
a69f355
Compare
dimitris-athanasiou
deleted the
parse-phase-progress-from-analytics-process
branch
dimitris-athanasiou
added a commit
that referenced
this pull request
Apr 22, 2020
…55587) Data frame analytics process currently reports progress as an integer `progress_percent`. We parse that and report it from the _stats API as the progress of the `analyzing` phase. However, we want to allow the DFA process to report progress for more than one phase. This commit prepares for this by parsing `phase_progress` from the process, an object that contains the `phase` name plus the `progress_percent` for that phase. Backport of #55580
dimitris-athanasiou
added a commit
to dimitris-athanasiou/elasticsearch
that referenced
this pull request
Apr 24, 2020
Since elastic#55580 we've introduced a new format for parsing progress from the data frame analytics process. As the process is now writing out progress in this new way, we can remove the parsing of the old format.
dimitris-athanasiou
added a commit
that referenced
this pull request
Apr 24, 2020
Since #55580 we've introduced a new format for parsing progress from the data frame analytics process. As the process is now writing out progress in this new way, we can remove the parsing of the old format.
dimitris-athanasiou
added a commit
to dimitris-athanasiou/elasticsearch
that referenced
this pull request
Apr 25, 2020
This is a continuation from elastic#55580. Now that we're parsing phase progresses from the analytics process we change `ProgressTracker` to allow for custom phases between the `loading_data` and `writing_results` phases. Each `DataFrameAnalysis` may declare its own phases. This commit sets things in place for the analytics process to start reporting different phases per analysis type. However, this is still preserving existing behaviour as all analyses currently declare a single `analyzing` phase.
dimitris-athanasiou
added a commit
that referenced
this pull request
Apr 27, 2020
This is a continuation from #55580. Now that we're parsing phase progresses from the analytics process we change `ProgressTracker` to allow for custom phases between the `loading_data` and `writing_results` phases. Each `DataFrameAnalysis` may declare its own phases. This commit sets things in place for the analytics process to start reporting different phases per analysis type. However, this is still preserving existing behaviour as all analyses currently declare a single `analyzing` phase.
dimitris-athanasiou
added a commit
that referenced
this pull request
Apr 27, 2020
) (#55791) This is a continuation from #55580. Now that we're parsing phase progresses from the analytics process we change `ProgressTracker` to allow for custom phases between the `loading_data` and `writing_results` phases. Each `DataFrameAnalysis` may declare its own phases. This commit sets things in place for the analytics process to start reporting different phases per analysis type. However, this is still preserving existing behaviour as all analyses currently declare a single `analyzing` phase. Backport of #55763
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Data frame analytics process currently reports progress as
an integer
progress_percent. We parse that and report itfrom the _stats API as the progress of the
analyzingphase.However, we want to allow the DFA process to report progress
for more than one phase. This commit prepares for this by
parsing
phase_progressfrom the process, an object thatcontains the
phasename plus theprogress_percentfor thatphase.