[ML] Improve residual model selection #468
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
edsavage
approved these changes
Apr 29, 2019
| add(maths_t::count(weight), n); | ||
| if (failed) { | ||
| LOG_ERROR(<< "Failed to compute log-likelihood"); | ||
| LOG_ERROR(<< "samples = " << core::CContainerPrinter::print(samples)); |
There was a problem hiding this comment.
Would it be useful to print out the weights here as well?
lib/maths/COneOfNPrior.cc
Outdated
| used.push_back(use); | ||
| varianceMismatchPenalties.push_back( | ||
| -m * MAXIMUM_LOG_BAYES_FACTOR * | ||
| std::max(1.0 - 9.0 * CBasicStatistics::variance(m_SampleMoments) / |
There was a problem hiding this comment.
I'm curious about the use of the 9.0 factor here - perhaps replacing with a named constant would aid understanding?
There was a problem hiding this comment.
This is the maximum relative error in the estimated variance of the model (vs the sample variance) for which we will not penalise it at all. I named this constant in 7c76c7f.
|
retest |
tveasey
added a commit
to tveasey/ml-cpp-1
that referenced
this pull request
Apr 30, 2019
tveasey
added a commit
that referenced
this pull request
May 1, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This addresses the remaining issues from #124, which were in fact related to poor model selection. These data sets pose problems because none of candidate residual models are a particularly good fit for all values. The outcome is we end up choosing a model with undesirable characteristics. This also interferes with detecting change points correctly.
This PR makes two changes:
I've reviewed the result changes across a range of our QA data sets and where it has affected results it is doing better job. This generally affects data sets where there are some low and some very high values and also small numbers of values which are very different from typical. In such cases, we will generally use less skewed and lighter tailed models as a result. This means we end up being more sensitive to values which are different from our predictions. This also makes model selection more stable, so we see fewer changes to the selected model; these events are often most clear when model plot is enabled and are associated with sudden changes in bounds.