[ML] Error - CTimeSeriesModel@990 Failed to compute P([nan] | weight = [[nan, 0.7091527]], time = 1547594505)

[edsavage]

Found in build: 6.3.2

With job config:

      "analysis_config" : {
        "bucket_span" : "1h",
        "detectors" : [
          { 
            "detector_description" : "mean(tiemposaltos)",
            "function" : "mean",
            "field_name" : "tiemposaltos",
            "partition_field_name" : "sistema.keyword",
            "detector_index" : 0
          },
          { 
            "detector_description" : "mean(errores)",
            "function" : "mean",
            "field_name" : "errores",
            "partition_field_name" : "sistema.keyword",
            "detector_index" : 1
          },
          { 
            "detector_description" : "mean(transacciones)",
            "function" : "mean",
            "field_name" : "transacciones",
            "partition_field_name" : "sistema.keyword",
            "detector_index" : 2
          }
        ],
        "influencers" : [
          "sistema.keyword"
        ]
      }

Errors, that repeat every 10 minutes, have been reported in the logs:

[2019-01-15T17:51:46,344][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [monitores] [autodetect/8947] [CDecompositionComponent.cc@147] Failed calculating confidence interval: Error in function boost::math::normal_distribution::normal_distribution: Scale parameter is nan, but must be > 0 !, n = 20.0602, m = nan, sd = nan, confidence = 50
[2019-01-15T17:51:46,344][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [monitores] [autodetect/8947] [CDecompositionComponent.cc@147] Failed calculating confidence interval: Error in function boost::math::normal_distribution::normal_distribution: Scale parameter is nan, but must be > 0 !, n = 13.5703, m = nan, sd = nan, confidence = 50
[2019-01-15T17:51:46,347][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [monitores] [autodetect/8947] [CNormalMeanPrecConjugate.cc@192] Error calculating joint distribution: Bad variance scale nan
[2019-01-15T17:51:46,347][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [monitores] [autodetect/8947] [CNormalMeanPrecConjugate.cc@274] Failed to compute probability of less likely samples
[2019-01-15T17:51:46,348][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [monitores] [autodetect/8947] [CNormalMeanPrecConjugate.cc@1152] Failed computing probability for [nan]
[2019-01-15T17:51:46,348][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [monitores] [autodetect/8947] [CTimeSeriesModel.cc@990] Failed to compute P([nan] | weight = [[nan, 0.7091527]], time = 1547594505)
[2019-01-15T17:51:46,348][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [monitores] [autodetect/8947] [CIndividualModelDetail.h@103] Failed to compute P([[0]] | feature = 'arithmetic mean value by person', @ [[1547594505]], elapsedTime = 14947200, person = )

Initial analysis indicates that the errors stem from a sample with NaN value being used in CUnivariateTimeSeriesModel::uncorrelatedProbability and then percolating up the call stack.

[tveasey]

Looking at this I'd say this is likely source of the NaNs is related to the CTimeSeriesDecomposition: this would explain both the NaN for the detrended value and NaN for the variance scale. I think [CDecompositionComponent.cc@147] Failed calculating confidence interval: Error in function boost::math::normal_distribution::normal_distribution: Scale parameter is nan, but must be > 0 !, n = 20.0602, m = nan, sd = nan, confidence = 50 is the root cause here. We need to understand mechanisms by which the mean and standard deviation from the spline could be NaN. This may be tricky without the data to generate, but we could probably make this more defensive. For example, as a minimum we can trap this case and reinitialise this aspect of the model.

[PirapP]

Hi.
I was who rise the ticket. Maybe I misunderstood your post. But do you need the data that the ml job use? Or do you need some other data?

[droberts195]

@PirapP yes, the ideal way for us to reproduce this would be to run with the same data that your job analysed. However, we realise that may be impractical if the job was running for many months before it had this problem.

From what I can see these values are simple integer counts right? So I guess the most interesting thing for us to try and reproduce this is to know if there's periodicity in the values. For example, is there a periodic pattern in transacciones but not errores? Does this periodic pattern vary for different values of sistema.keyword? Given that the likely cause of the error is something in CDecompositionComponent I imagine some feature of the periodicity is what has exposed the bug.

[rbochz28]

I am experiencing this issue and it is potentially causing my ML models to completely crash and fail to train properly. I noticed there was a v7.1.0 tag on the merge link above. Does that mean this is fixed in elastic 7.1?

[tveasey]

Sorry @rbochz28. This was a labelling mistake on my part when I updated. There was inconsistent labelling in the original PR. I just checked the branch commits and relabelled accordingly. This fix first made it into 7.2.

Note that although this reduces the reduces the damage that the numerical issue has on the model, by discarding bad state, it doesn't fix the underlying cause. I fixed a long standing issue we had recently in #1157, which may be related. This will be released in 7.8.

It would be good to know if you still have any problems after upgrading.

[tveasey]

I believe that all remaining causes for this were fixed in #1225 and we have not seen any errors of this this sort in internal testing subsequently. I'm therefore closing this issue as fixed.