[ML] Force closing an ML job can leave unfinished forecast jobs behind

[hendrikmuhs]

Affected versions: 6.1 -

Problem

When a ml job is force closed the native process gets terminated without waiting for unfinished forecasts. These forecasts end as started or in progress but will never finish. This is only a reporting problem, there are no hanging processes or tasks.

Mitigation

Manually delete forecast documents from the result index (replace job_id and forecast_id accordingly):

POST .ml-anomalies-.write-{job_id}/_delete_by_query
{
  "query": {
    "term": {
      "forecast_id": {
        "value": "{forecast_id}"
      }
    }
  }
}

Fix discussion

A force close should cleanup queued forecasts or mark them as failed.

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[sophiec20]

Please confirm if this mitigation is correct. Error seen when attempting to delete forecast job indicates it needs to be either FAILED or FINISHED to be deleted.

Provided there are no hanging processes or threads, the impact is forecast docs in the results index, which is undesirable especially for large forecasts.

Agree that force closing the anomaly detection job should also ensure forecast jobs are cleaned up. Suggest we also have a force delete for forecasts for consistency.

[droberts195]

When a ml job is force closed the native process gets terminated without waiting for unfinished forecasts

The same thing also happens when a node running a forecast leaves the cluster for any reason. (Power failure, bug, planned upgrade - any reason.)

We should probably check for and clean up forecast statuses in the persistent task assignment. When a job is getting assigned to a new node we know that any forecasts that were in progress on the old node have failed.