[ML] Node can go out of memory during daily maintainance or when listing snapshots

[hendrikmuhs]

Affected versions: 6.8-7.12

Large jobs with a lot of partitions can get very big, retrieving snapshots for such a job can cause a node to go out of memory.

This problem can happen:

• when using the Get model snapshot API either directly or indirectly triggered by the UI clicking on the Model Snapshots tab.
• during daily maintenance when expired model snapshots should get deleted (unfortunately the OOM happens in the preparation step, therefore this error might re-appear)

Mitigation:

Best, consider an upgrade to 7.12, if this is not an option:

• delete the offending job, consider re-creating it less resource-hungry(e.g. by splitting)
• Increasing the heap space of the node(temporarily until upgrading to 7.12)
• manually delete snapshots (use the size parameter with a small value to avoid going OOM), note as new snapshots get created this must be done regularly or after initial cleanup retain less snapshots

Solution:

It's not required to load all data when listing snapshots / find candidates to remove on daily maintenance. With the use of a source filter we avoid loading unnecessary parts of snapshots. This has the positive side-effect that less data will be transferred and the model snapshot API should become more responsive in general.

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[droberts195]

if this is not an option:

• delete the offending job, consider re-creating it less resource-hungry(e.g. by splitting)

Unfortunately, deleting the job may not work either, as one of the steps in deleting the job is to get model snapshots, to find the IDs of the state documents to be deleted 🤦 .