[ML] Handle new actual_memory_usage_bytes field in model size stats.

[edsavage]

The actual_memory_usage_bytes field represents the real, physical memory allocated to the autodetect process as reported by the OS. Reporting this value in the model size stats associated with an AD job is useful, especially in OOM situations.

Relates elastic/ml-cpp#2846

[elasticsearchmachine]

Hi @edsavage, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[edsavage]

The new actual_memory_usage_bytes field can be seen in the screenshot below. It will require formatting by Kibana to represent its value in a manner similar to model_bytes, peak_model_bytes etc.

[davidkyle]

LGTM

Please can you add a test that the new field appears in the Get model stats API. Add it to this test

elasticsearch/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/ml/jobs_get_stats.yml

Line 104 in f1b1983

- gte: { jobs.0.model_size_stats.model_bytes: 0 }

[davidkyle]

Is the config version change necessary? Model size stats are considered to be a result type not a config type and are stored in the .ml-anomalies-x indices.

ModelSizeStats has 2 parsers; a strict parser that errors if it does not recognise a field and a lenient parser that ignores unknown fields. The strict parser is used reading the output from autodetect and the lenient is used reading the documents from Elasticsearch. This way in a mixed version cluster if a upgraded node stores the stats doc with the actual_memory_usage_bytes field then when an old node reads the doc it won't error when it sees the unknown actual_memory_usage_bytes field. This mechanism works well for backwards compatibility and means that most of the time versioning is not required.

[edsavage]

TIL, Thanks Dave!