[ML] Revise refresh policy on requests to the config index

[dimitris-athanasiou]

Configurations are now stored in the config index. Throughout the code, requests to the config index have their refresh policy set to IMMEDIATE. This is problematic. When an action is performed that causes multiple refreshes to the index (e.g. deleting mutliple jobs/datafeeds), the refreshes are throttled and long delays (~10s) have been observed.

This is also probably the reason behind #30300.

Futhermore, requests that are not a result of a user action need not have a refresh. For example, when the established model memory is updated during runtime. Refreshing on those is delaying the job for no reason.

In order to avoid these delays, but also make sure single requests are responsive we need to:

• Use WAIT_FOR for requests that are a result of a user action
• Use NONE for internal requests
• Reduce the refresh_interval to a value that is responsive without putting pressure on the index (e.g. 100ms)? Note the config index is a low volume index so it should be able to handle quick refreshing.

[elasticmachine]

Pinging @elastic/ml-core

[dimitris-athanasiou]

This was a false alarm. The delays I observed were caused by a sleep in the master operation of the job delete action. This was blocking the master node and thus the datafeed delete action was stalled.