You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For anomaly detection memory limits/estimates are done in terms of whole megabytes.
For data frame analytics the estimates are done in kilobytes and limits can be as granular as single bytes.
Originally estimates were going to be in tenths of megabytes, because the feeling was that many analyses would use less than a megabyte and rounding up would be too wasteful. Since tenths of megabytes to not round nicely to byte values we decided to go with kilobytes instead.
However, as #1106 showed, even quite simple analyses are using ~10MB, so rounding up to the next megabyte would not be a major problem.
Rounding estimates to the nearest megabyte would avoid the excessive precision problem noted in elastic/elasticsearch#54506. It would also improve consistency with anomaly detection.
We would still have to accept limits set in units other than megabytes, as jobs that have such limits will already exist. However, we could nudge future jobs towards using whole numbers of megabytes for their memory limits by always returning estimates as whole numbers of megabytes.
Previously data frame analytics memory estimates were
rounded to the nearest kilobyte, but this results in
excessive precision for large analyses. This changes
the estimates to always be reported in whole megabytes,
rounded up from the low level estimate.
Closeselastic#1110Closeselastic/elasticsearch#54506
Previously data frame analytics memory estimates were
rounded to the nearest kilobyte, but this results in
excessive precision for large analyses. This changes
the estimates to always be reported in whole megabytes,
rounded up from the low level estimate.
Closes#1110Closeselastic/elasticsearch#54506
For anomaly detection memory limits/estimates are done in terms of whole megabytes.
For data frame analytics the estimates are done in kilobytes and limits can be as granular as single bytes.
Originally estimates were going to be in tenths of megabytes, because the feeling was that many analyses would use less than a megabyte and rounding up would be too wasteful. Since tenths of megabytes to not round nicely to byte values we decided to go with kilobytes instead.
However, as #1106 showed, even quite simple analyses are using ~10MB, so rounding up to the next megabyte would not be a major problem.
Rounding estimates to the nearest megabyte would avoid the excessive precision problem noted in elastic/elasticsearch#54506. It would also improve consistency with anomaly detection.
We would still have to accept limits set in units other than megabytes, as jobs that have such limits will already exist. However, we could nudge future jobs towards using whole numbers of megabytes for their memory limits by always returning estimates as whole numbers of megabytes.
/cc @peteharverson
The text was updated successfully, but these errors were encountered: