From 17dadfba57fc663e8805a2510b8503e8ee6f48fc Mon Sep 17 00:00:00 2001 From: David Roberts Date: Tue, 19 Jan 2021 14:25:12 +0000 Subject: [PATCH 1/5] Add known issue for zero memory on Debian 8 --- .../ml-troubleshooting.asciidoc | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc b/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc index 1a9159f17..44b358ee7 100644 --- a/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc +++ b/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc @@ -188,3 +188,40 @@ POST _ml/set_upgrade_mode?enabled=false&timeout=10m DELETE temp_ml_config -------------------------------------------------- // TEST[skip:TBD] + +[discrete] +[[ml-debian8-memory]] +== Suboptimal job assignment on Debian 8 + +Where possible {ml} jobs are assigned to nodes based on the memory requirement +of the job and the memory available on the node. This breaks down when the +amount of memory on a node cannot be accurately determined, and then jobs are +assigned simply by balancing numbers of jobs per {ml} node. This could lead to +a situation where all the big jobs are on one node and all the small jobs on +another. + +Elasticsearch will fail to determine the amount of memory on a machine that is +running Debian 8 with the default Cgroups setup, and certain updates of Java +versions earlier than Java 15. For example, Java 8u271 is known to be affected +while Java 8u272 is not. Java 15 was fixed from its initial release. + +If you are running Elasticsearch on Debian 8 with an old version of Java and +have not already modified the Cgroups setup then you should do one of the +following: + +1. Upgrade Java to version 15 +2. Upgrade to the latest Java update for the version of Java you are running +3. Enable the "memory" Cgroup by editing `/etc/default/grub` and adding: + +``` +GRUB_CMDLINE_LINUX_DEFAULT="quiet cgroup_enable=memory swapaccount=1" +``` + +Then update your GRUB configuration by running: + +``` +sudo update-grub +``` + +And finally reboot the machine. + From 3d538b611c4bd765108e63a2246a87d82f7ac524 Mon Sep 17 00:00:00 2001 From: David Roberts Date: Tue, 26 Jan 2021 12:01:12 +0000 Subject: [PATCH 2/5] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: István Zoltán Szabó --- .../ml-troubleshooting.asciidoc | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc b/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc index 44b358ee7..c39637d39 100644 --- a/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc +++ b/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc @@ -194,23 +194,23 @@ DELETE temp_ml_config == Suboptimal job assignment on Debian 8 Where possible {ml} jobs are assigned to nodes based on the memory requirement -of the job and the memory available on the node. This breaks down when the +of the job and the memory available on the node. However, in certain cases, the amount of memory on a node cannot be accurately determined, and then jobs are -assigned simply by balancing numbers of jobs per {ml} node. This could lead to -a situation where all the big jobs are on one node and all the small jobs on +assigned by balancing numbers of jobs per {ml} node. It may lead to +a situation where all the jobs with high memory requirements are on one node and the less memory-intensive jobs on another. -Elasticsearch will fail to determine the amount of memory on a machine that is +{es} fails to determine the amount of memory on a machine that is running Debian 8 with the default Cgroups setup, and certain updates of Java versions earlier than Java 15. For example, Java 8u271 is known to be affected -while Java 8u272 is not. Java 15 was fixed from its initial release. +while Java 8u272 is not. Java 15 was fixed from its initial release. -If you are running Elasticsearch on Debian 8 with an old version of Java and -have not already modified the Cgroups setup then you should do one of the +If you are running {es} on Debian 8 with an old version of Java and +have not already modified the Cgroups setup then it is recommended to do one of the following: -1. Upgrade Java to version 15 -2. Upgrade to the latest Java update for the version of Java you are running +1. Upgrade Java to version 15. +2. Upgrade to the latest Java update for the version of Java you are running. 3. Enable the "memory" Cgroup by editing `/etc/default/grub` and adding: ``` @@ -224,4 +224,3 @@ sudo update-grub ``` And finally reboot the machine. - From 4aa1e1a8ade5fac757e7a8cad0166402eaafb317 Mon Sep 17 00:00:00 2001 From: David Roberts Date: Tue, 26 Jan 2021 17:02:26 +0000 Subject: [PATCH 3/5] Apply suggestions from code review Co-authored-by: Lisa Cawley --- .../stack/ml/anomaly-detection/ml-troubleshooting.asciidoc | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc b/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc index c39637d39..e7351d73a 100644 --- a/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc +++ b/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc @@ -193,7 +193,7 @@ DELETE temp_ml_config [[ml-debian8-memory]] == Suboptimal job assignment on Debian 8 -Where possible {ml} jobs are assigned to nodes based on the memory requirement +Where possible, {ml} jobs are assigned to nodes based on the memory requirement of the job and the memory available on the node. However, in certain cases, the amount of memory on a node cannot be accurately determined, and then jobs are assigned by balancing numbers of jobs per {ml} node. It may lead to @@ -212,7 +212,8 @@ following: 1. Upgrade Java to version 15. 2. Upgrade to the latest Java update for the version of Java you are running. 3. Enable the "memory" Cgroup by editing `/etc/default/grub` and adding: - ++ +-- ``` GRUB_CMDLINE_LINUX_DEFAULT="quiet cgroup_enable=memory swapaccount=1" ``` @@ -223,4 +224,4 @@ Then update your GRUB configuration by running: sudo update-grub ``` -And finally reboot the machine. +4. Reboot the machine. From 32a0d746198cf790079d742eb0a7be9fd3d9f82b Mon Sep 17 00:00:00 2001 From: David Roberts Date: Tue, 26 Jan 2021 17:07:14 +0000 Subject: [PATCH 4/5] More edits --- .../ml-troubleshooting.asciidoc | 34 +++++++++++-------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc b/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc index e7351d73a..bd389db82 100644 --- a/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc +++ b/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc @@ -196,17 +196,18 @@ DELETE temp_ml_config Where possible, {ml} jobs are assigned to nodes based on the memory requirement of the job and the memory available on the node. However, in certain cases, the amount of memory on a node cannot be accurately determined, and then jobs are -assigned by balancing numbers of jobs per {ml} node. It may lead to -a situation where all the jobs with high memory requirements are on one node and the less memory-intensive jobs on -another. - -{es} fails to determine the amount of memory on a machine that is -running Debian 8 with the default Cgroups setup, and certain updates of Java -versions earlier than Java 15. For example, Java 8u271 is known to be affected -while Java 8u272 is not. Java 15 was fixed from its initial release. - -If you are running {es} on Debian 8 with an old version of Java and -have not already modified the Cgroups setup then it is recommended to do one of the +assigned by balancing numbers of jobs per {ml} node. It may lead to a situation +where all the jobs with high memory requirements are on one node and the less +memory-intensive jobs on another. + +One particular case of this problem is that {es} fails to determine the amount +of memory on a machine that is running Debian 8 with the default Cgroups setup, +and certain updates of Java versions earlier than Java 15. For example, Java +8u271 is known to be affected while Java 8u272 is not. Java 15 was fixed from +its initial release. + +If you are running {es} on Debian 8 with an old version of Java and have not +already modified the Cgroups setup then it is recommended to do one of the following: 1. Upgrade Java to version 15. @@ -217,11 +218,14 @@ following: ``` GRUB_CMDLINE_LINUX_DEFAULT="quiet cgroup_enable=memory swapaccount=1" ``` - +-- ++ Then update your GRUB configuration by running: - ++ +-- ``` sudo update-grub ``` - -4. Reboot the machine. +-- ++ +Reboot the machine. From aaa726e5d321e29ebcf26f41ecc81f76567996bf Mon Sep 17 00:00:00 2001 From: lcawl Date: Tue, 26 Jan 2021 10:40:54 -0800 Subject: [PATCH 5/5] [DOCS] Change list type --- .../ml-troubleshooting.asciidoc | 35 ++++++++----------- 1 file changed, 15 insertions(+), 20 deletions(-) diff --git a/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc b/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc index bd389db82..d3f66e9cb 100644 --- a/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc +++ b/docs/en/stack/ml/anomaly-detection/ml-troubleshooting.asciidoc @@ -195,14 +195,14 @@ DELETE temp_ml_config Where possible, {ml} jobs are assigned to nodes based on the memory requirement of the job and the memory available on the node. However, in certain cases, the -amount of memory on a node cannot be accurately determined, and then jobs are -assigned by balancing numbers of jobs per {ml} node. It may lead to a situation -where all the jobs with high memory requirements are on one node and the less +amount of memory on a node cannot be accurately determined and jobs are assigned +by balancing the number of jobs per {ml} node. It may lead to a situation where +all the jobs with high memory requirements are on one node and the less memory-intensive jobs on another. One particular case of this problem is that {es} fails to determine the amount -of memory on a machine that is running Debian 8 with the default Cgroups setup, -and certain updates of Java versions earlier than Java 15. For example, Java +of memory on a machine that is running Debian 8 with the default Cgroups setup +and certain updates of Java versions earlier than Java 15. For example, Java 8u271 is known to be affected while Java 8u272 is not. Java 15 was fixed from its initial release. @@ -210,22 +210,17 @@ If you are running {es} on Debian 8 with an old version of Java and have not already modified the Cgroups setup then it is recommended to do one of the following: -1. Upgrade Java to version 15. -2. Upgrade to the latest Java update for the version of Java you are running. -3. Enable the "memory" Cgroup by editing `/etc/default/grub` and adding: +* Upgrade Java to version 15. +* Upgrade to the latest Java update for the version of Java you are running. +* Enable the "memory" Cgroup by editing `/etc/default/grub` and adding: + -- -``` +[source,sh] +-------------------------------------------------- GRUB_CMDLINE_LINUX_DEFAULT="quiet cgroup_enable=memory swapaccount=1" -``` --- -+ -Then update your GRUB configuration by running: -+ --- -``` -sudo update-grub -``` +-------------------------------------------------- + +Update your GRUB configuration by running `sudo update-grub`, then reboot the +machine. + -- -+ -Reboot the machine.