Make sure swapping is disabled for Elasticsearch

[lukas-vlcek]

Especially for the performance reasons we need to make sure Elasticsearch is not subject of swapping.

We need to focus on two levels here:

• At the OS/Elasticsearch level. This means we can use techniques recommended here to disable swapping at the OS level
• At the Docker level using docker run [OPTIONS]. See
  • https://docs.docker.com/v1.10/engine/reference/run/#runtime-constraints-on-resources
  • https://docs.docker.com/v1.10/engine/reference/run/#swappiness-constraint

[lukas-vlcek]

Note, at Elasticsearch level we can use Cluster Nodes Stats API to learn if it is swapping

[lukas-vlcek]

Relevant OpenShift doc might be here: Disabling Swap Memory.

[sosiouxme]

OpenShift/Kubernetes don't provide a way to specify the container swappiness and I wouldn't expect that to change soon. We can certainly advise disabling swap on nodes designated for ES to run on.

We could provide a diagnostic that queries ES's stats API to warn if it is swapping.

[ewolinetz]

Couldn't we add the bootstrap.mlockall: true setting to the ES config?

https://www.elastic.co/guide/en/elasticsearch/reference/2.3/setup-configuration.html

[lukas-vlcek]

@ewolinetz I think we can but as discussed in official ES doc, this option may not be applied for various reasons, however, if we can make it work in our case then I am all for it!

[portante]

@lukas-vlcek, we run with this in our production environment. Works great.

What are you referring to when you mention official ES doc? Thanks!

[lukas-vlcek]

@portante Check the mlockall section in Memory Settings. There are mentioned some cases where the bootstrap.mlockall: true may not be applied (you need to check ES log to see it is was applied or not).

[portante]

@lukas-vlcek, okay good, that is all about runtime setup, which it seems we can take care of.

[lukas-vlcek]

@portante yes, that is why I said above:

[...] if we can make it work in our case then I am all for it!

But if we can be hit by container level swapping then I am not sure if we can learn about that do anything about it.

[lukas-vlcek]

Does this merged PR openshift/openshift-ansible#3884 disables swap for good or only temporarily?

@jcantrill I can only see that this PR disables swap (swapoff --all) if some conditions are met (not sure if this means swap can stay enabled in other situations - what kind of specific situations?). Is there any code in Openshift or Ansible configuration that enables swapping later (for example after node is upgraded/initialized)?

More on that, I think we should go ahead and explicitly try to disable swap in Elasticsearch configuration.

Notice, starting with ES 5.x the memory lock is applied:

The memory lock check verifies that if the bootstrap.memory_lock setting is enabled, that the JVM was successfully able to lock the heap.

If it fails to pass this check ES node does not start.

[portante]

@lukas-vlcek, why do we need to worry about swapping for ES if ES can ask the JVM to lock the HEAP in memory to prevent it from swapping?

[lukas-vlcek]

@portante we should be using memory lock. No question about this.

The only think I am not sure about is how the container kernel host swapping fits in and if we need to worry about this at all. And what we can do about it to make sure it does not kick in. Or in worst case how we can learn that it did kick in...

On the other hand that would be probably addressed at different level. So I will open PR to close this ticket (if there haven't been created one already).

[lukas-vlcek]

Also, shouldn't we make sure Xms and Xmx JVM options are not different? Currently, each can be different.

See recommendation here:

It is recommended to set the min and max memory to the same value, and enable mlockall.

[jcantrill]

Setting min and max heep to be the same is good practice in general. It has just been a minor detail that no one has addressed.

…

On Wed, Apr 19, 2017 at 9:15 AM, Lukáš Vlček ***@***.***> wrote: Also, shouldn't we make sure Xms and Xmx JVM options are not different? Currently, each can be different <https://github.com/openshift/origin-aggregated-logging/blob/e2314101a1c53226a4d2563efa7f942807f31537/elasticsearch/run.sh#L66> . See recommendation here <https://www.elastic.co/guide/en/elasticsearch/reference/2.4/setup-configuration.html> : It is recommended to set the min and max memory to the same value, and enable mlockall. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#206 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEVnOOj_Ly8uDwTMhyK_iLlK4HEKowKcks5rxgj9gaJpZM4JXNb4> .

-- -- Jeff Cantrill Senior Software Engineer, Red Hat Engineering OpenShift Integration Services Red Hat, Inc. *Office*: 703-748-4420 | 866-546-8970 ext. 8162420 [email protected] http://www.redhat.com

[ewolinetz]

@lukas-vlcek ideally yes, however that might make it more difficult for ES to start in the scenario where the container is placed on a node where it isn't able to receive its requested/maximum JVM memory

[jcantrill]

IMO, that is fine. It means the node is not sized as recommended which would avoid issues later. Users can always adjust the setting down to a value which will allow it to fit on the node.

…

On Wed, Apr 19, 2017 at 9:26 AM, Eric Wolinetz ***@***.***> wrote: @lukas-vlcek <https://github.com/lukas-vlcek> ideally yes, however that might make it more difficult for ES to start in the scenario where the container is placed on a node where it isn't able to receive its requested/maximum JVM memory — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#206 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEVnOHBtpi_WWwX9u4WTL5sR1cpj_wfuks5rxguSgaJpZM4JXNb4> .

-- -- Jeff Cantrill Senior Software Engineer, Red Hat Engineering OpenShift Integration Services Red Hat, Inc. *Office*: 703-748-4420 | 866-546-8970 ext. 8162420 [email protected] http://www.redhat.com

[lukas-vlcek]

Well, the thing is that if we use memory lock and Xms < Xmx at JVM startup then we can end up with some memory pages being swapped later anyway, see elastic/elasticsearch#17605 (comment)

[sosiouxme]

It will mean users who just want to try it out or run tests against it on typical (under-resourced) test nodes will have to know up front about the default memory requirement and how to adjust it. That doesn't mean demanding all the memory up front is a bad idea, just something to be aware of.

[lukas-vlcek]

@sosiouxme @ewolinetz @jcantrill let's move this discussion to #383

[lukas-vlcek]

#382 has been already closed, so I think we can also close this?

[richm]

AFAIK this will be handled at the K8s/OpenShift level by disabling swap for all containers.