Search preference custom string is ignored when request is sent to a non-client node

[rchampeimont]

Elasticsearch version (bin/elasticsearch --version): 6.6.1

Plugins installed:

• analysis-icu
• repository-s3
• ltr-1.1.0-es6.6.1.zip (https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/)

JVM version (java -version):

openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)

OS version (uname -a if on a Unix-like system):

Linux ip-172-31-2-42.eu-central-1.compute.internal 4.14.121-109.96.amzn2.x86_64 #1 SMP Wed May 22 16:54:10 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
We have developed a legal search engine based on Elasticsearch and we are using the "preference" (https://www.elastic.co/guide/en/elasticsearch/reference/6.6/search-request-preference.html) parameter to ensure that the order of the results does not change:

• when scrolling (which we implemented by running the same query with different from+size)
• when clicking on a search result, we display "result X / Y" and show arrows to go to the previous/next result in the associated search

Our cluster is composed of 2 zones (called A and B), each zone having:

• 1 client node
• 7 data nodes (of which some can also be master nodes)
  The Elasticsearch cluster is aware of the zones (which correspond to AWS zones).
  Our index has 12 primary shards and 1 replica, so each shard has exactly one copy in zone A and one copy in zone B (to be able to survive a complete zone outage).

We use a custom string as a preference (it is a basically a uuid generated for each unique user search, like 5d2a1ee677059bc926f7a39492f11a47).

Steps to reproduce:

When sending the query to client nodes, we get the expected behavior: The preference custom string is taken into account. Changing it yields different results. On the other hand, sending the request to the client node in zone A or B yields the same results if the preference custom string is the same. This is the expected behavior AFAIK.

However, when sending the request to a DATA node, we get an unexpected behavior which seems undocumented in https://www.elastic.co/guide/en/elasticsearch/reference/6.6/search-request-preference.html. Changing the preference custom string has no effect on the result. On the other hand, the result is different if the node to which we send the query is in zone A or B (but we get the same result for any node in zone A, and for any node in zone B).

Hypothesis: It seems that Elasticsearch is ignoring the preference custom string if the node to which we send the request is a data node, in which cases it chooses to use shards that belong the same zone, ignoring the preference custom string.

Note that this happens only with a custom string. When using "_primary" for instance, the behavior is the same regardless of the type of node to which we send the query.

This was an issue for us because we send request to 2 nodes (one in zone A and one in zone B) from our application (using the built-in Elasticsearch Node.js library mechanism to specify several hosts), and the results were inconsistent depending on the node to which the request was sent (probably a round robin?). To solve the issue, we added 2 client nodes to the cluster and send all our requests to these client nodes, and the preference custom string is now properly taken into account. However, the fact that custom string in preference only works when using CLIENT nodes in the cluster does not seem to be a documented behavior in https://www.elastic.co/guide/en/elasticsearch/reference/6.6/search-request-preference.html

So it seems to us that either this is a bug, or this a something missing in the documentation.

I am sorry that I don't have a simple reproducible scenario, but I don't know how to make an index for which I can immediately see differences between shards. For us it took several weeks for this issue to be revealed, as a "new" index gives the same results between all shards anyway.

Provide logs (if relevant):

[DaveCTurner]

Thanks for the report @raphaelchampeimont. This does indeed sound like surprising behaviour that disagrees with the documentation.

You can use the profile API to determine exactly which shard copies are used by a particular search. I think this might help you come up with a way to reproduce this on a smaller index without needing to rely on discrepancies in the results. The expectation is that the same preference argument results in hitting the same shard copies each time.

Do all the nodes have the same elasticsearch.yml config file apart from node.data and/or node.master? If not, can you share the exact differences in their configurations?

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[rchampeimont]

Do all the nodes have the same elasticsearch.yml config file apart from node.data and/or node.master? If not, can you share the exact differences in their configurations?

Almost. The amount of memory allocated is different. Here is the ansible configuration we use:

- hosts: all
  tasks:
    - name: Deploy SSH authorized public keys
      authorized_key:
        user: ec2-user
        key: "{{ lookup('file', 'public_keys') }}"

- name: ElasticSearch data-nodes nodes
  hosts: data-nodes
  roles:
    # never change the es_instance_name !
    - {
        role: elasticsearch,
        es_instance_name: "{{ node_name }}",
        es_config:
          {
            node.name: "{{ node_name }}",
            cluster.name: "es_6_cluster_prod",
            discovery.zen.ping.unicast.hosts: "{{ master_nodes_private }}",
            discovery.zen.minimum_master_nodes: 2,
            http.port: 9200,
            network.host: 0.0.0.0,
            node.data: true,
            node.ingest: false,
            node.master: "{{ master | bool }}",
            bootstrap.memory_lock: true,
            action.destructive_requires_name: true,
            thread_pool.bulk.queue_size: 200,
            node.attr.zone: "{{ zone }}",
            node.attr.hardware_type: "{{ hardware_type }}",
            cluster.routing.allocation.awareness.attributes: zone,
          },
        es_heap_size: 31g,
      }
  vars:
    es_major_version: 6.x
    es_version: 6.6.1
    es_plugins:
      - plugin: analysis-icu
      - plugin: repository-s3
      - plugin: file:///home/ec2-user/ltr-1.1.0-es6.6.1.zip
    es_restart_on_change: false
    es_action_auto_create_index: true
    es_xpack_features: ["alerting", "monitoring", "graph", "ml"]

- name: ElasticSearch client nodes
  hosts: client-nodes
  roles:
    # never change the es_instance_name !
    - {
        role: elasticsearch,
        es_instance_name: "{{ node_name }}",
        es_config:
          {
            node.name: "{{ node_name }}",
            cluster.name: "es_6_cluster_prod",
            discovery.zen.ping.unicast.hosts: "{{ master_nodes_private }}",
            discovery.zen.minimum_master_nodes: 2,
            http.port: 9200,
            network.host: 0.0.0.0,
            node.data: false,
            node.ingest: false,
            node.master: false,
            bootstrap.memory_lock: true,
            action.destructive_requires_name: true,
            thread_pool.bulk.queue_size: 200,
          },
        es_heap_size: 6g,
      }
  vars:
    es_major_version: 6.x
    es_version: 6.6.1
    es_plugins:
      - plugin: analysis-icu
      - plugin: repository-s3
      - plugin: file:///home/ec2-user/ltr-1.1.0-es6.6.1.zip
    es_restart_on_change: false
    es_action_auto_create_index: true
    es_xpack_features: ["alerting", "monitoring", "graph", "ml"]

- hosts: all
  tasks:
    - name: Copy .zshrc to ansible user home
      become: yes
      template: src=zshrc dest=/home/{{ ansible_user }}/.zshrc mode=0755 force=yes

    - name: exclude elasticsearch package from rpm
      become: yes
      lineinfile:
        path: /etc/yum.conf
        state: present
        line: "exclude=elasticsearch"

[rchampeimont]

You can use the profile API to determine exactly which shard copies are used by a particular search. I think this might help you come up with a way to reproduce this on a smaller index without needing to rely on discrepancies in the results. The expectation is that the same preference argument results in hitting the same shard copies each time.

I will try to reproduce that on a minimalist cluster.

[DaveCTurner]

Thanks, I see the issue. Coordinating nodes behave differently depending on whether cluster.routing.allocation.awareness.attributes is set locally or not:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/routing/OperationRouting.java

Lines 245 to 249 in 99ac2c0

	if (awarenessAttributes.isEmpty()) {
	return indexShard.activeInitializingShardsIt(routingHash);
	} else {
	return indexShard.preferAttributesActiveInitializingShardsIt(awarenessAttributes, nodes, routingHash);
	}

The search respects the preference value either way, but nodes in different zones will indeed route searches differently. This is fixed in 8.0.0 by #45735 (allocation awareness no longer has any bearing on search routing there) and optionally fixed in 7.5.0 by #46375 (if you opt-in to the 8.0.0 behaviour). I think this is worth documenting too.

As a workaround in earlier versions, if you always use coordinating nodes that do not have cluster.routing.allocation.awareness.attributes set, and you do not set cluster.routing.allocation.awareness.attributes through the cluster settings API, then you will see searches being routed consistently.

[elasticmachine]

Pinging @elastic/es-docs (>docs)

[snutu]

I opened #74856 to backport the fix that was applied in 7.x, but I was told that adding new cluster settings is not something that's typically accepted.

Unfortunately, our setup does not have client nodes and adding them just for this workaround would not be feasible.

We are currently using the _primary_first preference string in ES 6.8 as a workaround for the scroll requests issue, but we'll have to stop using it as we're planning to upgrade to ES7. Now we are in a position where the upgrade to 7.X takes away the _primary_first option and does not have an alternative using the same setup, without going through a period where the scroll requests return inconsistent results, depending on the nodes executing them, until the upgrade finishes.

Given that it turned out there is no ES6 replacement for the _primary_first removal, could you reconsider backporting the cluster setting? A different workaround that doesn't involve cluster setup changes would also be appreciated.

[rchampeimont]

We are currently using the _primary_first preference string in ES 6.8 as a workaround for the scroll requests issue, but we'll have to stop using it as we're planning to upgrade to ES7. Now we are in a position where the upgrade to 7.X takes away the _primary_first option and does not have an alternative using the same setup, without going through a period where the scroll requests return inconsistent results, depending on the nodes executing them, until the upgrade finishes.

Unless I misunderstood something, #46375 is merged in ES 7.5 so you could upgrade to ES7, set es.routing.search_ignore_awareness_attributes to true, and use a custom preference string based on a unique ID for each search like I explained in my first post (or use any ID which remains identical for a given search but still allows to balanced load on average, like a user ID, session ID, IP address...).

And if later you upgrade to ES8, you could get rid of es.routing.search_ignore_awareness_attributes since it becomes the default behavior.

[snutu]

The issue is that we cannot upgrade to ES7 while using the _primary_first, as it was removed from ES7. If we switch to a string preference before the upgrade, then the scroll searches will be inconsistent. I'm looking for something that does work in both ES6 and ES7, as the solution for this deprecation was intended to be before discovering the current issue.

To add some more nuance, we are also using cross cluster searches and there will be a period of time in which both ES6 and ES7 versions will be used at the same time.

[rchampeimont]

If you use zone-aware shard allocation and have exactly as many zones as shard copies (for instance you have 2 zones and have replica=1 for all your indices), you could use _prefer_nodes, which exists in both ES6 and ES7, and list all nodes from one zone. This would force to use the shard copies in this zone, while keeping failover to the other shards if a prefered shard fails. It would however put the all the load on half of your cluster until you finish the migration to ES7 so it might not be OK for you from a performance point of view.

If you don't use zone-based shard allocation, I would think the best option is to transform some of your nodes (2 for redundancy I would advise) into client nodes and send all requests to their IP addresses. If you have never used client nodes, don't worry, the CPU load on them will be low (the hard work is performed by the data nodes) so you don't need many of them like with data nodes. To give you an order of magnitude, on our cluster client nodes represent < 10% of the number of CPU cores in the cluster.

Of course a last and rather obvious option would be to set replica = 0 on all your indices, deploy code in production that uses a preference string, then upgrade to ES7 and set back replica = 1 (or more). But unlike the other solutions above, this means temporarily losing redundancy, so this is probably not acceptable for you.

[jtibshirani]

I updated the documentation in 6.8 and 7.x to warn about this issue (#83818). I'm going to close this out because we don't have further work planned. Thank you again for reporting this and for your flexibility in exploring workarounds.