geo_shapes stopped indexing in 6.4.1

[Destroy666x]

Elasticsearch version (bin/elasticsearch --version): 6.4.1

Plugins installed: []

JVM version (java -version): official Docker image

OS version (uname -a if on a Unix-like system): official Docker image

Description of the problem including expected versus actual behavior:

After upgrading from 6.3.2 to 6.4.1, many geo_shapes stopped importing.

Steps to reproduce:

1. Index has a field defined like this (FOSElastica bundle YML configuration):

    shape:
        type: 'geo_shape'
        tree: 'quadtree'
        precision: '10m'
        distance_error_pct: 0.001

2. Indexing many shapes in 6.4.1 that worked in 6.3.2 results in caused failed to parse error.
3. Adding ignore_malformed: true doesn't seem to help.

It looks like there was some sort of undocumented BC break regardless of the shapes being fully proper. If you want I can provide you an example of a shape.

[tlrx]

Thanks for reporting.

It looks like there was some sort of undocumented BC break regardless of the shapes being fully proper. If you want I can provide you an example of a shape.

Yes, that would help if you could provide an example of a shape that is correctly indexed in 6.3.2 and failed to be indexed in 6.4.1. We'll need the mappings too.

[elasticmachine]

Pinging @elastic/es-search-aggs

[Destroy666x]

We'll need the mappings too.

Which ones? I provided the ones for the affected field, do you need any other?

[tlrx]

@Destroy666x Thanks! If you can provide a full example with all the document fields mappings that's great. If you can reproduce with one field only then we only need the field mapping :)

[Destroy666x]

Question @tlrx - is there an easy way to check the bulk request the library does in some sort of Elastic logs? Because the library logger for some reason skips it, even if I simplify the request quite a lot.

Maybe these details will help in the meanwhile:

1. There's e.g. this shape stored as a multidimensional array https://pastebin.com/ULfUwAZJ
2. The code turns it into

'type' => 'geometrycollection',
'geometries' => []

where geometries is an array of polygons (3rd level) with flipped coordinate order (which I guess could be replaced with orientation change) and lat/lon order (4th level) before bulk insertion.
3. I can reproduce it with only this field mapped.
4. It also worked in earlier versions, e.g. 5.x.

[imotov]

@Destroy666x I tried to reproduce what you described based on the information that you have provided on 6.3.1 and 6.4.1 and got quite an opposite result. The following script https://gist.github.com/imotov/64dcfc84829141d54d4a244573902449 that uses both [] as geometies as well as your shape doesn't return any errors on 6.4.1 but fails with an error message on 6.3.1. So, in this test 6.4.1 seems to be more robust than 6.3.1 in ignoring malformed objects. Could you modify this example to reproduce the behaviour that you describe? Thanks!

[Destroy666x]

Well, you didn't reverse the order of vertices and lat/lon and didn't do it as a polygon array. I'll try your way of sending the data and then will submit a more relevant gist.

[Destroy666x]

@imotov I tried your PUTs directly in Kibana dev tools and for the 2nd I got shape must be an object consisting of type and coordinates if there is no ignore_malformed: true. Same goes for proper reverse lat/lon + vertices and:

    "type": "geometrycollection",
    "geometries": [
      {
        "type": "polygon",
        "coordinates": [
            ...
        ]
      }
    ]

format. Which on the other hand throws geo coordinates must be numbers error.

So I guess the Symfony library or Elastica for some reason ignores the setting. But if that's the case why is the setting suddenly required (BC break still) and what's wrong with the shape (the reverse lat/lon and vertices one that is)? The problem still exists in 6.4.2 BTW.

[Destroy666x]

Here's the shape with proper default Elasticsearch order: https://pastebin.com/c8xHeXx0

And here's a screen from 6.3.2 with the exact request https://pastebin.com/N0JFt4u2 and no ignore_malformed: true:

Your array-only formatted version doesn't work on earlier versions indeed, you get object mapping for [shape.geometries] tried to parse field [null] as object, but found a concrete value.

[imotov]

So I guess the Symfony library or Elastica for some reason ignores the setting.

The setting is only applied on the server side and not on the client side. So, I think the shape that your client app is sending is not what you think/reported it is sending. Perhaps, you can sniff the network traffic to see what is actually getting sent and try to reproduce it using Kibana. That would really help, because at the moment I am not actually sure what the issue is and I still cannot reproduce it.

[Destroy666x]

The setting is only applied on the server side and not on the client side

Yes, but you can configure the index mappings that are sent to the server.

That would really help, because at the moment I am not actually sure what the issue is and I still cannot reproduce it.

Did you try the PUT from the 2nd link in both 6.3.2 and 6.4.2?

So, I think the shape that your client app is sending is not what you think/reported it is sending. Perhaps, you can sniff the network traffic to see what is actually getting sent and try to reproduce it using Kibana

Ok, I'll try that when I'll find more time.

[imotov]

Did you try the PUT from the 2nd link in both 6.3.2 and 6.4.2?

@Destroy666x the 2nd link from where and with which mapping? Could you post a complete reproduction similar to what I posted in https://gist.github.com/imotov/64dcfc84829141d54d4a244573902449 ?

[Destroy666x]

From this comment: #34047 (comment) The rest of your gist is pretty much the same, apart from different index, type name and no ignore_malformed: true.

[imotov]

Thanks! I can reproduce it now. It looks like another case similar to #31428. A smaller reproduction:

DELETE test

PUT test
{
  "mappings": {
    "doc": {
      "properties": {
        "shape": {
          "ignore_malformed": true,
          "type": "geo_shape"
        }
      }
    }
  }
}

PUT /test/doc/9999
{
  "shape": {
    "type": "geometrycollection",
    "geometries": [
      {
        "type": "polygon",
        "coordinates": [
          ["46.6022226498514", "24.7237442867977"],
          ["46.6031857243798", "24.722968774929"],
          ["46.6006675445073", "24.7250001432471"],
          ["46.6012343057486", "24.7245313491405"],
          ["46.6020897839358", "24.7238517986582"],
          ["46.6021637530119", "24.7237921384342"],
          ["46.6022226498514", "24.7237442867977"]
        ]
      }
    ]
  }
}

[brccabral]

Just for reference, I solved my issue by using this solution
https://stackoverflow.com/questions/65031965/how-to-remove-quotation-marks-in-geo-coordinates-on-logstash-conf-file

filter{
    mutate {
      add_field => {
        "[location][type]" => "point"
      }
    }
	ruby{
		code => '
			event.set("[location][coordinates]", [event.get("latitude"), event.get("longitude")])
		'
	}
}