Support unicode collations

[clintongormley]

Hiya

Below is a test case which demonstrates that ES is not using a proper Unicode collation to sort. "João" should be sorted before "Jordania":

curl -XPUT 'http://127.0.0.2:9200/test_1/'  -d '
{}
'
# {
#    "ok" : true,
#    "acknowledged" : true
# }


curl -XPUT 'http://127.0.0.2:9200/test_1/foo/_mapping'  -d '
{
   "properties" : {
      "text" : {
         "type" : "string",
         "analyzer" : "keyword"
      }
   }
}
'
# {
#    "ok" : true,
#    "acknowledged" : true
# }


curl -XPUT 'http://127.0.0.2:9200/test_1/foo/1'  -d '
{
   "text" : "Jordania"
}
'
# {
#    "ok" : true,
#    "_index" : "test_1",
#    "_id" : "1",
#    "_type" : "foo"
# }


curl -XPUT 'http://127.0.0.2:9200/test_1/foo/2'  -d '
{
   "text" : "João"
}
'
# {
#    "ok" : true,
#    "_index" : "test_1",
#    "_id" : "2",
#    "_type" : "foo"
# }


curl -XGET 'http://127.0.0.2:9200/test_1/_search'  -d '
{
   "sort" : [
      "text"
   ],
   "query" : {
      "match_all" : {}
   }
}
'
# {
#    "hits" : {
#       "hits" : [
#          {
#             "_source" : {
#                "text" : "Jordania"
#             },
#             "_index" : "test_1",
#             "_id" : "1",
#             "_type" : "foo"
#          },
#          {
#             "_source" : {
#                "text" : "João"
#             },
#             "_index" : "test_1",
#             "_id" : "2",
#             "_type" : "foo"
#          }
#       ],
#       "total" : 2
#    },
#    "_shards" : {
#       "failed" : 0,
#       "successful" : 5,
#       "total" : 5
#    }
# }

http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/collation/package-summary.html

thanks

Clint

[kimchy]

hey, with the icu plugin, I think its ok?, closing it.