moreLikeThis search query and count return different results

[clintongormley]

Why does a moreLikeThis query in count return half the results that the same query does in search?

curl -XPUT 'http://127.0.0.2:9200/es_test_1/'  -d '
{}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/'  -d '
{}
'


curl -XPUT 'http://127.0.0.2:9200/_all/type_1/_mapping?ignoreConflicts=true'  -d '
{
   "allField" : {
      "store" : "yes",
      "termVector" : "with_positions_offsets"
   },
   "properties" : {
      "num" : {
         "store" : "yes",
         "type" : "integer"
      },
      "text" : {
         "store" : "yes",
         "type" : "string"
      }
   }
}
'


curl -XPUT 'http://127.0.0.2:9200/_all/type_2/_mapping?ignoreConflicts=true'  -d '
{
   "allField" : {
      "store" : "yes",
      "termVector" : "with_positions_offsets"
   },
   "properties" : {
      "num" : {
         "store" : "yes",
         "type" : "integer"
      },
      "text" : {
         "store" : "yes",
         "type" : "string"
      }
   }
}
'


curl -XGET 'http://127.0.0.2:9200/_cluster/health?timeout=2s&waitForStatus=green' 


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_1/1'  -d '
{
   "num" : 2,
   "text" : "foo"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_2/2'  -d '
{
   "num" : 3,
   "text" : "foo"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_1/3'  -d '
{
   "num" : 4,
   "text" : "foo"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_2/4'  -d '
{
   "num" : 5,
   "text" : "foo"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_1/5'  -d '
{
   "num" : 6,
   "text" : "foo bar"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_2/6'  -d '
{
   "num" : 7,
   "text" : "foo bar"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_1/7'  -d '
{
   "num" : 8,
   "text" : "foo bar"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_2/8'  -d '
{
   "num" : 9,
   "text" : "foo bar"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_1/9'  -d '
{
   "num" : 10,
   "text" : "foo bar baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_2/10'  -d '
{
   "num" : 11,
   "text" : "foo bar baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_1/11'  -d '
{
   "num" : 12,
   "text" : "foo bar baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_2/12'  -d '
{
   "num" : 13,
   "text" : "foo bar baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_1/13'  -d '
{
   "num" : 14,
   "text" : "bar baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_2/14'  -d '
{
   "num" : 15,
   "text" : "bar baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_1/15'  -d '
{
   "num" : 16,
   "text" : "bar baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_2/16'  -d '
{
   "num" : 17,
   "text" : "bar baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_1/17'  -d '
{
   "num" : 18,
   "text" : "baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_2/18'  -d '
{
   "num" : 19,
   "text" : "baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_1/19'  -d '
{
   "num" : 20,
   "text" : "baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_2/20'  -d '
{
   "num" : 21,
   "text" : "baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_1/21'  -d '
{
   "num" : 22,
   "text" : "bar"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_2/22'  -d '
{
   "num" : 23,
   "text" : "bar"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_1/23'  -d '
{
   "num" : 24,
   "text" : "bar"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_2/24'  -d '
{
   "num" : 25,
   "text" : "bar"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_1/25'  -d '
{
   "num" : 26,
   "text" : "foo baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_1/type_2/26'  -d '
{
   "num" : 27,
   "text" : "foo baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_1/27'  -d '
{
   "num" : 28,
   "text" : "foo baz"
}
'


curl -XPUT 'http://127.0.0.2:9200/es_test_2/type_2/28'  -d '
{
   "num" : 29,
   "text" : "foo baz"
}
'


curl -XGET 'http://127.0.0.2:9200/_cluster/health?timeout=2s&waitForStatus=green' 

echo "

SEARCH
"

curl -XGET 'http://127.0.0.2:9200/_all/_search?pretty=true'  -d '
{
   "query" : {
      "moreLikeThis" : {
         "minTermFrequency" : 1,
         "likeText" : "foo bar baz"
      }
   }
}
'


echo "

COUNT
"

curl -XGET 'http://127.0.0.2:9200/_all/_count?pretty=true'  -d '
{
   "moreLikeThis" : {
      "minTermFrequency" : 1,
      "likeText" : "foo bar baz"
   }
}
'

ta

clint

[kimchy]

Hey,

Can you check now, I pushed a fix that should resolve this. Count queries were being cached wrongly for this type of query (actually, the only type of query since mlt is a different beast).

Note, mlt relies on document frequencies, and they are (as the case in the terms API) not exact until deletes are expunged (documents in Lucene are not deleted, just marked as deleted until they are merged out).

For cases like mlt, this should be good enough (the query will not return a document that does not exists).

[clintongormley]

Fixed++