Deserialization issue when using Fields

[sporty81]

When doing a search the documents all come back as null objects when using "Fields" to select only certain fields. Elasticsearch returns the data correctly but the library does not deserialize the data back in the object type specified, MyObject, in this example.

        var results = client.Search<MyObject>(x => x
            .Index("logs-2014.04.16")
            .SortDescending("timestamp").SortDescending("_id")
            .Type("log")
            .QueryString("213498")
            .Fields("_id","timestamp", "message", "level")
            );

[Mpdreamz]

Which version of Elasticsearch are you using?

Prior to 1.0 elasticsearch would return fields like this:

 hits:[{
      _source: null
      fields : {
           _id: 1
          timestamp: 129308102931,
          message: ""
          level: ""
      }
 }]

NEST prior to 1.0 was trying to be clever and coerce this response into the .Documents collection which usually only looks for the _source in hits.

Elasticsearch 1.0 returns fields always as arrays except for internal fields so it now looks like this:

 hits:[{
      _source: null
      fields : {
           _id: 1
          timestamp: 129308102931,
          message: [""]
          level: [""]
      }
 }]

Which means NEST 1.0 can no longer coerce message to be a string.

This means that you now have to loop over results.Hits and for each hit fish out the values using something like this:

 hit.Fields.FieldValue<MyObject, string>(p => p.Message).FirstOrDefault();

 hit.Fields.FieldValue<string[]>("lang").FirstOrDefault();

The Search() method also allows for specifying a separate DTO to map the _source fields:

client.Search<MyObject, MyObjectReturnedSource>()

Perhaps in this case NEST can try the coercion of fields again though MyObectReturnedSource should look like this:

class MyObjectReturnedSource {
       public IList<string> Message { get; set; }
}

If anyone has thoughts on how to solve this in a cleaner manner please share!

[sporty81]

I was using the latest 1.0 code. So I guess in ES it is possible to have a document with the same field more than once, so it can return all the values. Is that right? Although that doesn't sound like a very normal use case. Seems like the best solution when trying to use this automatic object mapping would be to grab the first value in each array. Having it return the first value in the array would be much better than it returning null objects.

If there is no way to fix this it seems like you should consider disabling the Fields function in NEST since it would never work the way people expect it to? I simply expected it to return fewer fields in my same object. Having to create a different object with arrays for each property is not very user intuitive but I guess as long as you make it very clear in the online doc examples it would be OK.

[Mpdreamz]

A single field can have multiple values, see the full rationale for returning arrays here:

elastic/elasticsearch#4542

Elasticsearch returns a property called hits with some statistics about all the hits and inside of this another property call hits which is an array of metadata, source/fields.

ISearchResults has a .HitsMetaData property where all of this is deserialized into.

.Documents is basically a helper to flatten .Hits.Select(h=>h.Source) and .Hits calls into .HitsMetaData.Hits

Now as soon as you specify fields elasticsearch no longer returns _source for each hit but a keyvalue dictionary property fields.

i.e

fields : {
    "mydeeply.nested.field" : ["values"]
}

This is completely unparseable into the DTO you specify even if it returned a single value instead of an array. NEST 0.12 also would fail for complex field selection like this.

.Documents should not return a list of null objects though and perhaps the xmldocs for the .Documents property should explain that .Documents is not available when the search request specified fields.

An alternative to fields in Elasticsearch 1.0 is source extraction:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-source-filtering.html#search-request-source-filtering

Which NEST 1.0.0-beta1 supports and does allow you to reuse your DTO.

[Mpdreamz]

@sporty81 FYI the API surrounding fields will be improved in the next version:

#619