Warn FilterPath transforms the response and can break response parsing

[muzizongheng]

Elasticsearch version: 6.2.3

• Problem step description:

• index's object json like:

{
  "_index": "metric_ctdms",
  "_type": "ctdms",
  "_id": "lO_-kmQB1uD8GkS4gI32",
  "_version": 1,
  "_score": 2,
  "_source": {
    "type": "CTMonitorPa5min",
    "description": "CT monitor data",
    "guid": "99dc0c12-bf9d-46df-ae41-a98d7ba92f3e",
    "serialNumber": "300001",
    "productNumber": "88000032",
    "systemID": "101014010001 ",
    "sourceName": "",
    "level": "0",
    "priority": "1",
    "operationDateTime": "2018-07-13T17:02:07.5639857+08:00",
    "uID": "",
    "attamentFilePath": "",
    "attamentFileMD5": "",
    "devLogFilePath": "",
    "devLogFileMD5": "",
    "item": {
      "key": [
        {
          "name": "DetectorTempL",
          "lowerThreshold": 300435,
          "upperThreshold": 680912.76,
          "unit": "摄氏度",
          "value": 42.32
        },
        {
          "name": "DetectorTempM",
          "lowerThreshold": 4432.12,
          "upperThreshold": 431503.62,
          "unit": "摄氏度",
          "value": 38.2
        }
      ]
    }
  },
  "fields": {
    "operationDateTime": [
      "2018-07-13T09:02:07.563Z"
    ]
  }
}

• C# Nest client access code like this:

 var client = GetClient();
            var filterPath = new string[] { "aggregations.*.*.buckets" };

            var request = new SearchRequest(indices)
            {
                Query = new MatchQuery() { Field = "systemID", Query = systemID }
                    && new DateRangeQuery() { Field = "operationDateTime", LessThanOrEqualTo = "now", Format = "yyyyMMdd" },
                Size = 1,
                Sort = new List<ISort>
                {
                    new SortField {Field = "operationDateTime", Order = SortOrder.Descending}
                },

                Aggregations = new NestedAggregation("nested")
                {
                    Path = "item.key",
                    Aggregations = new TermsAggregation("terms")
                    {
                        Field = "item.key.name.keyword",

                        Aggregations = new AverageAggregation("avg", "item.key.value")
                    }
                },

                FilterPath = filterPath,

            };

            var response = await client.SearchAsync<CTDMS<CTDMSItem>>(request);

            var agg = response.Aggregations.Nested("nested").Terms("terms");

• response DebugInformation like this:

Valid NEST response built from a successful low level call on POST: /%2Adms%2A/_search?typed_keys=true&filter_path=aggregations.%2A.%2A.buckets
# Audit trail of this API call:
 - [1] HealthyResponse: Node: http://localhost:9200/ Took: 00:00:02.0860412
# Request:
{"size":1,"sort":[{"operationDateTime":{"order":"desc"}}],"query":{"bool":{"must":[{"match":{"systemID":{"query":"101092010010"}}},{"range":{"operationDateTime":{"lte":"now","format":"yyyyMMdd"}}}]}},"aggs":{"nested":{"nested":{"path":"item.key"},"aggs":{"terms":{"terms":{"field":"item.key.name.keyword"},"aggs":{"avg":{"avg":{"field":"item.key.value"}}}}}}}}
# Response:
{"aggregations":{"nested#nested":{"sterms#terms":{"buckets":[{"key":"DetectorTempL","doc_count":19,"avg#avg":{"value":34.906860753109584}},{"key":"DetectorTempM","doc_count":19,"avg#avg":{"value":35.17318605121813}},{"key":"DetectorTempR","doc_count":19,"avg#avg":{"value":35.05932958502518}},{"key":"RackENVTemp","doc_count":19,"avg#avg":{"value":24.34830103422466}}]}}}}

• Problem result description:

the value of response's NestedAggration is null

I donot know what happened, do anyone give me some hint?
thanks.

[muzizongheng]

I find the problem is caused by filter_path. If I use filter_path in NestedAggregation, its result value is null; if I delete the filter_path, everything is ok.

[muzizongheng]

solution:
// FilterPath = filterPath,

[Mpdreamz]

Aye, FilterPath will be near impossible for us to support as it transforms the way results are returned.

Thank you for opening this issue.

@russcam @codebrain FilterPath probably needs a <remark> xml docs, thoughts?

[codebrain]

@Mpdreamz - Agreed, we don't have the capability to interpret the results given a filter_path expression unless we build something overly clever (which I am not suggesting we do!).

Perhaps a remarks with a link to the docs? https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#common-options-response-filtering

[russcam]

I've opened #3379 to address