Scripted field returns the wrong value from metricbeat

[parisamir]

Elasticsearch version: 5.0

Plugins installed: []

JVM version: 1.8

OS version: CentOS

Description of the problem including expected versus actual behavior:
The scripted field return wrong value.

Steps to reproduce:
1.Have metricbeat running.
2.Create a script field from the field "system.cpu.idle.pct" from teh management tab:
doc['system.cpu.idle.pct'].value
3. Create a line chart visualization on the metricbeat with two Y Axis.
- the field "system.cpu.idle.pct"
- the scrtipted field doc['system.cpu.idle.pct'].value
4. The results linecharts should be the same but they are totally different.

Provide logs (if relevant):

Here is the results: you can compare them:

@timestamp per 30 seconds Average system.cpu.idle.pct Average total_used_cpu_pct
March 9th 2017, 15:43:00.000 --- 0.771 --- 0.01481
March 9th 2017, 15:43:30.000 --- 0.762 --- 0.01471
March 9th 2017, 15:44:00.000 --- 0.762 --- 0.01471
March 9th 2017, 15:44:30.000 --- 0.769 --- 0.01484
March 9th 2017, 15:45:00.000 --- 0.76 --- 0.01465
March 9th 2017, 15:45:30.000 --- 0.774 --- 0.01493
March 9th 2017, 15:46:00.000 --- 0.761 --- 0.01468
March 9th 2017, 15:46:30.000 --- 0.767 --- 0.01479
March 9th 2017, 15:47:00.000 --- 0.76 --- 0.01465
March 9th 2017, 15:47:30.000 --- 0.768 --- 0.01482

and here is the script it is sending:

{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "analyze_wildcard": true,
            "query": "*"
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": 1489092190480,
              "lte": 1489093090480,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "must_not": []
    }
  },
  "aggs": {
    "3": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "30s",
        "time_zone": "America/New_York",
        "min_doc_count": 1
      },
      "aggs": {
        "1": {
          "avg": {
            "field": "system.cpu.idle.pct"
          }
        },
        "2": {
          "avg": {
            "script": {
              "inline": "doc['system.cpu.idle.pct'].value",
              "lang": "painless"
            }
          }
        }
      }
    }
  }
}

[parisamir]

The strange fact is if I create my visualization over the "cpu stats" search, the answer is correct!

[jpountz]

I'm wondering this could be a bug with getScriptValues on scaled floats.

[parisamir]

Hi again,

I've found out why the visualization works with the "cpu stats" and not the metricbeat index itself.
Taking a look at the request that is sending you'll see this:

{
"size": 0,
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"interval": "30s",
"time_zone": "America/New_York",
"min_doc_count": 1
},
"aggs": {
"1": {
"avg": {
"script": {
"inline": "doc['system.cpu.idle.pct'].value",
"lang": "painless"
}
}
}
}
}
},
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "metricset.module: system AND metricset.name: cpu",
"analyze_wildcard": true
}
},
{
"range": {
"@timestamp": {
"gte": 1489434545480,
"lte": 1489435445480,
"format": "epoch_millis"
}
}
}
],
"must_not": []
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"*": {}
},
"require_field_match": false,
"fragment_size": 2147483647
}
}

The search filters if there is no value for the field and do not set it to value 0.
The metrics beat does not send values for all the timeslots. The timeslots that do not have the fields value will get the value zero for the script field. The search solves this by putting a filter.

I think This is not a bug on elasticsearch/ kibana. But metric beat should warn the users that this can happen using the script fields over the index itself.

[jdconrad]

@jpountz Were you ever able to confirm your hypothesis?

[stu-elastic]

This was #29286, prior to 6.4 the script would return a default value, which was impacting the aggregated denominator. This is different behavior between the avg agg, which would ignore such docs and the scripted avg agg which would return the default value.

From 6.4 on, users must explicitly check for existence of any values.