datastream doesn't work with ignore_malformed setting

[eljohnson92]

Elasticsearch version : 7.12.0

{
  "name" : "test-container",
  "cluster_name" : "test",
  "cluster_uuid" : "123456789",
  "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Plugins installed: [mapper-size]

Description of the problem including expected versus actual behavior:

We are using the index.mapping.ignore_malformed: true setting with datastream for our general purpose logging solution. the index-template would be applied successfully in elastic version 7.9.0 but after upgrading to 7.11.2, and then also trying 7.12.0 it errors out

Steps to reproduce:
request

curl "localhost:9200/_index_template/filebeat?pretty" -H "content-type: application/json" -XPUT -d '{
      "index_patterns": [
        "filebeat-*"
      ],
      "template": {
        "settings": {
          "index": {
            "mapping.ignore_malformed": true
          }
        }
      },
      "data_stream": {}
    }' | jq .

response:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "composable template [filebeat] template after composition is invalid"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "composable template [filebeat] template after composition is invalid",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "invalid composite mappings for [filebeat]",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "data stream timestamp field [@timestamp] has disallowed [ignore_malformed] attribute specified"
      }
    }
  },
  "status": 400
}

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[martijnvg]

Thanks for reporting @eljohnson92.

Document inside a data stream should always have a valid timestamp in the @timestamp field, so configuring this field accept malformed values (which don't get indexed), is something Elasticsearch shouldn't allow.

In this case, with an index wide index.mapper.ignore_malformed setting configured, all date fields can accept a malformed date/timestamp value. I think Elasticsearch should ignore the index.mapping.ignore_malformed setting for the @timestamp field and let the setting be applied for all other date field.

[eljohnson92]

Thanks for responding @martijnvg is there anything we should do to work around this outside of just turning off index.mapping.ignore_malformed for now?

[martijnvg]

I was going to reply that the following template (with index.mapping.ignore_malformed) would work:

PUT /_index_template/filebeat
{
    "index_patterns": [
        "filebeat-*"
    ],
    "template": {
        "settings": {
            "index": {
                "mapping.ignore_malformed": true
            }
        },
        "mappings": {
            "properties": {
                "@timestamp": {
                    "type": "date",
                    "ignore_malformed": false
                }
            }
        }
    },
    "data_stream": {}
}

Only it doesn't. Another validation error occurs, which in my opinion shouldn't. I will fix this.
The idea here is to opt out of ignoring malformed values only for the @timestamp field.

Unfortunately I don't see a work around this and it seems like disabling index.mapping.ignore_malformed index setting is the only way now (until the fix is in).