dynamic_date_formats ignored if dynamic_templates for date is present

[SergeyTsalkov]

My goal was to have a dynamically created date field with the ignore_malformed option set. However, when I have both dynamic_date_formats and dynamic_templates for the date type, the dynamic_date_formats is ignored.

So let's create our index..

curl -XPOST localhost:9200/test -d '{
  "mappings": {
    "test": {
      "dynamic_date_formats": [
        "yyyy-MM-dd"
      ],
      "dynamic_templates": [
        {
          "dates_ignore_malformed": {
            "path_match": "*",
            "match_mapping_type": "date",
            "mapping": {
              "ignore_malformed": true
            }
          }
        }
      ]
    }
  }
}'

And add some data..

curl -XPOST localhost:9200/test/test/1 -d '{
  "something": "2014-01-05"
}'

And now get the mappings back..

curl -XGET localhost:9200/test/_mappings
->
{
  "test": {
    "mappings": {
      "test": {
        "dynamic_date_formats": [
          "yyyy-MM-dd"
        ],
        "dynamic_templates": [
          {
            "dates_ignore_malformed": {
              "mapping": {
                "ignore_malformed": true
              },
              "match_mapping_type": "date",
              "path_match": "*"
            }
          }
        ],
        "properties": {
          "something": {
            "type": "date",
            "ignore_malformed": true,
            "format": "dateOptionalTime"
          }
        }
      }
    }
  }
}

See how the format for "something" is dateOptionalTime? Had I not included the dynamic_templates, that would have been (and should have been) "yyyy-MM-dd".

[clintongormley]

First Elasticsearch checks the unknown string field against the list of date formats in dynamic_date_formats to determine whether the field contains a date or a string.

Once it decides that it is a date, it applies the mapping found in the templates, in which you don't specify the date format. Thus it uses the default matching format which is dateOptionalTime. Why not just specify the format you want in the template mapping? eg:

DELETE test

POST /test
{
  "mappings": {
    "test": {
      "dynamic_date_formats": [
        "yyyy/MM/dd"  
      ],
      "dynamic_templates": [
        {
          "dates_ignore_malformed": {
            "path_match": "*",
            "match_mapping_type": "date",
            "mapping": {
              "format": "yyyy/MM/dd",
              "ignore_malformed": true
            }
          }
        }
      ]
    }
  }
}

POST /test/test/1
{
  "format_one": "2014-01-05"
}

POST /test/test/2
{
  "format_two": "2014/01/05"
}

GET _mapping

This returns:

{
   "test": {
      "mappings": {
         "test": {
            "dynamic_date_formats": [
               "yyyy/MM/dd"
            ],
            "dynamic_templates": [
               {
                  "dates_ignore_malformed": {
                     "mapping": {
                        "ignore_malformed": true,
                        "format": "yyyy/MM/dd"
                     },
                     "match_mapping_type": "date",
                     "path_match": "*"
                  }
               }
            ],
            "properties": {
               "format_one": {
                  "type": "string"
               },
               "format_two": {
                  "type": "date",
                  "ignore_malformed": true,
                  "format": "yyyy/MM/dd"
               }
            }
         }
      }
   }
} 

[SergeyTsalkov]

So what if you wanted to use dynamic_date_formats to specify multiple possible formats? Does that prevent you from also using ignore_malformed?

[clintongormley]

The first time that field is seen, it will be mapped as a date only if it matches one of the formats you listed in dynamic_date_formats, otherwise it will be mapped as a string.

Once the field is mapped, then ignore_malformed will allow you to ignore malformed dates later on. But if the first date seen is malformed, then the field will be a string instead.

This is the same way that the mapping would work if you do not specify dynamic_date_formats or templates.

[SergeyTsalkov]

I appreciate you taking the time to help me clear this up, but it still seems like the thing I want to do isn't possible. In my data set, the date might be in one of several formats, and I won't know which one ahead of time. That's why I'm trying to use dynamic_date_formats to specify the possibilities, but I guess my first example was unclear because I only listed one. This is more what I was going for:

{
  "mappings": {
    "test": {
      "dynamic_date_formats": [
        "yyyy-MM-dd",
        "yyyy/MM/dd"
      ],
      "dynamic_templates": [
        {
          "dates_ignore_malformed": {
            "path_match": "*",
            "match_mapping_type": "date",
            "mapping": {
              "ignore_malformed": true
            }
          }
        }
      ]
    }
  }
}

So whichever format the first date has would be applied to the new field, and the date would be expected in that format from then on. If it came in a different format from then on, ignore_malformed would cause it to be ignored.

I guess that isn't possible with ElasticSearch?

[clintongormley]

@SergeyTsalkov Actually you're completely right - the dynamic_date_formats are not used after adding the mapping to determine the format for the date field:

DELETE test

POST /test
{
  "mappings": {
    "test": {
      "dynamic_date_formats": [
        "yyyy-MM-dd"  ,
        "dd/MM/yyyy"
      ],
      "dynamic_templates": [
        {
          "dates_ignore_malformed": {
            "path_match": "*",
            "match_mapping_type": "date",
            "mapping": {
              "type":"date"
            }
          }
        }
      ]
    }
  }
}

This results in a date field with format dateOptionalTime:

POST /test/test/1
{
  "format_one": "2014-01-05"
}

This throws an exception indicating that it is ignoring the dynamic_date_formats: failed to parse date field [05/01/2014], tried both date format [dateOptionalTime], and timestamp number with locale []

POST /test/test/2
{
  "format_two": "05/01/2014"
}

GET _mapping

[rjernst]

the dynamic_date_formats are not used after adding the mapping to determine the format for the date field

But that's the way it should be? As you said before, dynamic_date_formats is to determine what is a date. There was then no format put on the dynamic field, and when it tried to parse the value (with the actual field mapping format of dateOptionalTime) it failed. I don't think dynamic_date_formats should in any ways adjust the dynamic field's format (that is what the dynamic field mapping is for).

[clintongormley]

@rjernst if you didn't specify any dynamic_date_formats or any templates, then it would check a string against the default list of dynamic date formats and apply the first format that matches to the field.

With the example above, it uses the specified dynamic date formats to determine whether the field is a date or not, but then it uses the default list of dynamic date formats to determine which format to apply. Instead, it should use the custom list.

[sqlboy]

This is the way to do it. Set your dynamic_date_format, then set the same ones as the value of "format" in your dynamic mapping.

  "dynamic_date_formats": [
        "yyyy/MM/dd HH:mm:ss Z",
        "yyyy/MM/dd Z",
        "yyyy/MM/dd HH:mm:ss",
        "yyyy/MM/dd",
        "yyyy-MM-dd HH:mm:ss Z",
        "yyyy-MM-dd Z",
        "yyyy-MM-dd HH:mm:ss",
        "yyyy-MM-dd",
        "MM/dd/yyyy",
        "MM-dd-yyyy"
      ],

          "date_template": {
            "match": "*",
            "match_mapping_type": "date",
            "mapping": {
              "type": "date",
              "index": "not_analyzed",
              "format": "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z||yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||MM/dd/yyyy||MM-dd-yyyy"
            }
          }

This way, you can add dates of different format and the field doesn't get locked down to a single format.