ES 7 changes which dates date_optional_time can successfully parse

[misterbyrne]

Elasticsearch version (bin/elasticsearch --version):

Version: 7.12.0, Build: default/tar/78722783c38caa25a70982b5b042074cde5d3b3a/2021-03-18T06:17:15.410153305Z, JVM: 15.0.1

Plugins installed: []

JVM version (java -version):

jdk.app/Contents/Home/bin/java --version
openjdk 15.0.1 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):

Darwin MacBook-Pro.local 19.6.0 Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64 x86_64

Description of the problem including expected versus actual behavior:

The values that the date_optional_time format can successfully parse have changed in ElasticSearch 7.
It seems to be related to dates where the year includes more than 5 digits.

I understand that ES7 includes a migration from Joda time formatters to Java time formatters and there are some expected breaking changes - as documented in the Java time migration guide.

This change in behaviour could be deliberate, but it's not clear from the documentation that this migration affects the date_optional_time built-in format. From the docs:

The switch to java time only impacts custom date and date_nanos formats.

Steps to reproduce:

1. Create an index with a date field using the date_optional_time format:

curl -X PUT "localhost:9200/my-index-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "date_optional_time"
      }
    }
  }
}
'

2. Attempt to create a new document with a value which was valid in ES6.8 (or an index created in ES6.8)

curl -X PUT "localhost:9200/my-index-000001/_create/1?pretty" -H 'Content-Type: application/json' -d'
{
  "date": "123456-01-02T12:34:56"
}
'

Error response:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [date] of type [date] in document with id '1'. Preview of field's value: '123456-01-02T12:34:56'"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse field [date] of type [date] in document with id '1'. Preview of field's value: '123456-01-02T12:34:56'",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "failed to parse date field [123456-01-02T12:34:56] with format [date_optional_time]",
      "caused_by" : {
        "type" : "date_time_parse_exception",
        "reason" : "Text '123456-01-02T12:34:56' could not be parsed, unparsed text found at index 5"
      }
    }
  },
  "status" : 400
}

Example of this working successfully in ES6.8:

curl -X PUT "localhost:9200/my-index-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "_doc": {
        "properties": {
            "date": {
                "type":   "date",
                "format": "date_optional_time"
            }
        }
    }
  }
}
'

curl -X PUT "localhost:9200/my-index-000001/_doc/1?pretty" -H 'Content-Type: application/json' -d'
{
  "date": "123456-01-02T12:34:56"
}
'

Response:

{
  "_index" : "my-index-000001",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[cbuescher]

@pgomulka do you have an opinion on this? I see we accepted up to 9 digits in 6.8 (I see from an error message in 6.8 that "year must be in the range [-292275055,292278994]") but maybe the restriction to 5 digits was introduced in 7x on purpose and this should be a documentation issue? I don't know all the background around these decisions I'm afraid.

[pgomulka]

this might be a bug. As stated in joda 'non strict' date_optional_time had year of length 4-9
the strict_date_optional_time which is a modified ES version had it changed to year of lenght 4-4
I was about to fix this #54260 but I forgot about this PR.
I will try to update the linked PR with the fix for the date_optional_time as well