Skip to content

Fix DateMathExpressionResolverTests tests #37037 #37059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1195,7 +1195,7 @@ private void validateBulkResponses(int nbItems, boolean[] errors, BulkResponse b
public void testUrlEncode() throws IOException {
String indexPattern = "<logstash-{now/M}>";
String expectedIndex = "logstash-" +
DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(DateTimeZone.UTC).monthOfYear().roundFloorCopy());
DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(DateTimeZone.UTC).monthOfYear().roundFloorCopy());
{
IndexRequest indexRequest = new IndexRequest(indexPattern).id("id#1");
indexRequest.source("field", "value");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ To run this example, first follow the steps in

The seat data contains:

* A date in the format `YYYY-MM-DD` where the second digit of both month and day
* A date in the format `yyyy-MM-DD` where the second digit of both month and day
is optional.
* A time in the format HH:MM* where the second digit of both hours and minutes
is optional. The star (*) represents either the `String` `AM` or `PM`.
Expand Down
10 changes: 5 additions & 5 deletions docs/reference/api-conventions.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Where:
[horizontal]
`static_name`:: is the static text part of the name
`date_math_expr`:: is a dynamic date math expression that computes the date dynamically
`date_format`:: is the optional format in which the computed date should be rendered. Defaults to `YYYY.MM.dd`.
`date_format`:: is the optional format in which the computed date should be rendered. Defaults to `yyyy.MM.dd`.
`time_zone`:: is the optional time zone . Defaults to `utc`.

Date math expressions are resolved locale-independent. Consequently, it is not possible to use any other
Expand Down Expand Up @@ -132,9 +132,9 @@ they resolve to given the current time is 22rd March 2024 noon utc.
| Expression |Resolves to
| `<logstash-{now/d}>` | `logstash-2024.03.22`
| `<logstash-{now/M}>` | `logstash-2024.03.01`
| `<logstash-{now/M{YYYY.MM}}>` | `logstash-2024.03`
| `<logstash-{now/M-1M{YYYY.MM}}>` | `logstash-2024.02`
| `<logstash-{now/d{YYYY.MM.dd\|+12:00}}>` | `logstash-2024.03.23`
| `<logstash-{now/M{yyyy.MM}}>` | `logstash-2024.03`
| `<logstash-{now/M-1M{yyyy.MM}}>` | `logstash-2024.02`
| `<logstash-{now/d{yyyy.MM.dd\|+12:00}}>` | `logstash-2024.03.23`
|======

To use the characters `{` and `}` in the static part of an index name template, escape them
Expand All @@ -144,7 +144,7 @@ with a backslash `\`, for example:

The following example shows a search request that searches the Logstash indices for the past
three days, assuming the indices use the default Logstash index name format,
`logstash-YYYY.MM.dd`.
`logstash-yyyy.MM.dd`.

[source,js]
----------------------------------------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/monitoring/http-export.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ xpack.monitoring.exporters:
headers: <7>
My-Proxy-Header: abc123
My-Other-Thing: [ def456, ... ]
index.name.time_format: YYYY-MM <8>
index.name.time_format: yyyy-MM <8>

----------------------------------
<1> A `local` exporter defined explicitly whose arbitrary name is `my_local`.
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/settings/monitoring-settings.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ anything defined here.
`index.name.time_format`::

A mechanism for changing the default date suffix for the, by default, daily Monitoring indices.
The default value is `YYYY.MM.DD`, which is why the indices are created daily.
The default value is `yyyy.MM.DD`, which is why the indices are created daily.

`use_ingest`::

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/sql/functions/date-time.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The table below shows the mapping between {es} and {es-sql}:
s|{es}
s|{es-sql}
2+h| Index/Table datetime math
2+|<index-{now/M{YYYY.MM}}>
2+|<index-{now/M{yyyy.MM}}>
2+h| Query date/time math
| 1y | INTERVAL 1 YEAR
| 2M | INTERVAL 2 MONTH
Expand Down
2 changes: 1 addition & 1 deletion libs/grok/src/main/resources/patterns/grok-patterns
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ MINUTE (?:[0-5][0-9])
# '60' is a leap second in most time standards and thus is valid.
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)
# datestamp is yyyy/MM/DD-HH:MM:SS.UUUU (or something like it)
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -355,8 +355,8 @@ public void testUrlEncoder() {
}

public void testUrlEncoderWithParam() throws Exception {
assertScript("{{#url}}{{index}}{{/url}}", singletonMap("index", "<logstash-{now/d{YYYY.MM.dd|+12:00}}>"),
equalTo("%3Clogstash-%7Bnow%2Fd%7BYYYY.MM.dd%7C%2B12%3A00%7D%7D%3E"));
assertScript("{{#url}}{{index}}{{/url}}", singletonMap("index", "<logstash-{now/d{yyyy.MM.dd|+12:00}}>"),
equalTo("%3Clogstash-%7Bnow%2Fd%7Byyyy.MM.dd%7C%2B12%3A00%7D%7D%3E"));

final String random = randomAlphaOfLength(10);
assertScript("{{#url}}prefix_{{s}}{{/url}}", singletonMap("s", random),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -716,7 +716,7 @@ private static Mapper.Builder<?,?> createBuilderFromDynamicValue(final ParseCont
} else if (parseableAsLong == false && parseableAsDouble == false && context.root().dateDetection()) {
// We refuse to match pure numbers, which are too likely to be
// false positives with date formats that include eg.
// `epoch_millis` or `YYYY`
// `epoch_millis` or `yyyy`
for (DateFormatter dateTimeFormatter : context.root().dynamicDateTimeFormatters()) {
try {
dateTimeFormatter.parse(text);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ protected StrictISODateTimeFormat() {
* <p>
* This method examines the fields provided and returns an ISO-style
* formatter that best fits. This can be useful for outputting
* less-common ISO styles, such as YearMonth (YYYY-MM) or MonthDay (--MM-DD).
* less-common ISO styles, such as YearMonth (yyyy-MM) or MonthDay (--MM-DD).
* <p>
* The list provided may have overlapping fields, such as dayOfWeek and
* dayOfMonth. In this case, the style is chosen based on the following
Expand Down Expand Up @@ -234,26 +234,26 @@ private static boolean dateByMonth(
bld.append(Constants.ye);
if (fields.remove(DateTimeFieldType.monthOfYear())) {
if (fields.remove(DateTimeFieldType.dayOfMonth())) {
// YYYY-MM-DD/YYYYMMDD
// yyyy-MM-DD/yyyyMMDD
appendSeparator(bld, extended);
bld.appendMonthOfYear(2);
appendSeparator(bld, extended);
bld.appendDayOfMonth(2);
} else {
// YYYY-MM/YYYY-MM
// yyyy-MM/yyyy-MM
bld.appendLiteral('-');
bld.appendMonthOfYear(2);
reducedPrec = true;
}
} else {
if (fields.remove(DateTimeFieldType.dayOfMonth())) {
// YYYY--DD/YYYY--DD (non-iso)
// yyyy--DD/yyyy--DD (non-iso)
checkNotStrictISO(fields, strictISO);
bld.appendLiteral('-');
bld.appendLiteral('-');
bld.appendDayOfMonth(2);
} else {
// YYYY/YYYY
// yyyy/yyyy
reducedPrec = true;
}
}
Expand Down Expand Up @@ -299,11 +299,11 @@ private static boolean dateByOrdinal(
if (fields.remove(DateTimeFieldType.year())) {
bld.append(Constants.ye);
if (fields.remove(DateTimeFieldType.dayOfYear())) {
// YYYY-DDD/YYYYDDD
// yyyy-DDD/yyyyDDD
appendSeparator(bld, extended);
bld.appendDayOfYear(3);
} else {
// YYYY/YYYY
// yyyy/yyyy
reducedPrec = true;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -276,42 +276,42 @@ public void testRolloverOnExistingIndex() throws Exception {
public void testRolloverWithDateMath() {
ZonedDateTime now = ZonedDateTime.now(ZoneOffset.UTC);
assumeTrue("only works on the same day", now.plusMinutes(5).getDayOfYear() == now.getDayOfYear());
String index = "test-" + DateFormatter.forPattern("YYYY.MM.dd").format(now) + "-1";
String index = "test-" + DateFormatter.forPattern("yyyy.MM.dd").format(now) + "-1";
String dateMathExp = "<test-{now/d}-1>";
assertAcked(prepareCreate(dateMathExp).addAlias(new Alias("test_alias")).get());
ensureGreen(index);
// now we modify the provided name such that we can test that the pattern is carried on
client().admin().indices().prepareClose(index).get();
client().admin().indices().prepareUpdateSettings(index).setSettings(Settings.builder()
.put(IndexMetaData.SETTING_INDEX_PROVIDED_NAME,
"<test-{now/M{YYYY.MM}}-1>")).get();
"<test-{now/M{yyyy.MM}}-1>")).get();

client().admin().indices().prepareOpen(index).get();
ensureGreen(index);
RolloverResponse response = client().admin().indices().prepareRolloverIndex("test_alias").get();
assertThat(response.getOldIndex(), equalTo(index));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("YYYY.MM").format(now) + "-000002"));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("yyyy.MM").format(now) + "-000002"));
assertThat(response.isDryRun(), equalTo(false));
assertThat(response.isRolledOver(), equalTo(true));
assertThat(response.getConditionStatus().size(), equalTo(0));

response = client().admin().indices().prepareRolloverIndex("test_alias").get();
assertThat(response.getOldIndex(), equalTo("test-" + DateFormatter.forPattern("YYYY.MM").format(now) + "-000002"));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("YYYY.MM").format(now) + "-000003"));
assertThat(response.getOldIndex(), equalTo("test-" + DateFormatter.forPattern("yyyy.MM").format(now) + "-000002"));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("yyyy.MM").format(now) + "-000003"));
assertThat(response.isDryRun(), equalTo(false));
assertThat(response.isRolledOver(), equalTo(true));
assertThat(response.getConditionStatus().size(), equalTo(0));

GetSettingsResponse getSettingsResponse = client().admin().indices().prepareGetSettings(response.getOldIndex(),
response.getNewIndex()).get();
assertEquals("<test-{now/M{YYYY.MM}}-000002>", getSettingsResponse.getSetting(response.getOldIndex(),
assertEquals("<test-{now/M{yyyy.MM}}-000002>", getSettingsResponse.getSetting(response.getOldIndex(),
IndexMetaData.SETTING_INDEX_PROVIDED_NAME));
assertEquals("<test-{now/M{YYYY.MM}}-000003>", getSettingsResponse.getSetting(response.getNewIndex(),
assertEquals("<test-{now/M{yyyy.MM}}-000003>", getSettingsResponse.getSetting(response.getNewIndex(),
IndexMetaData.SETTING_INDEX_PROVIDED_NAME));

response = client().admin().indices().prepareRolloverIndex("test_alias").setNewIndexName("<test-{now/d}-000004>").get();
assertThat(response.getOldIndex(), equalTo("test-" + DateFormatter.forPattern("YYYY.MM").format(now) + "-000003"));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("YYYY.MM.dd").format(now) + "-000004"));
assertThat(response.getOldIndex(), equalTo("test-" + DateFormatter.forPattern("yyyy.MM").format(now) + "-000003"));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("yyyy.MM.dd").format(now) + "-000004"));
assertThat(response.isDryRun(), equalTo(false));
assertThat(response.isRolledOver(), equalTo(true));
assertThat(response.getConditionStatus().size(), equalTo(0));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,11 @@ public void testExpression() throws Exception {
List<String> result = expressionResolver.resolve(context, indexExpressions);
assertThat(result.size(), equalTo(3));
assertThat(result.get(0),
equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
equalTo(".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
assertThat(result.get(1),
equalTo(".watch_history-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
equalTo(".watch_history-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
assertThat(result.get(2),
equalTo("logstash-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
equalTo("logstash-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
}

public void testEmpty() throws Exception {
Expand All @@ -86,12 +86,11 @@ public void testExpression_MultiParts() throws Exception {
List<String> result = expressionResolver.resolve(context, Arrays.asList("<.text1-{now/d}-text2-{now/M}>"));
assertThat(result.size(), equalTo(1));
assertThat(result.get(0), equalTo(".text1-"
+ DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC))
+ DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))
+ "-text2-"
+ DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC).withDayOfMonth(1))));
+ DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC).withDayOfMonth(1))));
}

@AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/37037")
public void testExpression_CustomFormat() throws Exception {
List<String> results = expressionResolver.resolve(context, Arrays.asList("<.marvel-{now/d{yyyy.MM.dd}}>"));
assertThat(results.size(), equalTo(1));
Expand All @@ -106,7 +105,6 @@ public void testExpression_EscapeStatic() throws Exception {
equalTo(".mar{v}el-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
}

@AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/37037")
public void testExpression_EscapeDateFormat() throws Exception {
List<String> result = expressionResolver.resolve(context, Arrays.asList("<.marvel-{now/d{'\\{year\\}'yyyy}}>"));
assertThat(result.size(), equalTo(1));
Expand All @@ -116,18 +114,17 @@ public void testExpression_EscapeDateFormat() throws Exception {

public void testExpression_MixedArray() throws Exception {
List<String> result = expressionResolver.resolve(context, Arrays.asList(
"name1", "<.marvel-{now/d}>", "name2", "<.logstash-{now/M{YYYY.MM}}>"
"name1", "<.marvel-{now/d}>", "name2", "<.logstash-{now/M{yyyy.MM}}>"
));
assertThat(result.size(), equalTo(4));
assertThat(result.get(0), equalTo("name1"));
assertThat(result.get(1),
equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
equalTo(".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
assertThat(result.get(2), equalTo("name2"));
assertThat(result.get(3), equalTo(".logstash-" +
DateTimeFormat.forPattern("YYYY.MM").print(new DateTime(context.getStartTime(), UTC).withDayOfMonth(1))));
DateTimeFormat.forPattern("yyyy.MM").print(new DateTime(context.getStartTime(), UTC).withDayOfMonth(1))));
}

@AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/37037")
public void testExpression_CustomTimeZoneInIndexName() throws Exception {
DateTimeZone timeZone;
int hoursOffset;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -662,7 +662,7 @@ public void testsTimeZoneParsing() {
// Formatter used to print and parse the sample date.
// Printing the date works but parsing it back fails
// with Joda 2.9.4
DateTimeFormatter formatter = DateTimeFormat.forPattern("YYYY-MM-dd'T'HH:mm:ss " + randomFrom("ZZZ", "[ZZZ]", "'['ZZZ']'"));
DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss " + randomFrom("ZZZ", "[ZZZ]", "'['ZZZ']'"));

String dateTimeAsString = formatter.print(expected);
assertThat(dateTimeAsString, startsWith("2016-11-10T05:37:59 "));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -127,20 +127,20 @@ public void testTimeZones() {
public void testEqualsAndHashcode() {
assertThat(DateFormatters.forPattern("strict_date_optional_time"),
sameInstance(DateFormatters.forPattern("strict_date_optional_time")));
assertThat(DateFormatters.forPattern("YYYY"), equalTo(DateFormatters.forPattern("YYYY")));
assertThat(DateFormatters.forPattern("YYYY").hashCode(),
is(DateFormatters.forPattern("YYYY").hashCode()));
assertThat(DateFormatters.forPattern("yyyy"), equalTo(DateFormatters.forPattern("yyyy")));
assertThat(DateFormatters.forPattern("yyyy").hashCode(),
is(DateFormatters.forPattern("yyyy").hashCode()));

// different timezone, thus not equals
assertThat(DateFormatters.forPattern("YYYY").withZone(ZoneId.of("CET")), not(equalTo(DateFormatters.forPattern("YYYY"))));
assertThat(DateFormatters.forPattern("yyyy").withZone(ZoneId.of("CET")), not(equalTo(DateFormatters.forPattern("yyyy"))));

// different locale, thus not equals
DateFormatter f1 = DateFormatters.forPattern("YYYY").withLocale(Locale.CANADA);
DateFormatter f1 = DateFormatters.forPattern("yyyy").withLocale(Locale.CANADA);
DateFormatter f2 = f1.withLocale(Locale.FRENCH);
assertThat(f1, not(equalTo(f2)));

// different pattern, thus not equals
assertThat(DateFormatters.forPattern("YYYY"), not(equalTo(DateFormatters.forPattern("YY"))));
assertThat(DateFormatters.forPattern("yyyy"), not(equalTo(DateFormatters.forPattern("yy"))));

DateFormatter epochSecondFormatter = DateFormatters.forPattern("epoch_second");
assertThat(epochSecondFormatter, sameInstance(DateFormatters.forPattern("epoch_second")));
Expand Down
Loading