Fix Rollover error when alias has closed indices

[gwbrown]

Rollover previously requested index stats for all indices in the
provided alias, which causes an exception when there is a closed index
with that alias.

This commit adjusts the IndicesOptions used on the index stats
request so that closed indices are ignored, rather than throwing
an exception.

Fixes #47146

[elasticmachine]

Pinging @elastic/es-core-features

[gwbrown]

The bug this PR attempts to address, #47146, is a regression introduced by #40774, and this PR is currently causing a test introduced by that fix to fail - we'll have to see if there's anything else we can do about that.

[dakrone]

I wonder if there is a combination of indices options (.indicesOptions(...)) we can set that will allow us to still use the alias, I think setting ignore_unavailable to true may skip closed indices?

[gwbrown]

This was a really good thought, ignore_unavailable looks like it might do the trick. I'll push a version that uses that shortly for a full CI run.

[gwbrown]

The default-distro failure is #47196
The bwc failure is something in snapshot/restore tests that I don't think is related. I'll dig in and file an issue if necessary.

[gwbrown]

@elasticmachine update branch

[gwbrown]

@elasticmachine update branch

[gwbrown]

@dakrone has found some very odd behavior when the write index for an alias is closed with this change. It looks like there is, somehow, data being returned from the indices stats call about the write index even when it's closed, which shouldn't be possible, and similar calls to the REST API display different results than what appears to be getting returned from the transport layer. I'm going to dig into this and see what's up.

[gwbrown]

I've determined the odd behavior we were seeing is due to the fact that index stats requests do in fact work correctly (at least as far as the information that rollover needs) in versions 7.2 and later due to the work on replicated closed indices (and #39698 in particular). However, index stats requests via the REST API currently fail for closed indices unless the forbid_closed_indices flag is explicitly set to false to make the behavior match 7.0/7.1. The methods used in this PR set forbid_closed_indices=false implicitly.

This leads to a somewhat odd behavior where the write index can be closed, but Rollover will work as normal. Because the index stats API returns the correct stats, this is probably fine - but this situation will not work the same if this change is backported to 6.8, because index stats still are not returned for closed indices in that version (due to lacking the changes for replicated closed indices).

This is a pretty obscure corner case though - I think it might be okay to have 6.8 fail if the current write index is closed, but roll over as normal in the 7.x/8.x versions.

As it is, if the index stats are missing from the index stats response, Rollover just silently doesn't roll over - a null response for the document stats is treated as a 0:

elasticsearch/server/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java

Lines 249 to 250 in 7275642

	final long numDocs = docsStats == null ? 0 : docsStats.getCount();
	final long indexSize = docsStats == null ? 0 : docsStats.getTotalSizeInBytes();

I'd like to be more strict here, but I'm concerned that erroring on null values instead of swallowing them would lead to an increase in errors on overloaded/unstable clusters that would exacerbate the issue we already have with ILM rollover. My preference would be to revisit making this stricter once we've addressed #44135.

That still leaves the question of what to do with this for 6.8 - with this change, if the write index is closed, rollover will silently never perform the rollover if a docs or size condition would otherwise be met (vs. failing with an error today).

What do you think @dakrone?

[dakrone]

Thanks for summarizing your findings, it's nice to have all the options documented.

My opinion would be to:

• For 7.4+, keep the behavior where we can still rollover even if the write index is closed, since the stats are available.
• For 6.8, add a change that treats this more strictly, where a closed write index causes the rollover to fail.

In both cases, we should fix the issue where the non-write index being closed causes an error on rollover.

I'm in favor of being strict for 6.8 because I think silently never rolling over (in 6.8) is just as bad as going to an error state, but harder to debug, since at least in the strict case it's apparent what's causing the rollover to fail. I also don't think this specific error is a concern for rollover accumulating data forever, since in this case the index can't actually accumulate any data.

#44135 is very unlikely to be backported to the 6.8 branch, so I think for 6.8 we should stick with the solution for it that is the most apparent to the user and points to a potential solution (the write index is closed, so you need to re-open it if you want to rollover).

[gwbrown]

@dakrone I've added a test to confirm the behavior with a closed write index and removed 6.8 as a target version for this PR. This should be ready for review.

Once this is merged, I'll put up a separate PR for 6.8 as it has different implications.

[dakrone]

LGTM