refactor MetaData.Builder to update aliasAndIndexLookup #29575
Conversation
talevy
added
>non-issue
review
:Core/Infra/Core
|
Pinging @elastic/es-core-infra |
It would be nice to be able to look up other indices' aliases while validating new aliases being introduced. For example, If we would like to prohibit certain aliases and allow others, depending on their respective definitions. The problem with moving the construction of this lookup data-structure to be iterative is that we do not have a clear view of *all* other possible duplicate aliases. We would raise an error with the first duplicate we see.
talevy
force-pushed
the
refactor-aliasandindexlookup
branch
from
ceda3c1 to
6d3415a
Compare
|
@martijnvg after realizing what it takes to make this change work, I'm leaning towards your suggestion that was made offline about having more validation in |
|
@talevy @martijnvg I understand your sentiment but there are also things like #27689 we should consider. We allow a list of atomic alias updates that are applied one by one. I think we need to truly execute them one by one while tracking the current state and throwing errors as soon as we get to a bad place. For example, what happens if people send two commands to update the write alias for the same alias? what will the second one use as a basis?
It should be sane (i.e. not a n^2 operation) but also doesn't need to be super high performance.
As discussed validation in the build method is required regardless. We don't always construct cluster states via the formal API (i.e., during recovery we don't). |
|
@bleskes I see, thanks. I will push forward with this. I believe I have fixed all the test failures after my last commit. |
| return alias; | ||
| } else { | ||
| String indexName = ((AliasOrIndex.Index) alias).getIndex().getIndex().getName(); | ||
| throw new IllegalStateException("index and alias names need to be unique, but the following duplicate was found [" |
There was a problem hiding this comment.
I understand I ported this over from MetaData.Builder#build where IllegalStateException makes more sense. I believe this should be InvalidAliasNameException, will update if others agree
|
So, I've moved along with the |
talevy
added
:Distributed Indexing/CRUD
|
Pinging @elastic/es-distributed |
talevy
added
:Data Management/Indices APIs
|
After discussion with the team, I am closing this. It is not necessary for me anymore |
This commit introduces the concept of an `is_write_index` flag for aliases. Before, it was invalid to write to indices which point to multiple indices. Now, users can configure one index to have an alias with `is_write_index: true` and others that are false. When one attempts to index to this alias, it will resolve to the one index that is the write index. in response to elastic#30061. This PR rebuilds aliasAndIndexLookup every time it wants to use it for alias validation. Depends on outcome of elastic#29575 for using a pre-built data-structure. This PR introduces a concept in aliases called `is_write_index`. When `true` on a specific alias definition associated with an `index`, index requests using the associated alias name will resolve to this `index` for indexing. Write now, if one attempts to index a document using an alias that points to multiple indices, there is no way to resolve which one, so an exception is thrown. With this flag, you can have an alias that points to a specific index in a set of indices as the one to write to. It is possible to set an index as an alias' write-index during index creation: ``` PUT foo1 { "aliases": { "bar": { "is_write_index": true } } } ``` Now, index requests like `PUT bar/_doc/_id` will route to index `foo1`. Another index, `foo2`, is free to also have an alias named "bar" pointing to it. This alias needs to be configured with `is_write_index` set to `false`. `is_write_index` is defaulted to `false` if not specified, or it can be explicitly set to `false`. ``` PUT foo2 { "aliases": { "bar": {} } } ``` Now, searches against `bar` will point to both indices, but index requests will be routed to `foo1`. These settings can be updated in the `_aliases` API like so: ``` POST _aliases { "actions": [ { "add": { "index": "foo1", "alias": "bar", "is_write_index": false } } ] } ``` How does this relate to Rollover? Rollover depended on rollover-aliases that only point to one index since multiple would break indexing. Now that the same alias can point to multiple indices, the only requirement we have is that there is only one write index. Here is an example scenario that didn't work before, but works with these changes: ``` PUT _template/my_template { "index_patterns": ["foo-*"], "settings": { "number_of_shards": 1, "number_of_replicas": 1 }, "aliases": { "logs": {} }, "mappings": { "_doc": { "_source": { "enabled": false } } } } PUT foo-000001 { "aliases": { "logs": { "is_write_index": true } } } PUT logs/_doc/_id { "hello": "world" } POST /logs/_rollover/ { "conditions": { "max_docs": 1 } } GET foo-000001 # now this is `is_write_index: false` GET foo-000002 # now this is `is_write_index: true` ```
It would be nice to be able to look up other indices' aliases
while validating new aliases being introduced. For example,
If we would like to prohibit certain aliases and allow others,
depending on their respective definitions.
The problem with moving the construction of this lookup data-structure
to be iterative is that we do not have a clear view of all other possible
duplicate aliases. We would raise an error with the first duplicate we see.