Skip to content

Use alias name from rollover request to query indices stats #40774

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Apr 16, 2019

Conversation

bizybot
Copy link
Contributor

@bizybot bizybot commented Apr 3, 2019

In TransportRolloverAction before doing rollover we resolve
source index name (write index) from the alias in the rollover request.
Before evaluating the conditions and executing rollover action, we
retrieve stats, but to do so we used the source index name
resolved from the alias instead of alias from the index.
This fails when the user is assigned a role with index privilege on the
alias instead of the concrete index. This commit fixes this by using
the alias from the request.

Closes #40771

In `TransportRolloverAction` before doing rollover we resolve
source index name (write index) from the alias in the rollover request.
Before evaluating the conditions and executing rollover action, we
retrieve stats, but to do so we used the source index name
resolved from the alias instead of what is in the request.
This fails when user is assigned role with index privilege on the
alias instead of concrete index. This commit fixes this by using
the alias from the request.

Closes elastic#40771
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security

@bizybot bizybot added the :Data Management/Indices APIs APIs to create and manage indices and templates label Apr 3, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

Copy link
Contributor

@albertzaharovits albertzaharovits left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am game with the main code change.
I would love if you'd add an IntegTest so that we don't catch authorization errors in Rest tests...

However, I think we should make the Rollover action use the client for index and alias creation. WDYT?

Yogesh Gaikwad added 2 commits April 4, 2019 18:30
When same alias points to multiple indices we can write to only one index
with `is_write_index` value `true`. The special handling for
PutMappingRequest filtered out such aliases making the request unauthorized.
The check has been modified to consider write index flag and only when the
requested index matches with the one with write index alias.

Closes elastic#40831
@bizybot
Copy link
Contributor Author

bizybot commented Apr 4, 2019

Hi @albertzaharovits, Thanks for your comment

I am game with the main code change.
I would love if you'd add an IntegTest so that we don't catch authorization errors in Rest tests...

I did not see one for Rollover Action but I can add unit test to cover this. Thanks.

However, I think we should make the Rollover action use the client for index and alias creation. WDYT?

Could you please elaborate? as I did not understand what you have proposed here. Thank you.

@albertzaharovits
Copy link
Contributor

Ah, I'm sorry, I mean createIndexService.createIndex to be replaced with client.indices().create() . I think the user of the rollup action should necessitate the create index privilege, at least this is my perception of the authorization with "action granularity".
This is a discussion suggestion for an issue as a follow-up, not to be addressed here.

@tvernum
Copy link
Contributor

tvernum commented Apr 5, 2019

@talevy Are you able to review the (1 line) change to the transport action? It looks like you were the last person to make actual functional changes to that action, and I'd appreciate a review from someone who knows that code better than we do.

@tvernum tvernum requested a review from talevy April 5, 2019 02:17
Copy link
Contributor

@talevy talevy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still need to test my comments, but I figure sending this out earlier is better. let me know if
what I said makes sense!

Copy link
Contributor

@talevy talevy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just ran this check to assert my theory:

PUT alpha
{
  "aliases": {
    "my_alias": {
      "is_write_index": true
    }
  }
}

PUT beta
{
  "aliases": {
    "my_alias": {
      "is_write_index": false
    }
  }
}

PUT alpha/_doc/1?refresh
{
  "foo": "bar"
}

PUT beta/_doc/1?refresh
{
  "foo": "bar"
}

POST /my_alias/_rollover/new_index?dry_run
{
  "conditions": {
    "max_docs":  2
  }
}

this returns an incorrect rollover evaluation (the write index "alpha" only has one document):

{
  "acknowledged" : false,
  "shards_acknowledged" : false,
  "old_index" : "alpha",
  "new_index" : "new_index",
  "rolled_over" : false,
  "dry_run" : true,
  "conditions" : {
    "[max_docs: 2]" : true
  }
}

I think the additional change that would make this work would be this:

@@ -249,7 +249,7 @@ public class TransportRolloverAction extends TransportMasterNodeAction<RolloverR
 
     static Map<String, Boolean> evaluateConditions(final Collection<Condition<?>> conditions, final IndexMetaData metaData,
                                                     final IndicesStatsResponse statsResponse) {
-        return evaluateConditions(conditions, statsResponse.getPrimaries().getDocs(), metaData);
+        return evaluateConditions(conditions, statsResponse.getIndex(metaData.getIndex().getName()).getPrimaries().getDocs(), metaData);
     }

The issue is that the code was looking at the total DocStats of all the indices returned in the stats-response. This change should make it so that we are looking at the sourceIndex only. The non-docstats condition (index age) is done on the correct IndexMetaData of the sourceIndex, so this should not be a problem.

When alias is used, we retrieve all the stats (including write + read indexes)
so the earlier commit, made the stats evaluation on collective stats instead
of considering only source index.
Copy link
Contributor

@talevy talevy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes to Rollover look good to me. I'm not against
getting another opinion regarding my comments around
performance concerns.

Due to the difficulty of controlling time for the additional test in RolloverIT,
what do you think of moving these additional tests as unit tests
in TransportRolloverActionTests. That way we can control the
state of the indices such that only the write-index meets the
relevant criteria (max_age, max_docs, max_size).

Copy link
Contributor

@talevy talevy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@polyfractal polyfractal removed the v7.0.0 label Apr 9, 2019
Copy link
Contributor

@tvernum tvernum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bizybot bizybot merged commit 3c66cff into elastic:master Apr 16, 2019
bizybot added a commit to bizybot/elasticsearch that referenced this pull request Apr 17, 2019
…40774)

In `TransportRolloverAction` before doing rollover we resolve
source index name (write index) from the alias in the rollover request.
Before evaluating the conditions and executing rollover action, we
retrieve stats, but to do so we used the source index name
resolved from the alias instead of alias from the index.
This fails when the user is assigned a role with index privilege on the
alias instead of the concrete index. This commit fixes this by using
the alias from the request.
After this change, verified that when we retrieve all the stats (including write + read indexes)
we are considering only source index.

Closes elastic#40771
bizybot added a commit to bizybot/elasticsearch that referenced this pull request Apr 17, 2019
…40774)

In `TransportRolloverAction` before doing rollover we resolve
source index name (write index) from the alias in the rollover request.
Before evaluating the conditions and executing rollover action, we
retrieve stats, but to do so we used the source index name
resolved from the alias instead of alias from the index.
This fails when the user is assigned a role with index privilege on the
alias instead of the concrete index. This commit fixes this by using
the alias from the request.
After this change, verified that when we retrieve all the stats (including write + read indexes)
we are considering only source index.

Closes elastic#40771
@bizybot bizybot added the v7.0.1 label Apr 17, 2019
bizybot added a commit that referenced this pull request Apr 17, 2019
…41284)

In `TransportRolloverAction` before doing rollover we resolve
source index name (write index) from the alias in the rollover request.
Before evaluating the conditions and executing rollover action, we
retrieve stats, but to do so we used the source index name
resolved from the alias instead of alias from the index.
This fails when the user is assigned a role with index privilege on the
alias instead of the concrete index. This commit fixes this by using
the alias from the request.
After this change, verified that when we retrieve all the stats (including write + read indexes)
we are considering only source index.

Closes #40771
bizybot added a commit that referenced this pull request Apr 17, 2019
…41285)

In `TransportRolloverAction` before doing rollover we resolve
source index name (write index) from the alias in the rollover request.
Before evaluating the conditions and executing rollover action, we
retrieve stats, but to do so we used the source index name
resolved from the alias instead of alias from the index.
This fails when the user is assigned a role with index privilege on the
alias instead of the concrete index. This commit fixes this by using
the alias from the request.
After this change, verified that when we retrieve all the stats (including write + read indexes)
we are considering only source index.

Closes #40771
bizybot added a commit to bizybot/elasticsearch that referenced this pull request Apr 17, 2019
…40774)

In `TransportRolloverAction` before doing rollover we resolve
source index name (write index) from the alias in the rollover request.
Before evaluating the conditions and executing rollover action, we
retrieve stats, but to do so we used the source index name
resolved from the alias instead of alias from the index.
This fails when the user is assigned a role with index privilege on the
alias instead of the concrete index. This commit fixes this by using
the alias from the request.
After this change, verified that when we retrieve all the stats (including write + read indexes)
we are considering only source index.

Closes elastic#40771
@colings86 colings86 removed the :Data Management/Indices APIs APIs to create and manage indices and templates label Apr 17, 2019
bizybot added a commit that referenced this pull request Apr 17, 2019
…41286)

In `TransportRolloverAction` before doing rollover we resolve
source index name (write index) from the alias in the rollover request.
Before evaluating the conditions and executing rollover action, we
retrieve stats, but to do so we used the source index name
resolved from the alias instead of alias from the index.
This fails when the user is assigned a role with index privilege on the
alias instead of the concrete index. This commit fixes this by using
the alias from the request.
After this change, verified that when we retrieve all the stats (including write + read indexes)
we are considering only source index.

Closes #40771
gurkankaymak pushed a commit to gurkankaymak/elasticsearch that referenced this pull request May 27, 2019
…40774)

In `TransportRolloverAction` before doing rollover we resolve
source index name (write index) from the alias in the rollover request.
Before evaluating the conditions and executing rollover action, we
retrieve stats, but to do so we used the source index name
resolved from the alias instead of alias from the index.
This fails when the user is assigned a role with index privilege on the
alias instead of the concrete index. This commit fixes this by using
the alias from the request.
After this change, verified that when we retrieve all the stats (including write + read indexes)
we are considering only source index.

Closes elastic#40771
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rollover action fails when user has index privilege only on alias
8 participants