Convert repository-hdfs to testclusters #41252
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alpar-t
added
>non-issue
:Delivery/Build
|
Pinging @elastic/es-core-infra |
mark-vieira
reviewed
Apr 17, 2019
| @@ -20,11 +20,7 @@ | |||
| // only configure immediate children of plugins dir | |||
| configure(subprojects.findAll { it.parent.path == project.path }) { | |||
| group = 'org.elasticsearch.plugin' | |||
| // TODO exclude some plugins as they require features not yet supproted by testclusters | |||
mark-vieira
approved these changes
Apr 17, 2019
| // Create a Integration Test suite just for security based tests | ||
| RestIntegTestTask integTestSecure = project.tasks.create('integTestSecure', RestIntegTestTask.class) { | ||
| description = "Runs rest tests against an elasticsearch cluster with HDFS secured by MIT Kerberos." | ||
| testClusters."${integTestTaskName}" { |
There was a problem hiding this comment.
If the two "secure" clusters are identical, is there any need to declare both of them? Could we get away with two test clusters, secure and non-secure and have the secure integration tests both use the same cluster?
There was a problem hiding this comment.
I don't think they end up being identical, one is HA and the other non HA, but even if they are, I don't want to got that far in this PR.
There was a problem hiding this comment.
Yeah, I guess my question is are they actually different if we don't configure them differently? Or at least it's not obvious to me that they are different. I'm ok with them being separate. Just thought it might be a slight optimization to not have another cluster we need to setup/teardown.
alpar-t
added a commit
that referenced
this pull request
Apr 19, 2019
* Convert repository-hdfs to testclusters Relates #40862
gurkankaymak
pushed a commit
to gurkankaymak/elasticsearch
that referenced
this pull request
May 27, 2019
* Convert repository-hdfs to testclusters Relates elastic#40862
bizybot
pushed a commit
to bizybot/elasticsearch
that referenced
this pull request
Jul 7, 2019
Due to recent changes done for converting `repository-hdfs` to test clusters (elastic#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of fixture to the task. The `secureHdfsFixture` is an AntFixture which is spawned process. Internally it waits for 30 seconds for the resources to be made available. For my local machine it took almost 45 seconds to be available so I have added the wait time as an input to the AntFixture defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. Another problem while running the `secureHdfsFixture` where it would fail due to port not being privileged port (i.e system port, port < 1024). By default datanode address key tries to find a free port and this would later on fail in case it is running in a secure mode. To address this in case of secure mode we find free port below 1024 and set it in the config. The config `DFSConfigKeys.IGNORE_SECURE_PORTS_FOR_TESTING_KEY` is set to `true` but it did not help. https://fisheye.apache.org/browse/~br=branch-2.8.1/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/SecureDataNodeStarter.java?hb=true#to140 The integ test for secure hdfs were disabled for long time and so the changes done in elastic#42090 to fix the tests are also done in this commit.
bizybot
added a commit
that referenced
this pull request
Jul 10, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in #42090 to fix the tests are also done in this commit.
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (elastic#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in elastic#42090 to fix the tests are also done in this commit.
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (elastic#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in elastic#42090 to fix the tests are also done in this commit.
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (elastic#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in elastic#42090 to fix the tests are also done in this commit.
bizybot
added a commit
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in #42090 to fix the tests are also done in this commit.
bizybot
added a commit
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in #42090 to fix the tests are also done in this commit.
bizybot
added a commit
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in #42090 to fix the tests are also done in this commit.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relates #40862