Test clusters ( was clusterformation ) - part 3 #33264
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- This allows to move all all .java files from .groovy. - Will prevent eclipse from tangling up in this setup - make it possible to use Version from Java
``` > Task :plugins:ingest-user-agent:listElasticSearchClusters Starting cluster: myTestCluster * myTestCluster: /home/alpar/work/elastic/elasticsearch/plugins/ingest-user-agent/foo Asked to unClaimAndStop myTestCluster, since cluster still has 1 claim it will not be stopped > Task :plugins:ingest-user-agent:testme UP-TO-DATE Stopping myTestCluster, since no of claims is 0 ``` - Meant to auto manage the clusters lifecycle - Add integration test for cluster formation
``` > Task :plugins:ingest-user-agent:listElasticSearchClusters Starting cluster: myTestCluster * myTestCluster: /home/alpar/work/elastic/elasticsearch/plugins/ingest-user-agent/foo Asked to unClaimAndStop myTestCluster, since cluster still has 1 claim it will not be stopped > Task :plugins:ingest-user-agent:testme UP-TO-DATE Stopping myTestCluster, since no of claims is 0 ``` - Meant to auto manage the clusters lifecycle - Add integration test for cluster formation
Without this the tasks from different version can be ordered randomly causing the snapshot test to fail ( i.e. creating snapshot for multiple versions in the same location ). Gradle happens to order them correctly when ran without "parallel" but switches up the ordering when not.
Rather than fighting it, move to do it right before the runner
And check resulting artifact when starting up the cluster
Seems this hoggs some internal Gradle file locks, and messes with Gradle's internals.
To be able to use process builder, the min java version for build was bumped to 9.
This way building buildSrc will not be delayed for all invocations, and will be less disruptive if integration tests fail
- support for multi project when not all projects apply the plugin.
- add a test for it
- only register task liseners once per build ( not once per project )
- make the node name safe for paths and cluster name, even if it has `:` or `#` ( will be replaced by `-`)
Clean up the processes after even after `^C` is pressed. The way Gradle does this is depending on the use of the Daemon, but for us, even with the Daemon interupting it does not work as it should and it eventually self terminates. The downside is that the ES processes will be cleaned up with a delay caused by Gradle trying and failing to clean up the Daemon.
Allows to pass in closures that would evaluate at execution time.
…er-formation-part3
cmd.exe on windows does not close it's children, so we clean up everything.
alpar-t
added
>non-issue
:Delivery/Build
|
Pinging @elastic/es-core-infra |
|
@elasticmachine test this please |
rjernst
reviewed
Sep 27, 2018
There was a problem hiding this comment.
I stared having a look at this, but the sheer size makes it difficult to review. Here are some general suggestions to start:
- Please move unrelated changes out to separate PRs. For example, subproject -> allprojects in root build.gradle (although I don't understand why this is needed), or java 9 changes
- Do renames in separate PRs. Moving files that show up as full delete and full add makes files almost impossible to review well since everything shows as new.
- I have the same concerns I had in the last PR. I think all state management for the clusters should stay in the plugin, and the ElasticsearchNode should be for configuration only.
|
I'm closing this and opening separate PRs. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change focuses on the process creation and management aspect of test clusters for a single node cluster.
The goal is to be able to start using this for plugins and modules that start a single node cluster, thus the change in the plugin builder plugin to toggle it.
I was able to test this for a few actual projects. To be able to switch over to using this for plugins we will need to port the plugin install functionality.
CI changes are required so that Gradle is always ran with java > 9 and thus :buildSrc can be built with java 9 support.
This should not be a problem for external plugin authors as we already require Java 10.
Relates to #30904