Skip to content

Commit a6312a5

Browse files
committed
Merge branch '6.x' into ccr-6.x
* 6.x: Mute test watcher usage stats output [Rollup] Fix FullClusterRestart test TEST: Disable soft-deletes in ParentChildTestCase TEST: Disable randomized soft-deletes settings Integrates soft-deletes into Elasticsearch (#33222) drop `index.shard.check_on_startup: fix` (#32279) Fix AwaitsFix issue number Mute SmokeTestWatcherWithSecurityIT testsi [DOCS] Moves ml folder from x-pack/docs to docs (#33248) TEST: mute more SmokeTestWatcherWithSecurityIT tests [DOCS] Move rollup APIs to docs (#31450) [DOCS] Rename X-Pack Commands section (#33005) Fixes SecurityIntegTestCase so it always adds at least one alias (#33296) TESTS: Fix Random Fail in MockTcpTransportTests (#33061) (#33307) MINOR: Remove Dead Code from PathTrie (#33280) (#33306) Fix pom for build-tools (#33300) Lazy evaluate java9home (#33301) SQL: test coverage for JdbcResultSet (#32813) Work around to be able to generate eclipse projects (#33295) Different handling for security specific errors in the CLI. Fix for #33230 (#33255) [ML] Refactor delimited file structure detection (#33233) SQL: Support multi-index format as table identifier (#33278) Enable forbiddenapis server java9 (#33245) [MUTE] SmokeTestWatcherWithSecurityIT flaky tests Add region ISO code to GeoIP Ingest plugin (#31669) (#33276) Don't be strict for 6.x Update serialization versions for custom IndexMetaData backport Replace IndexMetaData.Custom with Map-based custom metadata (#32749) Painless: Fix Bindings Bug (#33274) SQL: prevent duplicate generation for repeated aggs (#33252) TEST: Mute testMonitorClusterHealth Fix serialization of empty field capabilities response (#33263) Fix nested _source retrieval with includes/excludes (#33180) [DOCS] TLS file resources are reloadable (#33258) Watcher: Ensure TriggerEngine start replaces existing watches (#33157) Ignore module-info in jar hell checks (#33011) Fix docs build after #33241 [DOC] Repository GCS ADC not supported (#33238) Upgrade to latest Gradle 4.10 (#32801) Fix/30904 cluster formation part2 (#32877) Move file-based discovery to core (#33241) HLRC: add client side RefreshPolicy (#33209) [Kerberos] Add unsupported languages for tests (#33253) Watcher: Reload properly on remote shard change (#33167) Fix classpath security checks for external tests. (#33066) [Rollup] Only allow aggregating on multiples of configured interval (#32052) Added deprecation warning for rescore in scroll queries (#33070) Apply settings filter to get cluster settings API (#33247) [Rollup] Re-factor Rollup Indexer into a generic indexer for re-usability (#32743) HLRC: create base timed request class (#33216) HLRC: Use Optional in validation logic (#33104) Painless: Add Bindings (#33042)
2 parents 1a71078 + b9c45ff commit a6312a5

File tree

272 files changed

+6856
-2478
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

272 files changed

+6856
-2478
lines changed

build.gradle

+12-5
Original file line numberDiff line numberDiff line change
@@ -16,22 +16,19 @@
1616
* specific language governing permissions and limitations
1717
* under the License.
1818
*/
19+
1920
import com.github.jengelman.gradle.plugins.shadow.ShadowPlugin
21+
import org.apache.tools.ant.taskdefs.condition.Os
2022
import org.elasticsearch.gradle.BuildPlugin
2123
import org.elasticsearch.gradle.LoggedExec
2224
import org.elasticsearch.gradle.Version
2325
import org.elasticsearch.gradle.VersionCollection
2426
import org.elasticsearch.gradle.VersionProperties
2527
import org.elasticsearch.gradle.plugin.PluginBuildPlugin
2628
import org.gradle.plugins.ide.eclipse.model.SourceFolder
27-
import org.gradle.util.GradleVersion
28-
import org.gradle.util.DistributionLocator
29-
import org.apache.tools.ant.taskdefs.condition.Os
30-
import org.apache.tools.ant.filters.ReplaceTokens
3129

3230
import java.nio.file.Files
3331
import java.nio.file.Path
34-
import java.security.MessageDigest
3532

3633
plugins {
3734
id 'com.gradle.build-scan' version '1.13.2'
@@ -516,6 +513,16 @@ allprojects {
516513
tasks.cleanEclipse.dependsOn(wipeEclipseSettings)
517514
// otherwise the eclipse merging is *super confusing*
518515
tasks.eclipse.dependsOn(cleanEclipse, copyEclipseSettings)
516+
517+
// work arround https://github.com/gradle/gradle/issues/6582
518+
tasks.eclipseProject.mustRunAfter tasks.cleanEclipseProject
519+
tasks.matching { it.name == 'eclipseClasspath' }.all {
520+
it.mustRunAfter { tasks.cleanEclipseClasspath }
521+
}
522+
tasks.matching { it.name == 'eclipseJdt' }.all {
523+
it.mustRunAfter { tasks.cleanEclipseJdt }
524+
}
525+
tasks.copyEclipseSettings.mustRunAfter tasks.wipeEclipseSettings
519526
}
520527

521528
allprojects {

buildSrc/build.gradle

+1-1
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
* under the License.
1818
*/
1919
import java.nio.file.Files
20+
import org.gradle.util.GradleVersion
2021

2122
plugins {
2223
id 'java-gradle-plugin'
@@ -166,7 +167,6 @@ if (project != rootProject) {
166167
it.tasks.matching { it.name == 'publishNebulaPublicationToLocalTestRepository'}
167168
}
168169
exclude "**/*Tests.class"
169-
include "**/*IT.class"
170170
testClassesDirs = sourceSets.test.output.classesDirs
171171
classpath = sourceSets.test.runtimeClasspath
172172
inputs.dir(file("src/testKit"))

buildSrc/src/main/groovy/org/elasticsearch/gradle/BuildPlugin.groovy

+2-1
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,6 @@ import org.gradle.api.artifacts.ModuleDependency
3838
import org.gradle.api.artifacts.ModuleVersionIdentifier
3939
import org.gradle.api.artifacts.ProjectDependency
4040
import org.gradle.api.artifacts.ResolvedArtifact
41-
import org.gradle.api.artifacts.SelfResolvingDependency
4241
import org.gradle.api.artifacts.dsl.RepositoryHandler
4342
import org.gradle.api.execution.TaskExecutionGraph
4443
import org.gradle.api.plugins.JavaPlugin
@@ -212,6 +211,7 @@ class BuildPlugin implements Plugin<Project> {
212211
project.rootProject.ext.minimumRuntimeVersion = minimumRuntimeVersion
213212
project.rootProject.ext.inFipsJvm = inFipsJvm
214213
project.rootProject.ext.gradleJavaVersion = JavaVersion.toVersion(gradleJavaVersion)
214+
project.rootProject.ext.java9Home = "${-> findJavaHome("9")}"
215215
}
216216

217217
project.targetCompatibility = project.rootProject.ext.minimumRuntimeVersion
@@ -225,6 +225,7 @@ class BuildPlugin implements Plugin<Project> {
225225
project.ext.javaVersions = project.rootProject.ext.javaVersions
226226
project.ext.inFipsJvm = project.rootProject.ext.inFipsJvm
227227
project.ext.gradleJavaVersion = project.rootProject.ext.gradleJavaVersion
228+
project.ext.java9Home = project.rootProject.ext.java9Home
228229
}
229230

230231
private static String getPaddedMajorVersion(JavaVersion compilerJavaVersionEnum) {

buildSrc/src/main/groovy/org/elasticsearch/gradle/precommit/JarHellTask.groovy

+6-2
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,8 @@ package org.elasticsearch.gradle.precommit
2222
import com.github.jengelman.gradle.plugins.shadow.ShadowPlugin
2323
import org.elasticsearch.gradle.LoggedExec
2424
import org.gradle.api.file.FileCollection
25+
import org.gradle.api.tasks.Classpath
2526
import org.gradle.api.tasks.OutputFile
26-
2727
/**
2828
* Runs CheckJarHell on a classpath.
2929
*/
@@ -35,9 +35,13 @@ public class JarHellTask extends LoggedExec {
3535
* inputs (ie the jars/class files).
3636
*/
3737
@OutputFile
38-
File successMarker = new File(project.buildDir, 'markers/jarHell')
38+
File successMarker
39+
40+
@Classpath
41+
FileCollection classpath
3942

4043
public JarHellTask() {
44+
successMarker = new File(project.buildDir, 'markers/jarHell-' + getName())
4145
project.afterEvaluate {
4246
FileCollection classpath = project.sourceSets.test.runtimeClasspath
4347
if (project.plugins.hasPlugin(ShadowPlugin)) {

buildSrc/src/main/groovy/org/elasticsearch/gradle/precommit/PrecommitTasks.groovy

+9-3
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ class PrecommitTasks {
3131

3232
/** Adds a precommit task, which depends on non-test verification tasks. */
3333
public static Task create(Project project, boolean includeDependencyLicenses) {
34-
Configuration forbiddenApisConfiguration = project.configurations.create("forbiddenApisCliJar")
34+
project.configurations.create("forbiddenApisCliJar")
3535
project.dependencies {
3636
forbiddenApisCliJar ('de.thetaphi:forbiddenapis:2.5')
3737
}
@@ -43,7 +43,7 @@ class PrecommitTasks {
4343
project.tasks.create('forbiddenPatterns', ForbiddenPatternsTask.class),
4444
project.tasks.create('licenseHeaders', LicenseHeadersTask.class),
4545
project.tasks.create('filepermissions', FilePermissionsTask.class),
46-
project.tasks.create('jarHell', JarHellTask.class),
46+
configureJarHell(project),
4747
configureThirdPartyAudit(project)
4848
]
4949

@@ -80,6 +80,12 @@ class PrecommitTasks {
8080
return project.tasks.create(precommitOptions)
8181
}
8282

83+
private static Task configureJarHell(Project project) {
84+
Task task = project.tasks.create('jarHell', JarHellTask.class)
85+
task.classpath = project.sourceSets.test.runtimeClasspath
86+
return task
87+
}
88+
8389
private static Task configureThirdPartyAudit(Project project) {
8490
ThirdPartyAuditTask thirdPartyAuditTask = project.tasks.create('thirdPartyAudit', ThirdPartyAuditTask.class)
8591
ExportElasticsearchBuildResourcesTask buildResources = project.tasks.getByName('buildResources')
@@ -94,7 +100,7 @@ class PrecommitTasks {
94100

95101
private static Task configureForbiddenApisCli(Project project) {
96102
Task forbiddenApisCli = project.tasks.create('forbiddenApis')
97-
project.sourceSets.forEach { sourceSet ->
103+
project.sourceSets.all { sourceSet ->
98104
forbiddenApisCli.dependsOn(
99105
project.tasks.create(sourceSet.getTaskName('forbiddenApis', null), ForbiddenApisCliTask) {
100106
ExportElasticsearchBuildResourcesTask buildResources = project.tasks.getByName('buildResources')
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
/*
2+
* Licensed to Elasticsearch under one or more contributor
3+
* license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright
5+
* ownership. Elasticsearch licenses this file to you under
6+
* the Apache License, Version 2.0 (the "License"); you may
7+
* not use this file except in compliance with the License.
8+
* You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
package org.elasticsearch;
20+
21+
import org.gradle.api.Action;
22+
import org.gradle.api.Project;
23+
import org.gradle.api.file.CopySpec;
24+
import org.gradle.api.file.FileTree;
25+
import org.gradle.api.tasks.WorkResult;
26+
import org.gradle.process.ExecResult;
27+
import org.gradle.process.JavaExecSpec;
28+
29+
import java.io.File;
30+
31+
/**
32+
* Facilitate access to Gradle services without a direct dependency on Project.
33+
*
34+
* In a future release Gradle will offer service injection, this adapter plays that role until that time.
35+
* It exposes the service methods that are part of the public API as the classes implementing them are not.
36+
* Today service injection is <a href="https://github.com/gradle/gradle/issues/2363">not available</a> for
37+
* extensions.
38+
*
39+
* Everything exposed here must be thread safe. That is the very reason why project is not passed in directly.
40+
*/
41+
public class GradleServicesAdapter {
42+
43+
public final Project project;
44+
45+
public GradleServicesAdapter(Project project) {
46+
this.project = project;
47+
}
48+
49+
public static GradleServicesAdapter getInstance(Project project) {
50+
return new GradleServicesAdapter(project);
51+
}
52+
53+
public WorkResult copy(Action<? super CopySpec> action) {
54+
return project.copy(action);
55+
}
56+
57+
public WorkResult sync(Action<? super CopySpec> action) {
58+
return project.sync(action);
59+
}
60+
61+
public ExecResult javaexec(Action<? super JavaExecSpec> action) {
62+
return project.javaexec(action);
63+
}
64+
65+
public FileTree zipTree(File zipPath) {
66+
return project.zipTree(zipPath);
67+
}
68+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
/*
2+
* Licensed to Elasticsearch under one or more contributor
3+
* license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright
5+
* ownership. Elasticsearch licenses this file to you under
6+
* the Apache License, Version 2.0 (the "License"); you may
7+
* not use this file except in compliance with the License.
8+
* You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
package org.elasticsearch.gradle;
20+
21+
public enum Distribution {
22+
23+
INTEG_TEST("integ-test-zip"),
24+
ZIP("zip"),
25+
ZIP_OSS("zip-oss");
26+
27+
private final String name;
28+
29+
Distribution(String name) {
30+
this.name = name;
31+
}
32+
33+
public String getName() {
34+
return name;
35+
}
36+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
/*
2+
* Licensed to Elasticsearch under one or more contributor
3+
* license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright
5+
* ownership. Elasticsearch licenses this file to you under
6+
* the Apache License, Version 2.0 (the "License"); you may
7+
* not use this file except in compliance with the License.
8+
* You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
package org.elasticsearch.gradle.clusterformation;
20+
21+
import groovy.lang.Closure;
22+
import org.elasticsearch.GradleServicesAdapter;
23+
import org.gradle.api.NamedDomainObjectContainer;
24+
import org.gradle.api.Plugin;
25+
import org.gradle.api.Project;
26+
import org.gradle.api.Task;
27+
import org.gradle.api.execution.TaskActionListener;
28+
import org.gradle.api.execution.TaskExecutionListener;
29+
import org.gradle.api.logging.Logger;
30+
import org.gradle.api.logging.Logging;
31+
import org.gradle.api.plugins.ExtraPropertiesExtension;
32+
import org.gradle.api.tasks.TaskState;
33+
34+
import java.util.ArrayList;
35+
import java.util.Collections;
36+
import java.util.HashMap;
37+
import java.util.List;
38+
import java.util.Map;
39+
40+
public class ClusterformationPlugin implements Plugin<Project> {
41+
42+
public static final String LIST_TASK_NAME = "listElasticSearchClusters";
43+
public static final String EXTENSION_NAME = "elasticSearchClusters";
44+
45+
private final Logger logger = Logging.getLogger(ClusterformationPlugin.class);
46+
47+
@Override
48+
public void apply(Project project) {
49+
NamedDomainObjectContainer<? extends ElasticsearchConfiguration> container = project.container(
50+
ElasticsearchNode.class,
51+
(name) -> new ElasticsearchNode(name, GradleServicesAdapter.getInstance(project))
52+
);
53+
project.getExtensions().add(EXTENSION_NAME, container);
54+
55+
Task listTask = project.getTasks().create(LIST_TASK_NAME);
56+
listTask.setGroup("ES cluster formation");
57+
listTask.setDescription("Lists all ES clusters configured for this project");
58+
listTask.doLast((Task task) ->
59+
container.forEach((ElasticsearchConfiguration cluster) ->
60+
logger.lifecycle(" * {}: {}", cluster.getName(), cluster.getDistribution())
61+
)
62+
);
63+
64+
Map<Task, List<ElasticsearchConfiguration>> taskToCluster = new HashMap<>();
65+
66+
// register an extension for all current and future tasks, so that any task can declare that it wants to use a
67+
// specific cluster.
68+
project.getTasks().all((Task task) ->
69+
task.getExtensions().findByType(ExtraPropertiesExtension.class)
70+
.set(
71+
"useCluster",
72+
new Closure<Void>(this, this) {
73+
public void doCall(ElasticsearchConfiguration conf) {
74+
taskToCluster.computeIfAbsent(task, k -> new ArrayList<>()).add(conf);
75+
}
76+
})
77+
);
78+
79+
project.getGradle().getTaskGraph().whenReady(taskExecutionGraph ->
80+
taskExecutionGraph.getAllTasks()
81+
.forEach(task ->
82+
taskToCluster.getOrDefault(task, Collections.emptyList()).forEach(ElasticsearchConfiguration::claim)
83+
)
84+
);
85+
project.getGradle().addListener(
86+
new TaskActionListener() {
87+
@Override
88+
public void beforeActions(Task task) {
89+
// we only start the cluster before the actions, so we'll not start it if the task is up-to-date
90+
taskToCluster.getOrDefault(task, new ArrayList<>()).forEach(ElasticsearchConfiguration::start);
91+
}
92+
@Override
93+
public void afterActions(Task task) {}
94+
}
95+
);
96+
project.getGradle().addListener(
97+
new TaskExecutionListener() {
98+
@Override
99+
public void afterExecute(Task task, TaskState state) {
100+
// always un-claim the cluster, even if _this_ task is up-to-date, as others might not have been and caused the
101+
// cluster to start.
102+
taskToCluster.getOrDefault(task, new ArrayList<>()).forEach(ElasticsearchConfiguration::unClaimAndStop);
103+
}
104+
@Override
105+
public void beforeExecute(Task task) {}
106+
}
107+
);
108+
}
109+
110+
}

0 commit comments

Comments
 (0)