Skip to content

Commit 73ea5aa

Browse files
author
Hendrik Muhs
committed
Merge branch 'master' of github.com:elastic/elasticsearch into feature/fib
2 parents 9c59ae1 + 9dcf3f5 commit 73ea5aa

File tree

1,026 files changed

+28512
-6853
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

1,026 files changed

+28512
-6853
lines changed

.ci/matrix-runtime-javas.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,5 +7,6 @@
77

88
ES_RUNTIME_JAVA:
99
- java8
10+
- java8fips
1011
- java10
1112
- java11

CONTRIBUTING.md

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,12 @@ JDK 10 and testing on a JDK 8 runtime; to do this, set `RUNTIME_JAVA_HOME`
100100
pointing to the Java home of a JDK 8 installation. Note that this mechanism can
101101
be used to test against other JDKs as well, this is not only limited to JDK 8.
102102

103+
> Note: It is also required to have `JAVA7_HOME`, `JAVA8_HOME` and
104+
`JAVA10_HOME` available so that the tests can pass.
105+
106+
> Warning: do not use `sdkman` for Java installations which do not have proper
107+
`jrunscript` for jdk distributions.
108+
103109
Elasticsearch uses the Gradle wrapper for its build. You can execute Gradle
104110
using the wrapper via the `gradlew` script in the root of the repository.
105111

@@ -214,7 +220,7 @@ If your changes affect only the documentation, run:
214220
```sh
215221
./gradlew -p docs check
216222
```
217-
For more information about testing code examples in the documentation, see
223+
For more information about testing code examples in the documentation, see
218224
https://github.com/elastic/elasticsearch/blob/master/docs/README.asciidoc
219225

220226
### Project layout
@@ -305,6 +311,39 @@ the `qa` subdirectory functions just like the top level `qa` subdirectory. The
305311
Elasticsearch process. The `transport-client` subdirectory contains extensions
306312
to Elasticsearch's standard transport client to work properly with x-pack.
307313

314+
### Gradle Build
315+
316+
We use Gradle to build Elasticsearch because it is flexible enough to not only
317+
build and package Elasticsearch, but also orchestrate all of the ways that we
318+
have to test Elasticsearch.
319+
320+
#### Configurations
321+
322+
Gradle organizes dependencies and build artifacts into "configurations" and
323+
allows you to use these configurations arbitrarilly. Here are some of the most
324+
common configurations in our build and how we use them:
325+
326+
<dl>
327+
<dt>`compile`</dt><dd>Code that is on the classpath at both compile and
328+
runtime. If the [`shadow`][shadow-plugin] plugin is applied to the project then
329+
this code is bundled into the jar produced by the project.</dd>
330+
<dt>`runtime`</dt><dd>Code that is not on the classpath at compile time but is
331+
on the classpath at runtime. We mostly use this configuration to make sure that
332+
we do not accidentally compile against dependencies of our dependencies also
333+
known as "transitive" dependencies".</dd>
334+
<dt>`compileOnly`</dt><dd>Code that is on the classpath at comile time but that
335+
should not be shipped with the project because it is "provided" by the runtime
336+
somehow. Elasticsearch plugins use this configuration to include dependencies
337+
that are bundled with Elasticsearch's server.</dd>
338+
<dt>`shadow`</dt><dd>Only available in projects with the shadow plugin. Code
339+
that is on the classpath at both compile and runtime but it *not* bundled into
340+
the jar produced by the project. If you depend on a project with the `shadow`
341+
plugin then you need to depend on this configuration because it will bring
342+
along all of the dependencies you need at runtime.</dd>
343+
<dt>`testCompile`</dt><dd>Code that is on the classpath for compiling tests
344+
that are part of this project but not production code. The canonical example
345+
of this is `junit`.</dd>
346+
</dl>
308347

309348
Contributing as part of a class
310349
-------------------------------
@@ -337,3 +376,4 @@ repeating in this section because it has come up in this context.
337376

338377
[eclipse]: http://www.eclipse.org/community/eclipse_newsletter/2017/june/
339378
[intellij]: https://blog.jetbrains.com/idea/2017/07/intellij-idea-2017-2-is-here-smart-sleek-and-snappy/
379+
[shadow-plugin]: https://github.com/johnrengelman/shadow

README.textile

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -209,10 +209,6 @@ The distribution for each project will be created under the @build/distributions
209209

210210
See the "TESTING":TESTING.asciidoc file for more information about running the Elasticsearch test suite.
211211

212-
h3. Upgrading from Elasticsearch 1.x?
212+
h3. Upgrading from older Elasticsearch versions
213213

214-
In order to ensure a smooth upgrade process from earlier versions of
215-
Elasticsearch (1.x), it is required to perform a full cluster restart. Please
216-
see the "setup reference":
217-
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html
218-
for more details on the upgrade process.
214+
In order to ensure a smooth upgrade process from earlier versions of Elasticsearch, please see our "upgrade documentation":https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html for more details on the upgrade process.

Vagrantfile

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -115,11 +115,6 @@ Vagrant.configure(2) do |config|
115115
'opensuse-42'.tap do |box|
116116
config.vm.define box, define_opts do |config|
117117
config.vm.box = 'elastic/opensuse-42-x86_64'
118-
119-
# https://github.com/elastic/elasticsearch/issues/30295
120-
config.vm.provider 'virtualbox' do |vbox|
121-
vbox.customize ['storagectl', :id, '--name', 'SATA Controller', '--hostiocache', 'on']
122-
end
123118
suse_common config, box
124119
end
125120
end

benchmarks/README.md

Lines changed: 20 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -4,36 +4,39 @@ This directory contains the microbenchmark suite of Elasticsearch. It relies on
44

55
## Purpose
66

7-
We do not want to microbenchmark everything but the kitchen sink and should typically rely on our
8-
[macrobenchmarks](https://elasticsearch-benchmarks.elastic.co/app/kibana#/dashboard/Nightly-Benchmark-Overview) with
9-
[Rally](http://github.com/elastic/rally). Microbenchmarks are intended to spot performance regressions in performance-critical components.
7+
We do not want to microbenchmark everything but the kitchen sink and should typically rely on our
8+
[macrobenchmarks](https://elasticsearch-benchmarks.elastic.co/app/kibana#/dashboard/Nightly-Benchmark-Overview) with
9+
[Rally](http://github.com/elastic/rally). Microbenchmarks are intended to spot performance regressions in performance-critical components.
1010
The microbenchmark suite is also handy for ad-hoc microbenchmarks but please remove them again before merging your PR.
1111

1212
## Getting Started
1313

14-
Just run `gradle :benchmarks:jmh` from the project root directory. It will build all microbenchmarks, execute them and print the result.
14+
Just run `gradlew -p benchmarks run` from the project root
15+
directory. It will build all microbenchmarks, execute them and print
16+
the result.
1517

1618
## Running Microbenchmarks
1719

18-
Benchmarks are always run via Gradle with `gradle :benchmarks:jmh`.
19-
20-
Running via an IDE is not supported as the results are meaningless (we have no control over the JVM running the benchmarks).
20+
Running via an IDE is not supported as the results are meaningless
21+
because we have no control over the JVM running the benchmarks.
2122

22-
If you want to run a specific benchmark class, e.g. `org.elasticsearch.benchmark.MySampleBenchmark` or have special requirements
23-
generate the uberjar with `gradle :benchmarks:jmhJar` and run it directly with:
23+
If you want to run a specific benchmark class like, say,
24+
`MemoryStatsBenchmark`, you can use `--args`:
2425

2526
```
26-
java -jar benchmarks/build/distributions/elasticsearch-benchmarks-*.jar
27+
gradlew -p benchmarks run --args ' MemoryStatsBenchmark'
2728
```
2829

29-
JMH supports lots of command line parameters. Add `-h` to the command above to see the available command line options.
30+
Everything in the `'` gets sent on the command line to JMH. The leading ` `
31+
inside the `'`s is important. Without it parameters are sometimes sent to
32+
gradle.
3033

3134
## Adding Microbenchmarks
3235

33-
Before adding a new microbenchmark, make yourself familiar with the JMH API. You can check our existing microbenchmarks and also the
36+
Before adding a new microbenchmark, make yourself familiar with the JMH API. You can check our existing microbenchmarks and also the
3437
[JMH samples](http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/).
3538

36-
In contrast to tests, the actual name of the benchmark class is not relevant to JMH. However, stick to the naming convention and
39+
In contrast to tests, the actual name of the benchmark class is not relevant to JMH. However, stick to the naming convention and
3740
end the class name of a benchmark with `Benchmark`. To have JMH execute a benchmark, annotate the respective methods with `@Benchmark`.
3841

3942
## Tips and Best Practices
@@ -42,15 +45,15 @@ To get realistic results, you should exercise care when running benchmarks. Here
4245

4346
### Do
4447

45-
* Ensure that the system executing your microbenchmarks has as little load as possible. Shutdown every process that can cause unnecessary
48+
* Ensure that the system executing your microbenchmarks has as little load as possible. Shutdown every process that can cause unnecessary
4649
runtime jitter. Watch the `Error` column in the benchmark results to see the run-to-run variance.
4750
* Ensure to run enough warmup iterations to get the benchmark into a stable state. If you are unsure, don't change the defaults.
4851
* Avoid CPU migrations by pinning your benchmarks to specific CPU cores. On Linux you can use `taskset`.
49-
* Fix the CPU frequency to avoid Turbo Boost from kicking in and skewing your results. On Linux you can use `cpufreq-set` and the
52+
* Fix the CPU frequency to avoid Turbo Boost from kicking in and skewing your results. On Linux you can use `cpufreq-set` and the
5053
`performance` CPU governor.
5154
* Vary the problem input size with `@Param`.
5255
* Use the integrated profilers in JMH to dig deeper if benchmark results to not match your hypotheses:
53-
* Run the generated uberjar directly and use `-prof gc` to check whether the garbage collector runs during a microbenchmarks and skews
56+
* Run the generated uberjar directly and use `-prof gc` to check whether the garbage collector runs during a microbenchmarks and skews
5457
your results. If so, try to force a GC between runs (`-gc true`) but watch out for the caveats.
5558
* Use `-prof perf` or `-prof perfasm` (both only available on Linux) to see hotspots.
5659
* Have your benchmarks peer-reviewed.
@@ -59,4 +62,4 @@ To get realistic results, you should exercise care when running benchmarks. Here
5962

6063
* Blindly believe the numbers that your microbenchmark produces but verify them by measuring e.g. with `-prof perfasm`.
6164
* Run more threads than your number of CPU cores (in case you run multi-threaded microbenchmarks).
62-
* Look only at the `Score` column and ignore `Error`. Instead take countermeasures to keep `Error` low / variance explainable.
65+
* Look only at the `Score` column and ignore `Error`. Instead take countermeasures to keep `Error` low / variance explainable.

benchmarks/build.gradle

Lines changed: 3 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,8 @@
1818
*/
1919

2020
apply plugin: 'elasticsearch.build'
21-
22-
// order of this section matters, see: https://github.com/johnrengelman/shadow/issues/336
23-
apply plugin: 'application' // have the shadow plugin provide the runShadow task
21+
apply plugin: 'application'
2422
mainClassName = 'org.openjdk.jmh.Main'
25-
apply plugin: 'com.github.johnrengelman.shadow' // build an uberjar with all benchmarks
2623

2724
// Not published so no need to assemble
2825
tasks.remove(assemble)
@@ -50,10 +47,8 @@ compileJava.options.compilerArgs << "-Xlint:-cast,-deprecation,-rawtypes,-try,-u
5047
// needs to be added separately otherwise Gradle will quote it and javac will fail
5148
compileJava.options.compilerArgs.addAll(["-processor", "org.openjdk.jmh.generators.BenchmarkProcessor"])
5249

53-
forbiddenApis {
54-
// classes generated by JMH can use all sorts of forbidden APIs but we have no influence at all and cannot exclude these classes
55-
ignoreFailures = true
56-
}
50+
// classes generated by JMH can use all sorts of forbidden APIs but we have no influence at all and cannot exclude these classes
51+
forbiddenApisMain.enabled = false
5752

5853
// No licenses for our benchmark deps (we don't ship benchmarks)
5954
dependencyLicenses.enabled = false
@@ -69,20 +64,3 @@ thirdPartyAudit.excludes = [
6964
'org.openjdk.jmh.profile.HotspotRuntimeProfiler',
7065
'org.openjdk.jmh.util.Utils'
7166
]
72-
73-
runShadow {
74-
executable = new File(project.runtimeJavaHome, 'bin/java')
75-
}
76-
77-
// alias the shadowJar and runShadow tasks to abstract from the concrete plugin that we are using and provide a more consistent interface
78-
task jmhJar(
79-
dependsOn: shadowJar,
80-
description: 'Generates an uberjar with the microbenchmarks and all dependencies',
81-
group: 'Benchmark'
82-
)
83-
84-
task jmh(
85-
dependsOn: runShadow,
86-
description: 'Runs all microbenchmarks',
87-
group: 'Benchmark'
88-
)

benchmarks/src/main/resources/log4j2.properties

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
appender.console.type = Console
22
appender.console.name = console
33
appender.console.layout.type = PatternLayout
4-
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%m%n
4+
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c] [%node_name]%marker %m%n
55

66
# Do not log at all if it is not really critical - we're in a benchmark
77
rootLogger.level = error

build.gradle

Lines changed: 32 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -16,21 +16,17 @@
1616
* specific language governing permissions and limitations
1717
* under the License.
1818
*/
19-
2019
import com.github.jengelman.gradle.plugins.shadow.ShadowPlugin
21-
import org.apache.tools.ant.taskdefs.condition.Os
22-
import org.apache.tools.ant.filters.ReplaceTokens
2320
import org.elasticsearch.gradle.BuildPlugin
2421
import org.elasticsearch.gradle.LoggedExec
2522
import org.elasticsearch.gradle.Version
2623
import org.elasticsearch.gradle.VersionCollection
2724
import org.elasticsearch.gradle.VersionProperties
2825
import org.gradle.plugins.ide.eclipse.model.SourceFolder
29-
30-
import org.gradle.api.tasks.wrapper.Wrapper
31-
import org.gradle.api.tasks.wrapper.Wrapper.DistributionType
3226
import org.gradle.util.GradleVersion
3327
import org.gradle.util.DistributionLocator
28+
import org.apache.tools.ant.taskdefs.condition.Os
29+
import org.apache.tools.ant.filters.ReplaceTokens
3430

3531
import java.nio.file.Files
3632
import java.nio.file.Path
@@ -516,6 +512,31 @@ allprojects {
516512
tasks.eclipse.dependsOn(cleanEclipse, copyEclipseSettings)
517513
}
518514

515+
allprojects {
516+
/*
517+
* IntelliJ and Eclipse don't know about the shadow plugin so when we're
518+
* in "IntelliJ mode" or "Eclipse mode" add "runtime" dependencies
519+
* eveywhere where we see a "shadow" dependency which will cause them to
520+
* reference shadowed projects directly rather than rely on the shadowing
521+
* to include them. This is the correct thing for it to do because it
522+
* doesn't run the jar shadowing at all. This isn't needed for the project
523+
* itself because the IDE configuration is done by SourceSets but it is
524+
* *is* needed for projects that depends on the project doing the shadowing.
525+
* Without this they won't properly depend on the shadowed project.
526+
*/
527+
if (isEclipse || isIdea) {
528+
configurations.all { Configuration configuration ->
529+
dependencies.all { Dependency dep ->
530+
if (dep instanceof ProjectDependency) {
531+
if (dep.getTargetConfiguration() == 'shadow') {
532+
configuration.dependencies.add(project.dependencies.project(path: dep.dependencyProject.path, configuration: 'runtime'))
533+
}
534+
}
535+
}
536+
}
537+
}
538+
}
539+
519540
// we need to add the same --debug-jvm option as
520541
// the real RunTask has, so we can pass it through
521542
class Run extends DefaultTask {
@@ -537,7 +558,7 @@ task run(type: Run) {
537558
}
538559

539560
wrapper {
540-
distributionType = DistributionType.ALL
561+
distributionType = 'ALL'
541562
doLast {
542563
final DistributionLocator locator = new DistributionLocator()
543564
final GradleVersion version = GradleVersion.version(wrapper.gradleVersion)
@@ -546,6 +567,10 @@ wrapper {
546567
final String sha256Sum = new String(sha256Uri.toURL().bytes)
547568
wrapper.getPropertiesFile() << "distributionSha256Sum=${sha256Sum}\n"
548569
println "Added checksum to wrapper properties"
570+
// Update build-tools to reflect the Gradle upgrade
571+
// TODO: we can remove this once we have tests to make sure older versions work.
572+
project(':build-tools').file('src/main/resources/minimumGradleVersion').text = gradleVersion
573+
println "Updated minimum Gradle Version"
549574
}
550575
}
551576

buildSrc/build.gradle

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,9 @@ plugins {
2525

2626
group = 'org.elasticsearch.gradle'
2727

28-
if (GradleVersion.current() < GradleVersion.version('4.9')) {
29-
throw new GradleException('Gradle 4.9+ is required to build elasticsearch')
28+
String minimumGradleVersion = file('src/main/resources/minimumGradleVersion').text.trim()
29+
if (GradleVersion.current() < GradleVersion.version(minimumGradleVersion)) {
30+
throw new GradleException("Gradle ${minimumGradleVersion}+ is required to build elasticsearch")
3031
}
3132

3233
if (JavaVersion.current() < JavaVersion.VERSION_1_8) {
@@ -182,4 +183,12 @@ if (project != rootProject) {
182183
testClass = 'org.elasticsearch.gradle.test.GradleUnitTestCase'
183184
integTestClass = 'org.elasticsearch.gradle.test.GradleIntegrationTestCase'
184185
}
186+
187+
/*
188+
* We alread configure publication and we don't need or want this one that
189+
* comes from the java-gradle-plugin.
190+
*/
191+
afterEvaluate {
192+
generatePomFileForPluginMavenPublication.enabled = false
193+
}
185194
}

0 commit comments

Comments
 (0)