Merge branch 'master' into feature/ml-inference

Author: benwtrent

CONTRIBUTING.md

@@ -219,11 +219,47 @@ by running Gradle with `-Dspotless.paddedcell`.
 
 ### License Headers
 
-We require license headers on all Java files. You will notice that all the Java files in
-the top-level `x-pack` directory contain a separate license from the rest of the repository. This
-directory contains commercial code that is associated with a separate license. It can be helpful
-to have the IDE automatically insert the appropriate license header depending which part of the project
-contributions are made to.
+We require license headers on all Java files. With the exception of the
+top-level `x-pack` directory, all contributed code should have the following
+license header unless instructed otherwise:
+
+    /*
+     * Licensed to Elasticsearch under one or more contributor
+     * license agreements. See the NOTICE file distributed with
+     * this work for additional information regarding copyright
+     * ownership. Elasticsearch licenses this file to you under
+     * the Apache License, Version 2.0 (the "License"); you may
+     * not use this file except in compliance with the License.
+     * You may obtain a copy of the License at
+     *
+     *    http://www.apache.org/licenses/LICENSE-2.0
+     *
+     * Unless required by applicable law or agreed to in writing,
+     * software distributed under the License is distributed on an
+     * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+     * KIND, either express or implied.  See the License for the
+     * specific language governing permissions and limitations
+     * under the License.
+     */
+
+The top-level `x-pack` directory contains code covered by the [Elastic
+license](licenses/ELASTIC-LICENSE.txt). Community contributions to this code are
+welcome, and should have the following license header unless instructed
+otherwise:
+
+    /*
+     * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+     * or more contributor license agreements. Licensed under the Elastic License;
+     * you may not use this file except in compliance with the Elastic License.
+     */
+
+It is important that the only code covered by the Elastic licence is contained
+within the top-level `x-pack` directory. The build will fail its pre-commit
+checks if contributed code does not have the appropriate license headers.
+
+You may find it helpful to configure your IDE to automatically insert the
+appropriate license header depending on the part of the project to which you are
+contributing.
 
 #### IntelliJ: Copyright & Scope Profiles
 

TESTING.asciidoc

@@ -152,7 +152,7 @@ Default value provided below in [brackets].
 
 === Load balancing and caches.
 
-By default the tests run on multiple processes using all the available cores on all 
+By default the tests run on multiple processes using all the available cores on all
 available CPUs. Not including hyper-threading.
 If you want to explicitly specify the number of JVMs you can do so on the command
 line:
@@ -300,9 +300,17 @@ e.g. -Dtests.rest.blacklist=index/*/Index document,get/10_basic/*
 
 == Testing packaging
 
-The packaging tests use Vagrant virtual machines to verify that installing
-and running elasticsearch distributions works correctly on supported operating systems.
-These tests should really only be run in vagrant vms because they're destructive.
+The packaging tests use Vagrant virtual machines or cloud instances to verify
+that installing and running Elasticsearch distributions works correctly on
+supported operating systems. These tests should really only be run on ephemeral
+systems because they're destructive; that is, these tests install and remove
+packages and freely modify system settings, so you will probably regret it if
+you execute them on your development machine.
+
+When you run a packaging test, Gradle will set up the target VM and mount your
+repository directory in the VM. Once this is done, a Gradle task will issue a
+Vagrant command to run a *nested* Gradle task on the VM. This nested Gradle
+runs the actual "destructive" test classes.
 
 . Install Virtual Box and Vagrant.
 +
@@ -313,40 +321,29 @@ a bit more performance out of the process:
 vagrant plugin install vagrant-cachier
 --------------------------------------
 +
-. Validate your installed dependencies:
+. You can run all of the OS packaging tests with `./gradlew packagingTest`.
+This task includes our legacy `bats` tests. To run only the OS tests that are
+written in Java, run `.gradlew distroTest`, will cause Gradle to build the tar,
+zip, and deb packages and all the plugins.  It will then run the tests on every
+available system. This will take a very long time.
 +
--------------------------------------
-./gradlew :qa:vagrant:vagrantCheckVersion
--------------------------------------
+Fortunately, the various systems under test have their own Gradle tasks under
+`qa/os`. To find out what packaging combinations can be tested on a system, run
+the `tasks` task.  For example:
 +
-. Download and smoke test the VMs with `./gradlew vagrantSmokeTest` or
-`./gradlew -Pvagrant.boxes=all vagrantSmokeTest`. The first time you run this it will
-download the base images and provision the boxes and immediately quit. Downloading all
-the images may take a long time. After the images are already on your machine, they won't
-be downloaded again unless they have been updated to a new version.
+----------------------------------
+./gradlew :qa:os:ubuntu-1804:tasks
+----------------------------------
 +
-. Run the tests with `./gradlew packagingTest`. This will cause Gradle to build
-the tar, zip, and deb packages and all the plugins. It will then run the tests
-on ubuntu-1604 and centos-7. We chose those two distributions as the default
-because they cover deb and rpm packaging and SyvVinit and systemd.
-
-You can choose which boxes to test by setting the `-Pvagrant.boxes` project property. All of
-the valid options for this property are:
-
-* `sample` - The default, only chooses ubuntu-1604 and centos-7
-* List of box names, comma separated (e.g. `oel-7,fedora-28`) - Chooses exactly the boxes listed.
-* `linux-all` - All linux boxes.
-* `windows-all` - All Windows boxes. If there are any Windows boxes which do not
-have images available when this value is provided, the build will fail.
-* `all` - All boxes we test. If there are any boxes (e.g. Windows) which do not have images
-available when this value is provided, the build will fail.
-
-For a complete list of boxes on which tests can be run, run `./gradlew :qa:vagrant:listAllBoxes`.
-For a list of boxes that have images available from your configuration, run
-`./gradlew :qa:vagrant:listAvailableBoxes`
-
-Note that if you interrupt gradle in the middle of running these tasks, any boxes started
-will remain running and you'll have to stop them manually with `./gradlew stop` or
+If you want a quick test of the tarball and RPM packagings for Centos 7, you
+would run:
++
+-------------------------------------------------------------------------------------------------
+./gradlew :qa:os:centos-7:distroTest.default-rpm :qa:os:centos-7:distroTest.default-linux-archive
+-------------------------------------------------------------------------------------------------
+
+Note that if you interrupt Gradle in the middle of running these tasks, any boxes started
+will remain running and you'll have to stop them manually with `./gradlew --stop` or
 `vagrant halt`.
 
 All the regular vagrant commands should just work so you can get a shell in a
@@ -393,9 +390,7 @@ or passed on the command line like `-Pvagrant.windows-2012r2.id=my-image-id`
 `-Pvagrant.windows-2016=another-image-id`
 
 These properties are required for Windows support in all gradle tasks that
-handle packaging tests. Either or both may be specified. Remember that to run tests
-on these boxes, the project property `vagrant.boxes` still needs to be set to a
-value that will include them.
+handle packaging tests. Either or both may be specified.
 
 If you're running vagrant commands outside of gradle, specify the Windows boxes
 with the environment variables
@@ -438,84 +433,23 @@ that'd consume a ton of ram.
 
 === Iterating on packaging tests
 
-Running the packaging tests through gradle can take a while because it will start
-and stop the VM each time. You can iterate faster by keeping the VM up and running
-the tests directly.
-
-The packaging tests use a random seed to determine which past version to use for
-testing upgrades. To use a single past version fix the test seed when running
-the commands below (see <<Seed and repetitions.>>)
-
-First build the packaging tests and their dependencies
-
---------------------------------------------
-./gradlew :qa:vagrant:setupPackagingTest
---------------------------------------------
-
-Then choose the VM you want to test on and bring it up. For example, to bring
-up Debian 9 use the gradle command below. Bringing the box up with vagrant directly
-may not mount the packaging test project in the right place. Once the VM is up, ssh
-into it
-
---------------------------------------------
-./gradlew :qa:vagrant:vagrantDebian9#up
-vagrant ssh debian-9
---------------------------------------------
-
-Now inside the VM, start the packaging tests from the terminal. There are two packaging
-test projects. The old ones are written with https://github.com/sstephenson/bats[bats]
-and only run on linux. To run them do
-
---------------------------------------------
-cd $PACKAGING_ARCHIVES
-
-# runs all bats tests
-sudo bats $BATS_TESTS/*.bats
-
-# you can also pass specific test files
-sudo bats $BATS_TESTS/20_tar_package.bats $BATS_TESTS/25_tar_plugins.bats
---------------------------------------------
-
-The new packaging tests are written in Java and run on both linux and windows. On
-linux (again, inside the VM)
-
---------------------------------------------
-# run the full suite
-sudo bash $PACKAGING_TESTS/run-tests.sh
-
-# run specific test cases
-sudo bash $PACKAGING_TESTS/run-tests.sh \
-  org.elasticsearch.packaging.test.DefaultWindowsZipTests \
-  org.elasticsearch.packaging.test.OssWindowsZipTests
---------------------------------------------
-
-or on Windows, from a terminal running as Administrator
-
---------------------------------------------
-# run the full suite
-powershell -File $Env:PACKAGING_TESTS/run-tests.ps1
-
-# run specific test cases
-powershell -File $Env:PACKAGING_TESTS/run-tests.ps1 `
-  org.elasticsearch.packaging.test.DefaultWindowsZipTests `
-  org.elasticsearch.packaging.test.OssWindowsZipTests
---------------------------------------------
-
-Note that on Windows boxes when running from inside the GUI, you may have to log out and
-back in to the `vagrant` user (password `vagrant`) for the environment variables that
-locate the packaging tests and distributions to take effect, due to how vagrant provisions
-Windows machines.
+Because our packaging tests are capable of testing many combinations of OS
+(e.g., Windows, Linux, etc.), package type (e.g., zip file, RPM, etc.),
+Elasticsearch distribution type (e.g., default or OSS), and so forth, it's
+faster to develop against smaller subsets of the tests. For example, to run
+tests for the default archive distribution on Fedora 28:
 
-When you've made changes you want to test, keep the VM up and reload the tests and
-distributions inside by running (on the host)
+-----------------------------------------------------------
+./gradlew :qa:os:fedora-28:distroTest.default-linux-archive
+-----------------------------------------------------------
 
---------------------------------------------
-./gradlew :qa:vagrant:clean :qa:vagrant:setupPackagingTest
---------------------------------------------
+These test tasks can use the `--tests`, `--info`, and `--debug` parameters just like
+non-OS tests can. For example:
 
-Note: Starting vagrant VM outside of the elasticsearch folder requires to
-indicates the folder that contains the Vagrantfile using the VAGRANT_CWD
-environment variable.
+-----------------------------------------------------------
+./gradlew :qa:os:fedora-28:distroTest.default-linux-archive \
+  --tests "com.elasticsearch.packaging.test.ArchiveTests"
+-----------------------------------------------------------
 
 == Testing backwards compatibility
 
@@ -536,7 +470,7 @@ version 5.3.2 run:
 Tests are ran for versions that are not yet released but with which the current version will be compatible with.
 These are automatically checked out and built from source.
 See link:./buildSrc/src/main/java/org/elasticsearch/gradle/VersionCollection.java[VersionCollection]
-and link:./distribution/bwc/build.gradle[distribution/bwc/build.gradle] 
+and link:./distribution/bwc/build.gradle[distribution/bwc/build.gradle]
 for more information.
 
 When running `./gradlew check`, minimal bwc checks are also run against compatible versions that are not yet released.
@@ -657,7 +591,7 @@ care.
 == Test coverage analysis
 
 Generating test coverage reports for Elasticsearch is currently not possible through Gradle.
-However, it _is_ possible to gain insight in code coverage using IntelliJ's built-in coverage 
+However, it _is_ possible to gain insight in code coverage using IntelliJ's built-in coverage
 analysis tool that can measure coverage upon executing specific tests. Eclipse may also be able
 to do the same using the EclEmma plugin.
 

build.gradle

@@ -32,14 +32,15 @@ import org.gradle.util.GradleVersion
 import static org.elasticsearch.gradle.tool.Boilerplate.maybeConfigure
 
 plugins {
-    id 'com.gradle.build-scan' version '2.4'
+    id 'com.gradle.build-scan' version '2.4.2'
     id 'lifecycle-base'
     id 'elasticsearch.global-build-info'
     id "com.diffplug.gradle.spotless" version "3.24.2" apply false
 }
 
 apply plugin: 'nebula.info-scm'
 apply from: 'gradle/build-scan.gradle'
+apply from: 'gradle/build-complete.gradle'
 
 // common maven publishing configuration
 allprojects {

buildSrc/src/main/java/org/elasticsearch/gradle/testclusters/RestTestRunnerTask.java

@@ -5,10 +5,9 @@
 import org.gradle.api.tasks.testing.Test;
 
 import java.util.Collection;
+import java.util.Collections;
 import java.util.HashSet;
 
-import static org.elasticsearch.gradle.testclusters.TestDistribution.INTEG_TEST;
-
 /**
  * Customized version of Gradle {@link Test} task which tracks a collection of {@link ElasticsearchCluster} as a task input. We must do this
  * as a custom task type because the current {@link org.gradle.api.tasks.TaskInputs} runtime API does not have a way to register
@@ -20,9 +19,17 @@ public class RestTestRunnerTask extends Test implements TestClustersAware {
     private Collection<ElasticsearchCluster> clusters = new HashSet<>();
 
     public RestTestRunnerTask() {
-        super();
-        this.getOutputs().doNotCacheIf("Build cache is only enabled for tests against clusters using the 'integ-test' distribution",
-            task -> clusters.stream().flatMap(c -> c.getNodes().stream()).anyMatch(n -> n.getTestDistribution() != INTEG_TEST));
+        this.getOutputs().doNotCacheIf("Caching disabled for this task since it uses a cluster shared by other tasks",
+            /*
+             * Look for any other tasks which use the same cluster as this task. Since tests often have side effects for the cluster they
+             * execute against, this state can cause issues when trying to cache tests results of tasks that share a cluster. To avoid any
+             * undesired behavior we simply disable the cache if we detect that this task uses a cluster shared between multiple tasks.
+             */
+            t -> getProject().getTasks().withType(RestTestRunnerTask.class)
+                .stream()
+                .filter(task -> task != this)
+                .anyMatch(task -> Collections.disjoint(task.getClusters(), getClusters()) == false)
+        );
     }
 
     @Override

buildSrc/version.properties

@@ -1,5 +1,5 @@
 elasticsearch     = 8.0.0
-lucene            = 8.3.0-snapshot-47aece66b48
+lucene            = 8.3.0-snapshot-25968e3b75e
 
 bundled_jdk_vendor = adoptopenjdk
 bundled_jdk = 13+33

client/rest-high-level/build.gradle

@@ -129,8 +129,11 @@ testClusters.integTest {
 
   setting 'indices.lifecycle.poll_interval', '1000ms'
   keystore 'xpack.security.transport.ssl.truststore.secure_password', 'testnode'
+  extraConfigFile 'roles.yml', file('roles.yml')
   user username:  System.getProperty('tests.rest.cluster.username', 'test_user'),
-          password:  System.getProperty('tests.rest.cluster.password', 'test-password')
+          password:  System.getProperty('tests.rest.cluster.password', 'test-password'),
+          role:  System.getProperty('tests.rest.cluster.role', 'admin')
+  user username:  'admin_user', password: 'admin-password'
 
   extraConfigFile nodeCert.name, nodeCert
   extraConfigFile nodeTrustStore.name, nodeTrustStore

client/rest-high-level/roles.yml

@@ -0,0 +1,12 @@
+admin:
+  cluster:
+    - all
+  indices:
+    - names: '*'
+      privileges:
+        - all
+  run_as: [ '*' ]
+  applications:
+    - application: '*'
+      privileges: [ '*' ]
+      resources: [ '*' ]